Pedro Cuenca b97bc3966e llama : support Llama 3 HF conversion (#6745) ... * Support Llama 3 conversion The tokenizer is BPE. * style * Accept suggestion Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> * llama : add llama_token_is_eog() ggml-ci * llama : auto-detect more EOT tokens when missing in KV data * convert : replacing EOS token is a hack * llama : fix codegemma EOT token + add TODOs * llama : fix model type string for 8B model --------- Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>		2024-04-21 14:50:41 +03:00
..
LibLlama.swift	llama : support Llama 3 HF conversion (#6745)	2024-04-21 14:50:41 +03:00