3 Commits

Author	SHA1	Message	Date
Galunid	69a6735087	Update special token handling in conversion scripts for gpt2 derived tokenizers (#3746) ... We still have the heads up in `README.md` regarding `bpe` tokenizers and this patch is needed for - a couple of tokenizer tests - some more `special` and `non-special` added tokens handling (as far as I understand it) * Update special token handling * Add mpt	2023-10-23 21:46:00 +02:00
Kerfuffle	a5e7dbd614	llama : validate special token ids are in range when loading GGUF model (#3635) ... * Add validation for special token ids to llama.cpp Small optimization for llama_byte_to_token SPM mode * Fix BPE newline check, only I could break something so simple * Killll meeeeee * Account for GGUF_KEY_KEY only setting when the key exists * Minor code cleanups. * Fix convert.py error msg when added tokens are out of range * Make gguf SpecialVocab vocab size-aware Update conversion scripts accordingly * Avoid a string copy Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2023-10-22 21:14:56 +03:00
Xingchen Song(宋星辰)	02d2875def	llm : add bloom models (#3553) ... * feat: Support bloom models * fix(bloom): fix model size --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2023-10-10 17:48:21 +03:00