Ren Xuancheng
|
229ffff872
|
llama : add BPE pre-tokenization for Qwen2 (#7114)
* Add BPE pre-tokenization for Qwen2.
* minor : fixes
---------
Co-authored-by: Ren Xuancheng <17811943+jklj077@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
2024-05-08 15:06:43 +03:00 |
|