Georgi Gerganov 92139b90af

tests : add test-tokenizer-0.sh + fix some tokenizers (#7036) ...

* tests : add test-tokenizer-0.sh

* unicode : add all unicode number ranges

* starcoder : fix pre-tokenizer

* tests : add test that fails with DeepSeek tokenizers

* falcon : fix regex

* unicode : regenerate unicode tables

* refact : add tokenizer model

* lint : fix

* tests : disable failing tests

ggml-ci

* refact : add tests files

ggml-ci

* convert : print -> logging

ggml-ci

* lint : fix

* unicode : digit -> number

* phi-3 : update

2024-05-04 08:32:32 +03:00

1.6 MiB

Raw Permalink History

View Raw