Commit Graph

4 Commits

Author SHA1 Message Date
Georgi Gerganov
5ec18934ad
convert-new.py : pick #2427 for HF 70B support 2023-08-16 20:16:15 +03:00
goerch
afc4ca2889
convert : update convert-new.py with tokenizer fixes (#2614)
* Merge tokenizer fixes into the gguf branch.

* Add test vocabularies

* Adapt convert-new.py (and fix a clang-cl compiler error on windows)
2023-08-14 20:20:04 +03:00
Georgi Gerganov
d2bb3ac10b
convert.py : remove GGML vocab + other obsolete stuff 2023-07-27 16:36:35 +03:00
Georgi Gerganov
68f53485e4
convert.py : start a new simplified implementation by removing old stuff 2023-07-27 15:56:53 +03:00