6 Commits

Author	SHA1	Message	Date
goerch	49c25cce19	tests : use new tokenizer type API (#2692) ... * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows) * Improved tokenizer test But does it work on MacOS? * Improve token type support - Added @klosax code to convert.py - Improved token type support in vocabulary * Exclude platform dependent tests * More sentencepiece compatibility by eliminating magic numbers * Restored accidentally removed comment * Improve commentary * Use token type API in test-tokenizer-1.cpp	2023-08-21 20:11:14 +03:00
goerch	8d177eddeb	llama : improve token type support (#2668) ... * Merge tokenizer fixes into the gguf branch. * Add test vocabularies * Adapt convert-new.py (and fix a clang-cl compiler error on windows) * Improved tokenizer test But does it work on MacOS? * Improve token type support - Added @klosax code to convert.py - Improved token type support in vocabulary * Exclude platform dependent tests * More sentencepiece compatibility by eliminating magic numbers * Restored accidentally removed comment	2023-08-21 18:56:02 +03:00
Georgi Gerganov	2d6c2c757c	llama : remove C++ API + reorganize common source in /common dir	2023-08-18 16:22:48 +03:00
Georgi Gerganov	660ca9bbca	llama : re-order functions	2023-08-18 14:56:36 +03:00
Georgi Gerganov	7494c78428	llama : sync gguf-llama with llama (#2613) ... * llama : sync gguf-llama with llama * tests : fix build + warnings (test-tokenizer-1 still fails) * tests : fix wstring_convert * convert : fix layer names * llama : sync gguf-llama.cpp * convert : update HF converter to new tokenizer voodoo magics	2023-08-14 21:33:33 +03:00
goerch	ec1b100720	llama : tokenizer fixes (#2549) ... * Merge tokenizer fixes into the gguf branch. * Add test vocabularies	2023-08-14 19:30:28 +03:00