root/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggerganov/llama.cpp.git synced 2024-12-27 03:44:35 +00:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity
8d177eddeb
llama.cpp/tests
History
goerch 8d177eddeb
llama : improve token type support (#2668) 
* Merge tokenizer fixes into the gguf branch.

* Add test vocabularies

* Adapt convert-new.py (and fix a clang-cl compiler error on windows)

* Improved tokenizer test

But does it work on MacOS?

* Improve token type support

- Added @klosax code to convert.py
- Improved token type support in vocabulary

* Exclude platform dependent tests

* More sentencepiece compatibility by eliminating magic numbers

* Restored accidentally removed comment
2023-08-21 18:56:02 +03:00
..
CMakeLists.txt llama : remove C++ API + reorganize common source in /common dir 2023-08-18 16:22:48 +03:00
test-double-float.cpp tests : Fix compilation warnings (Linux/GCC) (#2451) 2023-08-02 11:06:19 +03:00
test-grad0.cpp tests : Fix compilation warnings (Linux/GCC) (#2451) 2023-08-02 11:06:19 +03:00
test-grammar-parser.cpp llama : remove C++ API + reorganize common source in /common dir 2023-08-18 16:22:48 +03:00
test-llama-grammar.cpp llama : remove C++ API + reorganize common source in /common dir 2023-08-18 16:22:48 +03:00
test-opt.cpp tests : Fix compilation warnings (Linux/GCC) (#2451) 2023-08-02 11:06:19 +03:00
test-quantize-fns.cpp ggml : generalize quantize_fns for simpler FP16 handling (#1237) 2023-07-05 19:13:06 +03:00
test-quantize-perf.cpp ggml : generalize quantize_fns for simpler FP16 handling (#1237) 2023-07-05 19:13:06 +03:00
test-sampling.cpp ci : integrate with ggml-org/ci (#2250) 2023-07-18 14:24:43 +03:00
test-tokenizer-0.cpp llama : remove C++ API + reorganize common source in /common dir 2023-08-18 16:22:48 +03:00
test-tokenizer-1.cpp llama : improve token type support (#2668) 2023-08-21 18:56:02 +03:00