Francis Couture-Harpin f9d42c598b convert_hf : identify more added control tokens for SPM tokenziers ... This makes Gemma and Gemma-2 tokenize pretty much EVERYTHING correctly, including HTML tags and consecutive spaces, but it unfortunately requires model re-conversion. There seems to be a weird behavior of the HF tokenizer for Gemma, which prefers to use the 16-space token over more lengthy space tokens, while using the SentencePiece tokenizer does not do this. (the implementation in llama.cpp has the same behavior as SentencePiece) * llama : fix wrong pre-tokenization of byte tokens		2024-07-07 23:28:38 -04:00
..
.gitignore	tests : gitignore ggml-common.h	2024-03-09 14:17:11 +02:00
CMakeLists.txt	ggml : fix loongson compile warnings (#7537)	2024-05-31 14:17:10 +03:00
get-model.cpp	ci : add model tests + script wrapper (#4586)	2024-01-26 14:18:00 +02:00
get-model.h	ci : add model tests + script wrapper (#4586)	2024-01-26 14:18:00 +02:00
run-json-schema-to-grammar.mjs	json-schema-to-grammar improvements (+ added to server) (#5978)	2024-03-21 11:50:43 +00:00
test-autorelease.cpp	ggml : add numa options (#5377)	2024-02-16 11:31:07 +02:00
test-backend-ops.cpp	cuda : update supports_op for matrix multiplication (#8245)	2024-07-02 09:39:38 +03:00
test-c.c	Nomic Vulkan backend (#4456)	2024-01-29 15:50:50 -05:00
test-chat-template.cpp	llama : support glm3 and glm4 (#8031)	2024-07-07 15:52:10 +03:00
test-double-float.cpp	ggml : move FP16 <-> FP32 code to ggml-impl.h (#3861)	2023-10-30 19:19:15 +02:00
test-grad0.cpp	ggml : refactor rope norm/neox (#7634)	2024-06-05 11:29:20 +03:00
test-grammar-integration.cpp	json: restore default additionalProperties to false, fix some pattern escapes (#8180)	2024-06-28 09:26:45 +01:00
test-grammar-parser.cpp	grammars: x{min,max} repetition operator (#6640)	2024-06-06 10:07:06 +01:00
test-json-schema-to-grammar.cpp	json: attempt to skip slow tests when running under emulator (#8189)	2024-06-28 18:02:05 +01:00
test-llama-grammar.cpp	llama : return nullptr from llama_grammar_init (#8093)	2024-06-25 15:07:28 -04:00
test-model-load-cancel.cpp	ggml : add numa options (#5377)	2024-02-16 11:31:07 +02:00
test-opt.cpp	code : normalize enum names (#5697)	2024-02-25 12:09:09 +02:00
test-quantize-fns.cpp	tests : include IQ2_XXS and IQ2_XS in test-quantize-fns (#6303)	2024-03-25 19:33:15 +02:00
test-quantize-perf.cpp	ggml : add mmla kernels for quantized GEMM (#4966)	2024-02-11 15:22:33 +02:00
test-rope.cpp	Removes multiple newlines at the end of files that is breaking the editorconfig step of CI. (#8258)	2024-07-02 12:18:10 -04:00
test-sampling.cpp	sampling: fix top_k <= 0 (#5388)	2024-02-08 09:46:30 +01:00
test-tokenizer-0.cpp	llama : fix detection of control-like user-defined tokens	2024-07-07 16:13:35 -04:00
test-tokenizer-0.py	py : logging and flake8 suppression refactoring (#7081)	2024-05-05 08:07:48 +03:00
test-tokenizer-0.sh	tests : fix test-tokenizer-0.sh	2024-05-28 15:04:09 +03:00
test-tokenizer-1-bpe.cpp	Detokenizer fixes (#8039)	2024-07-05 19:01:35 +02:00
test-tokenizer-1-spm.cpp	Detokenizer fixes (#8039)	2024-07-05 19:01:35 +02:00
test-tokenizer-random.py	convert_hf : identify more added control tokens for SPM tokenziers	2024-07-07 23:28:38 -04:00