llama.cpp/common
Jared Van Bortel 1b67731e18
BERT tokenizer fixes (#6498)
Key changes:
* BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS
* Nomic Embed conversion: pad vocab instead of slicing embedding tensor
* llama_tokenize: handle added special tokens like HF does
2024-04-09 13:44:08 -04:00
..
base64.hpp llava : expose as a shared library for downstream projects (#3613) 2023-11-07 00:36:23 +03:00
build-info.cpp.in build : link against build info instead of compiling against it (#3879) 2023-11-02 08:50:16 +02:00
CMakeLists.txt lookup: complement data from context with general text statistics (#5479) 2024-03-23 01:24:36 +01:00
common.cpp BERT tokenizer fixes (#6498) 2024-04-09 13:44:08 -04:00
common.h BERT tokenizer fixes (#6498) 2024-04-09 13:44:08 -04:00
console.cpp check C++ code with -Wmissing-declarations (#3184) 2023-09-15 15:38:27 -04:00
console.h gguf : new file format with flexible meta data (beta) (#2398) 2023-08-21 23:07:43 +03:00
grammar-parser.cpp grammar : verify parsed state (#5950) 2024-03-10 17:17:43 +02:00
grammar-parser.h gguf : new file format with flexible meta data (beta) (#2398) 2023-08-21 23:07:43 +03:00
json-schema-to-grammar.cpp json-schema-to-grammar : fix order of props + non-str const/enum (#6232) 2024-03-22 15:07:44 +02:00
json-schema-to-grammar.h json-schema-to-grammar : fix order of props + non-str const/enum (#6232) 2024-03-22 15:07:44 +02:00
json.hpp json-schema-to-grammar improvements (+ added to server) (#5978) 2024-03-21 11:50:43 +00:00
log.h [SYCL] fix SYCL backend build on windows is break by LOG() error (#6290) 2024-03-25 15:52:41 +08:00
ngram-cache.cpp Fixed lookup compilation issues on Windows (#6273) 2024-03-24 14:21:17 +01:00
ngram-cache.h lookup: complement data from context with general text statistics (#5479) 2024-03-23 01:24:36 +01:00
sampling.cpp sampling : deduplicated code for probability distribution access (#6240) 2024-03-24 10:54:07 +02:00
sampling.h llama : support negative ith in llama_get_ API (#6519) 2024-04-08 16:02:30 +03:00
stb_image.h examples: support LLaVA v1.5 (multimodal model) (#3436) 2023-10-12 18:23:18 +03:00
train.cpp code : normalize enum names (#5697) 2024-02-25 12:09:09 +02:00
train.h sync : ggml (backend v2) (#3912) 2023-11-13 14:16:23 +02:00