goerch b08e75baea Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170) ... * Fix für #2721 * Reenable tokenizer test for LLaMa * Add `console.cpp` dependency * Fix dependency to `common` * Fixing wrong fix. * Make console usage platform specific Work on compiler warnings. * Adapting makefile * Remove trailing whitespace * Adapting the other parts of the makefile * Fix typo. * Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 * Simplify logic * Add missing change... * Fix ugly compiler warning * llama_tokenize should accept strings containing NUL now * Adding huichen's test case		2023-09-16 13:41:33 +02:00
..
CMakeLists.txt	gguf : new file format with flexible meta data (beta) (#2398)	2023-08-21 23:07:43 +03:00
common.cpp	Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170)	2023-09-16 13:41:33 +02:00
common.h	examples : add compiler version and target to build info (#2998)	2023-09-15 16:59:49 -04:00
console.cpp	check C++ code with -Wmissing-declarations (#3184)	2023-09-15 15:38:27 -04:00
console.h	gguf : new file format with flexible meta data (beta) (#2398)	2023-08-21 23:07:43 +03:00
grammar-parser.cpp	check C++ code with -Wmissing-declarations (#3184)	2023-09-15 15:38:27 -04:00
grammar-parser.h	gguf : new file format with flexible meta data (beta) (#2398)	2023-08-21 23:07:43 +03:00
log.h	examples : replace fprintf to stdout with printf (#3017)	2023-09-05 15:10:27 -04:00