Merge branch 'ggerganov-gg/tokenizer-cleanup' into refactor-tokenizer

2024-11-11 13:30:35 +00:00 · 2024-09-20 18:48:58 +08:00 · 2024-09-20 18:48:58 +08:00 · 8be5d11c0d
commit 8be5d11c0d
parent 02629d98f1 d653d25116
1 changed files with 1 additions and 1 deletions
--- a/src/llama-vocab.h
+++ b/src/llama-vocab.h
@ -139,4 +139,4 @@ int32_t llama_detokenize_impl(
                            char * text,
                         int32_t   text_len_max,
                            bool   remove_special,
-                            bool   unparse_special);
+                            bool   unparse_special);