llama : fix Viking pre-tokenizer regex

The order was previously wrong, which caused errors in some tests.
2024-12-26 03:14:35 +00:00 · 2024-07-08 16:34:39 -04:00 · 2024-07-08 16:34:39 -04:00 · 31a1b0eeaa
commit 31a1b0eeaa
parent f9d42c598b
1 changed files with 1 additions and 1 deletions
--- a/src/llama.cpp
+++ b/src/llama.cpp
@ -15440,8 +15440,8 @@ struct llm_tokenizer_bpe {
                break;
            case LLAMA_VOCAB_PRE_TYPE_VIKING:
                regex_exprs = {
-                    "\\p{N}",
                    " ?[^(\\s|.,!?…。，、।۔،)]+",
+                    "\\p{N}",
                };
                break;
            default: