mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-11-13 14:29:52 +00:00
f4ab2a4147
* merged the changes from deepseeker models to main branch
* Moved regex patterns to unicode.cpp and updated unicode.h
* Moved header files
* Resolved issues
* added and refactored unicode_regex_split and related functions
* Updated/merged the deepseek coder pr
* Refactored code
* Adding unicode regex mappings
* Adding unicode regex function
* Added needed functionality, testing remains
* Fixed issues
* Fixed issue with gpt2 regex custom preprocessor
* unicode : fix? unicode_wstring_to_utf8
* lint : fix whitespaces
* tests : add tokenizer tests for numbers
* unicode : remove redundant headers
* tests : remove and rename tokenizer test scripts
* tests : add sample usage
* gguf-py : reader prints warnings on duplicate keys
* llama : towards llama3 tokenization support (wip)
* unicode : shot in the dark to fix tests on Windows
* unicode : first try custom implementations
* convert : add "tokenizer.ggml.pre" GGUF KV (wip)
* llama : use new pre-tokenizer type
* convert : fix pre-tokenizer type writing
* lint : fix
* make : add test-tokenizer-0-llama-v3
* wip
* models : add llama v3 vocab file
* llama : adapt punctuation regex + add llama 3 regex
* minor
* unicode : set bomb
* unicode : set bomb
* unicode : always use std::wregex
* unicode : support \p{N}, \p{L} and \p{P} natively
* unicode : try fix windows
* unicode : category support via std::regex
* unicode : clean-up
* unicode : simplify
* convert : add convert-hf-to-gguf-update.py
ggml-ci
* lint : update
* convert : add falcon
ggml-ci
* unicode : normalize signatures
* lint : fix
* lint : fix
* convert : remove unused functions
* convert : add comments
* convert : exercise contractions
ggml-ci
* lint : fix
* cmake : refactor test targets
* tests : refactor vocab tests
ggml-ci
* tests : add more vocabs and tests
ggml-ci
* unicode : cleanup
* scripts : ignore new update script in check-requirements.sh
* models : add phi-3, mpt, gpt-2, starcoder
* tests : disable obsolete
ggml-ci
* tests : use faster bpe test
ggml-ci
* llama : more prominent warning for old BPE models
* tests : disable test-tokenizer-1-bpe due to slowness
ggml-ci
---------
Co-authored-by: Jaggzh <jaggz.h@gmail.com>
Co-authored-by: Kazim Abrar Mahi <kazimabrarmahi135@gmail.com>
126 lines
1.4 KiB
Plaintext
126 lines
1.4 KiB
Plaintext
*.o
|
|
*.a
|
|
*.so
|
|
*.gguf
|
|
*.bin
|
|
*.exe
|
|
*.dll
|
|
*.log
|
|
*.gcov
|
|
*.gcno
|
|
*.gcda
|
|
*.dot
|
|
*.bat
|
|
*.tmp
|
|
*.metallib
|
|
*.etag
|
|
*.lastModified
|
|
.DS_Store
|
|
.build/
|
|
.cache/
|
|
.ccls-cache/
|
|
.direnv/
|
|
.envrc
|
|
.swiftpm
|
|
.venv
|
|
.clang-tidy
|
|
.vs/
|
|
.vscode/
|
|
.idea/
|
|
|
|
ggml-metal-embed.metal
|
|
|
|
lcov-report/
|
|
gcovr-report/
|
|
|
|
build*
|
|
!build.zig
|
|
cmake-build-*
|
|
out/
|
|
tmp/
|
|
|
|
models/*
|
|
models-mnt
|
|
|
|
/Pipfile
|
|
/baby-llama
|
|
/beam-search
|
|
/benchmark-matmult
|
|
/convert-llama2c-to-ggml
|
|
/embd-input-test
|
|
/embedding
|
|
/eval-callback
|
|
/gguf
|
|
/gguf-llama-simple
|
|
/gguf-split
|
|
/gritlm
|
|
/imatrix
|
|
/infill
|
|
/libllama.so
|
|
/llama-bench
|
|
/llava-cli
|
|
/lookahead
|
|
/lookup
|
|
/lookup-create
|
|
/lookup-merge
|
|
/lookup-stats
|
|
/main
|
|
/metal
|
|
/passkey
|
|
/perplexity
|
|
/q8dot
|
|
/quantize
|
|
/quantize-stats
|
|
/result
|
|
/save-load-state
|
|
/server
|
|
/simple
|
|
/batched
|
|
/batched-bench
|
|
/export-lora
|
|
/finetune
|
|
/retrieval
|
|
/speculative
|
|
/parallel
|
|
/train-text-from-scratch
|
|
/tokenize
|
|
/vdot
|
|
/common/build-info.cpp
|
|
arm_neon.h
|
|
compile_commands.json
|
|
CMakeSettings.json
|
|
|
|
__pycache__
|
|
dist
|
|
|
|
zig-out/
|
|
zig-cache/
|
|
|
|
ppl-*.txt
|
|
qnt-*.txt
|
|
perf-*.txt
|
|
|
|
examples/jeopardy/results.txt
|
|
examples/server/*.html.hpp
|
|
examples/server/*.js.hpp
|
|
examples/server/*.mjs.hpp
|
|
|
|
poetry.lock
|
|
poetry.toml
|
|
nppBackup
|
|
|
|
# Test binaries
|
|
/tests/test-grammar-parser
|
|
/tests/test-llama-grammar
|
|
/tests/test-double-float
|
|
/tests/test-grad0
|
|
/tests/test-opt
|
|
/tests/test-quantize-fns
|
|
/tests/test-quantize-perf
|
|
/tests/test-sampling
|
|
/tests/test-tokenizer-0
|
|
/tests/test-tokenizer-1-spm
|
|
/tests/test-tokenizer-1-bpe
|
|
/tests/test-rope
|
|
/tests/test-backend-ops
|