llama.cpp

root/llama.cpp

Fork 0

mirror of https://github.com/ggerganov/llama.cpp.git synced 2024-12-26 11:24:35 +00:00

Commit Graph

Author	SHA1	Message	Date
Jiří Podivín	f7d5e97542	py : remove superfluous import statements (#4076 ) Signed-off-by: Jiri Podivin <jpodivin@gmail.com> Co-authored-by: Jiri Podivin <jpodivin@redhat.com>	2023-11-17 17:20:53 +02:00
goerch	233fc1c69f	Minor improvements in GPT2 tokenizer (#3567 ) * Fixing minor bugs in bpe_gpt2_preprocess * Don't add bos token in test	2023-10-10 18:59:52 +02:00
Georgi Gerganov	edd4c14817	llama : more tokenizer fixes (#2810 ) * tests : write a Python tokenizer test (wip) * llama : prefix input text for tokenization with whitespace * llama : distinguish pieces from decoded text + fix detokenization * common : add comments * examples : no longer manually add leading space when tokenizing * tests : use Python to generate tokenizer tests for C++ * tests : add option to tokenize text files ggml-ci * tests : add test-tokenizer-1.py * llama.cpp : fix LF token * hellaswag : move the concat space for clarity * tests : add falcon tests (py + cpp, currently do not pass Unicode) ggml-ci * common : temporary separate llama_detokenize calls for SPM and BPE --------- Co-authored-by: klosax <131523366+klosax@users.noreply.github.com>	2023-08-27 14:19:19 +03:00

Author

SHA1

Message

Date

Jiří Podivín

f7d5e97542

py : remove superfluous import statements (#4076 )

Signed-off-by: Jiri Podivin <jpodivin@gmail.com>
Co-authored-by: Jiri Podivin <jpodivin@redhat.com>

2023-11-17 17:20:53 +02:00

goerch

233fc1c69f

Minor improvements in GPT2 tokenizer (#3567 )

* Fixing minor bugs in bpe_gpt2_preprocess

* Don't add bos token in test

2023-10-10 18:59:52 +02:00

Georgi Gerganov

edd4c14817

llama : more tokenizer fixes (#2810 )

* tests : write a Python tokenizer test (wip)

* llama : prefix input text for tokenization with whitespace

* llama : distinguish pieces from decoded text + fix detokenization

* common : add comments

* examples : no longer manually add leading space when tokenizing

* tests : use Python to generate tokenizer tests for C++

* tests : add option to tokenize text files

ggml-ci

* tests : add test-tokenizer-1.py

* llama.cpp : fix LF token

* hellaswag : move the concat space for clarity

* tests : add falcon tests (py + cpp, currently do not pass Unicode)

ggml-ci

* common : temporary separate llama_detokenize calls for SPM and BPE

---------

Co-authored-by: klosax <131523366+klosax@users.noreply.github.com>

2023-08-27 14:19:19 +03:00

3 Commits