root/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggerganov/llama.cpp.git synced 2024-12-28 12:24:35 +00:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Update test-tokenizer-random.py

Browse Source
This commit is contained in:
Robert 2024-11-12 22:28:52 -08:00 committed by GitHub
parent 3275e29360
commit 5edd022d6a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
tests/test-tokenizer-random.py
View File

@ -3,13 +3,13 @@
# Test libllama tokenizer == AutoTokenizer. # Test libllama tokenizer == AutoTokenizer.
# Brute force random words/text generation. # Brute force random words/text generation.
# #
#Sample usage: # Sample usage:
# #
# python3 tests/test-tokenizer-random.py ./models/ggml-vocab-llama-bpe.gguf ./models/tokenizers/llama-bpe # python3 tests/test-tokenizer-random.py ./models/ggml-vocab-llama-bpe.gguf ./models/tokenizers/llama-bpe
# #
from __future__ import annotations from __future__ import annotations
#
import time import time
import logging import logging
import argparse import argparse