Daniel Bevenius 6f63d646c1 tokenize : add --show-count (token) option (#8299) ... This commit adds a new option to the tokenize example, --show-count. When this is set the total number of tokens are printed to stdout. This was added as an option as I was concerned that there might be scripts that use the output from this program and it might be better to not print this information by default. The motivation for this is that can be useful to find out how many tokens a file contains, for example when trying to determine prompt input file sizes for testing. Signed-off-by: Daniel Bevenius <daniel.bevenius@gmail.com>		2024-07-04 19:38:58 +03:00
..
CMakeLists.txt	build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809)	2024-06-13 00:41:52 +01:00
tokenize.cpp	tokenize : add --show-count (token) option (#8299)	2024-07-04 19:38:58 +03:00