7 Commits

Author	SHA1	Message	Date
M. Yusuf Sarıgöz	1d93d04ce2	gguf : refactor pth to gguf conversion script	2023-08-17 19:58:27 +03:00
Georgi Gerganov	e0429d38e4	convert-new.py : output gguf (#2635) ... * convert-new.py : output gguf (WIP) * convert-new.py : add gguf key-value pairs * llama : add hparams.ctx_train + no longer print ftype * convert-new.py : minor fixes * convert-new.py : vocab-only option should work now * llama : fix tokenizer to use llama_char_to_byte * tests : add new ggml-vocab-llama.gguf * convert-new.py : tensor name mapping * convert-new.py : add map for skipping tensor serialization * convert-new.py : convert script now works * gguf.py : pick some of the refactoring from #2644 * convert-new.py : minor fixes	2023-08-17 17:19:52 +03:00
Georgi Gerganov	c8ee87f141	gguf.py : merge all files in gguf.py	2023-08-16 19:55:49 +03:00
Georgi Gerganov	88b5769487	gguf : deduplicate (#2629) ... * gguf : better type names * dedup : CPU + Metal is working * ggml : fix warnings about unused results * llama.cpp : fix line feed and compiler warning * llama : fix strncpy warning + note token_to_str does not write null * llama : restore the original load/save session implementation Will migrate this to GGUF in the future * convert-llama-h5-to-gguf.py : support alt ctx param name * ggml : assert when using ggml_mul with non-F32 src1 * examples : dedup simple --------- Co-authored-by: klosax <131523366+klosax@users.noreply.github.com>	2023-08-16 19:25:29 +03:00
klosax	2ae0e985b3	convert-llama-7b-pth-to-gguf.py : add tensor data layout	2023-08-15 19:55:13 +02:00
klosax	ab2cbd03ca	convert-llama-7b-pth-to-gguf.py : add token types	2023-08-14 22:10:50 +02:00
klosax	6f64b6c0f8	Create convert-llama-7b-pth-to-gguf.py	2023-08-14 13:51:09 +02:00