Georgi Gerganov
6490ff7198
py : fix whitespace
2023-08-21 16:42:27 +03:00
klosax
9070e330ab
convert-llama-7b-pth-to-gguf.py : rope scale fix
2023-08-21 14:11:22 +02:00
klosax
dc1f051013
convert-llama-7b-pth-to-gguf.py : rope scale and added tokens
2023-08-21 13:27:53 +02:00
klosax
f838faa874
convert-llama-7b-pth-to-gguf.py : special tokens
2023-08-20 16:56:48 +02:00
klosax
593b04fdcd
convert-llama-7b-pth-to-gguf.py : remove extra kv
2023-08-19 00:32:27 +02:00
Georgi Gerganov
dea5be61d7
editorconfig : fix whitespaces
2023-08-18 12:42:38 +03:00
klosax
3c1b7217a9
convert-llama-7b-pth-to-gguf.py : fixes
2023-08-17 21:44:34 +02:00
M. Yusuf Sarıgöz
fc3a523211
gguf.py : write tensors in a single pass ( #2644 )
...
* gguf : single pass for writing tensors + refactoring writer
* gguf : single pass for writing tensors + refactoring writer
* gguf : single pass for writing tensors + refactoring writer
* gguf : style fixes in simple conversion script
* gguf : refactor gptneox conversion script
* gguf : rename h5 to hf (for HuggingFace)
* gguf : refactor pth to gguf conversion script
* gguf : rm file_type key and method
* gguf.py : fix vertical alignment
* gguf.py : indentation
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-08-17 21:57:39 +03:00
Georgi Gerganov
e0429d38e4
convert-new.py : output gguf ( #2635 )
...
* convert-new.py : output gguf (WIP)
* convert-new.py : add gguf key-value pairs
* llama : add hparams.ctx_train + no longer print ftype
* convert-new.py : minor fixes
* convert-new.py : vocab-only option should work now
* llama : fix tokenizer to use llama_char_to_byte
* tests : add new ggml-vocab-llama.gguf
* convert-new.py : tensor name mapping
* convert-new.py : add map for skipping tensor serialization
* convert-new.py : convert script now works
* gguf.py : pick some of the refactoring from #2644
* convert-new.py : minor fixes
2023-08-17 17:19:52 +03:00
Georgi Gerganov
c8ee87f141
gguf.py : merge all files in gguf.py
2023-08-16 19:55:49 +03:00
Georgi Gerganov
88b5769487
gguf : deduplicate ( #2629 )
...
* gguf : better type names
* dedup : CPU + Metal is working
* ggml : fix warnings about unused results
* llama.cpp : fix line feed and compiler warning
* llama : fix strncpy warning + note token_to_str does not write null
* llama : restore the original load/save session implementation
Will migrate this to GGUF in the future
* convert-llama-h5-to-gguf.py : support alt ctx param name
* ggml : assert when using ggml_mul with non-F32 src1
* examples : dedup simple
---------
Co-authored-by: klosax <131523366+klosax@users.noreply.github.com>
2023-08-16 19:25:29 +03:00
klosax
2ae0e985b3
convert-llama-7b-pth-to-gguf.py : add tensor data layout
2023-08-15 19:55:13 +02:00
klosax
ab2cbd03ca
convert-llama-7b-pth-to-gguf.py : add token types
2023-08-14 22:10:50 +02:00
klosax
6f64b6c0f8
Create convert-llama-7b-pth-to-gguf.py
2023-08-14 13:51:09 +02:00
