klosax
278ada9572
gguf.py : bytesarray for gpt2bpe tokenizer
2023-08-04 04:07:57 +02:00
klosax
49380a23a3
gguf.py : accumulate kv and tensor info data + special tokens
2023-08-01 23:37:48 +02:00
M. Yusuf Sarıgöz
7aa0a0e7f7
gguf : support custom alignment value
2023-07-31 09:59:36 +03:00
klosax
ccd81a751b
gguf.py : add layer norm eps and merges
2023-07-30 14:48:14 +02:00
M. Yusuf Sarıgöz
32e037ffbe
gguf : fix set is not subscriptable
2023-07-30 01:01:13 +03:00
klosax
9577821487
gguf.py : support any type
2023-07-29 21:29:07 +02:00
M. Yusuf Sarıgöz
0f5e57f01d
gguf : handle already encoded string
2023-07-29 19:56:06 +03:00
M. Yusuf Sarıgöz
cc3dd7f042
gguf : write tokenizer data
2023-07-29 13:30:22 +03:00
M. Yusuf Sarıgöz
8a76dd8a85
gguf : write tensors one by one
2023-07-29 13:17:28 +03:00
M. Yusuf Sarıgöz
c861e234f4
gguf : write tensors one by one
2023-07-29 12:49:01 +03:00
M. Yusuf Sarıgöz
0c219fb5b5
gguf : fix writing gguf arrays
2023-07-29 12:42:54 +03:00
M. Yusuf Sarıgöz
93f7f7aef7
gguf : write tensors one by one and code reuse
2023-07-29 12:34:35 +03:00
M. Yusuf Sarıgöz
ea5f9ad2ca
gguf : fix writing gguf arrays
2023-07-29 12:25:43 +03:00
M. Yusuf Sarıgöz
06f423a8e1
gguf : write sample tensors to read
2023-07-29 10:26:26 +03:00
M. Yusuf Sarıgöz
1495735aac
gguf : fix writing tensors
2023-07-29 00:26:22 +03:00
M. Yusuf Sarıgöz
11ef380c2a
GGUF : write tensor ( #2426 )
...
* WIP: Write tensor
* GGUF : Support writing tensors in Python
* refactor : rm unused import and upd todos
* fix : fix errors upd writing example
* rm example.gguf
* gitignore *.gguf
* undo formatting
2023-07-28 11:34:16 +03:00
Georgi Gerganov
158be8f7f4
gguf.py : some code style changes
2023-07-27 15:37:06 +03:00
M. Yusuf Sarıgöz
c85d3178b3
refactor : reduce code duplication and better API ( #2415 )
2023-07-27 10:29:29 +03:00
M. Yusuf Sarıgöz
bae6b125f6
wip : implement GGUF ( #2397 )
...
* Add LLAMA_DEFAULT_RMS_EPS so we can change the default (#2384 )
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
* WIP: python class to write GGUF, incomplete C apı for reading
---------
Co-authored-by: Kawrakow <48489457+ikawrakow@users.noreply.github.com>
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
2023-07-26 18:21:13 +03:00
