980 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
klosax	49380a23a3	gguf.py : accumulate kv and tensor info data + special tokens	2023-08-01 23:37:48 +02:00
klosax	ff1cb02397	constants.py : special tokens	2023-08-01 23:17:21 +02:00
klosax	36a36c32a3	Update gptneox-main.cpp	2023-08-01 14:44:28 +02:00
klosax	c77fabb1f9	gptneox-main.cpp : special tokens	2023-08-01 14:32:53 +02:00
klosax	e7a741695c	convert-gptneox-h5-to-gguf.py : Special tokens	2023-08-01 14:30:00 +02:00
klosax	da4900e835	Update convert-llama-h5-to-gguf.py	2023-07-31 23:04:03 +02:00
M. Yusuf Sarıgöz	f3de876a12	fix : update convert-llama-h5-to-gguf.py	2023-07-31 23:58:29 +03:00
M. Yusuf Sarıgöz	bb42aefaeb	gguf : mmap tensor data example	2023-07-31 17:46:12 +03:00
M. Yusuf Sarıgöz	b26f5b2e43	gguf : fix typo in function call	2023-07-31 16:23:54 +03:00
M. Yusuf Sarıgöz	7aa0a0e7f7	gguf : support custom alignment value	2023-07-31 09:59:36 +03:00
klosax	6b3a7b9f4f	Update convert-llama-h5-to-gguf.py	2023-07-31 03:02:00 +02:00
klosax	4f5b6224be	Update convert-gptneox-h5-to-gguf.py	2023-07-31 03:00:20 +02:00
klosax	2a0914673c	Update convert-gptneox-h5-to-gguf.py	2023-07-30 17:31:11 +02:00
klosax	068a8e0fbe	Update convert-llama-h5-to-gguf.py	2023-07-30 17:29:56 +02:00
klosax	30c4ea47e6	add gptneox gguf example	2023-07-30 16:59:26 +02:00
klosax	2fabc176ce	Update convert-llama-h5-to-gguf.py	2023-07-30 16:28:08 +02:00
klosax	f175b05872	Makefile : add gptneox gguf example	2023-07-30 15:08:37 +02:00
klosax	e9192b0135	add gptneox gguf example	2023-07-30 15:05:37 +02:00
klosax	4ed98bf1ab	Update convert-llama-h5-to-gguf.py	2023-07-30 15:01:47 +02:00
klosax	b19c11750b	ggml.c : add gguf_get_arr_n	2023-07-30 14:58:50 +02:00
klosax	b4676ee447	ggml.h : increase GGML_MAX_NAME to 64	2023-07-30 14:51:37 +02:00
klosax	ccd81a751b	gguf.py : add layer norm eps and merges	2023-07-30 14:48:14 +02:00
klosax	0790c121aa	constants.py : add layer norm eps	2023-07-30 14:46:36 +02:00
M. Yusuf Sarıgöz	87c34e4dd4	gguf : update convert-llama-h5-to-gguf.py	2023-07-30 01:09:22 +03:00
M. Yusuf Sarıgöz	32e037ffbe	gguf : fix set is not subscriptable	2023-07-30 01:01:13 +03:00
klosax	06c3e4a1a7	Update convert-llama-h5-to-gguf.py	2023-07-29 21:38:01 +02:00
klosax	9577821487	gguf.py : support any type	2023-07-29 21:29:07 +02:00
klosax	2c22e3bcdb	ggml.c : get arr str and f32	2023-07-29 20:37:47 +02:00
klosax	34469b9ea7	ggml.h : get array str and f32	2023-07-29 20:36:06 +02:00
M. Yusuf Sarıgöz	0f5e57f01d	gguf : handle already encoded string	2023-07-29 19:56:06 +03:00
klosax	8ad7cd49fb	Update convert-llama-h5-to-gguf.py	2023-07-29 16:47:00 +02:00
M. Yusuf Sarıgöz	0317c41d98	gguf : upd gguf conversion script	2023-07-29 13:31:07 +03:00
M. Yusuf Sarıgöz	cc3dd7f042	gguf : write tokenizer data	2023-07-29 13:30:22 +03:00
M. Yusuf Sarıgöz	8a76dd8a85	gguf : write tensors one by one	2023-07-29 13:17:28 +03:00
M. Yusuf Sarıgöz	c861e234f4	gguf : write tensors one by one	2023-07-29 12:49:01 +03:00
M. Yusuf Sarıgöz	0c219fb5b5	gguf : fix writing gguf arrays	2023-07-29 12:42:54 +03:00
M. Yusuf Sarıgöz	93f7f7aef7	gguf : write tensors one by one and code reuse	2023-07-29 12:34:35 +03:00
M. Yusuf Sarıgöz	aa99562d70	Merge branch 'gguf' of https://github.com//ggerganov/llama.cpp into gguf	2023-07-29 12:26:11 +03:00
M. Yusuf Sarıgöz	ea5f9ad2ca	gguf : fix writing gguf arrays	2023-07-29 12:25:43 +03:00
klosax	999431c4b6	quick and dirty conversion example	2023-07-29 11:20:05 +02:00
M. Yusuf Sarıgöz	d54f53ca51	gguf : add tokenization constants	2023-07-29 12:04:45 +03:00
M. Yusuf Sarıgöz	06f423a8e1	gguf : write sample tensors to read	2023-07-29 10:26:26 +03:00
M. Yusuf Sarıgöz	08dc8fd884	gguf : do not hardcode tensor names to read	2023-07-29 10:24:46 +03:00
M. Yusuf Sarıgöz	9475cdb7a3	Merge branch 'gguf-write-tokenization' into gguf	2023-07-29 00:36:35 +03:00
M. Yusuf Sarıgöz	1495735aac	gguf : fix writing tensors	2023-07-29 00:26:22 +03:00
klosax	3492f848d7	gguf : add gguf_find_key (#2438) ... * gguf.cpp : find key example * ggml.h : add gguf_find_key * ggml.c : add gguf_find_key	2023-07-28 23:45:24 +03:00
M. Yusuf Sarıgöz	11ef380c2a	GGUF : write tensor (#2426) ... * WIP: Write tensor * GGUF : Support writing tensors in Python * refactor : rm unused import and upd todos * fix : fix errors upd writing example * rm example.gguf * gitignore *.gguf * undo formatting	2023-07-28 11:34:16 +03:00
Georgi Gerganov	d2bb3ac10b	convert.py : remove GGML vocab + other obsolete stuff	2023-07-27 16:36:35 +03:00
Georgi Gerganov	68f53485e4	convert.py : start a new simplified implementation by removing old stuff	2023-07-27 15:56:53 +03:00
Georgi Gerganov	158be8f7f4	gguf.py : some code style changes	2023-07-27 15:37:06 +03:00