mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-09-23 13:36:20 +00:00
![]() * imatrix: save the dataset file used in the output file * llama: support kv overrides type string string * common: factorize KV Overrides parsing between common and server * quantize: add imatrix n entries and dataset KV metadata quantize: factorize KV Overrides parsing between common #6656 * llama: remove kv override str_value initialization as it does not compile on some toolchain * quantize: add imatrix m_last_call as `quantize.imatrix.chunks_count` * quantize: add imatrix filename in KV * llama: add llama_model_kv_override_free * common: add llama_model_kv_override_free common: free kv override if used after model loading * llama: finally move the string KV override value to the stack * llama : minor * no need to add a NUL to the std::vector, std::string can be initialized from a pair of iterators. Co-authored-by: slaren <slarengh@gmail.com> * kv override: ensure string termination --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: slaren <slarengh@gmail.com> |
||
---|---|---|
.. | ||
CMakeLists.txt | ||
quantize.cpp | ||
README.md | ||
tests.sh |
quantize
TODO
Llama 2 7B
Quantization | Bits per Weight (BPW) |
---|---|
Q2_K | 3.35 |
Q3_K_S | 3.50 |
Q3_K_M | 3.91 |
Q3_K_L | 4.27 |
Q4_K_S | 4.58 |
Q4_K_M | 4.84 |
Q5_K_S | 5.52 |
Q5_K_M | 5.68 |
Q6_K | 6.56 |
Llama 2 13B
Quantization | Bits per Weight (BPW) |
---|---|
Q2_K | 3.34 |
Q3_K_S | 3.48 |
Q3_K_M | 3.89 |
Q3_K_L | 4.26 |
Q4_K_S | 4.56 |
Q4_K_M | 4.83 |
Q5_K_S | 5.51 |
Q5_K_M | 5.67 |
Q6_K | 6.56 |
Llama 2 70B
Quantization | Bits per Weight (BPW) |
---|---|
Q2_K | 3.40 |
Q3_K_S | 3.47 |
Q3_K_M | 3.85 |
Q3_K_L | 4.19 |
Q4_K_S | 4.53 |
Q4_K_M | 4.80 |
Q5_K_S | 5.50 |
Q5_K_M | 5.65 |
Q6_K | 6.56 |