mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-12-25 10:54:36 +00:00
0c4d489e29
* imatrix: save the dataset file used in the output file * llama: support kv overrides type string string * common: factorize KV Overrides parsing between common and server * quantize: add imatrix n entries and dataset KV metadata quantize: factorize KV Overrides parsing between common #6656 * llama: remove kv override str_value initialization as it does not compile on some toolchain * quantize: add imatrix m_last_call as `quantize.imatrix.chunks_count` * quantize: add imatrix filename in KV * llama: add llama_model_kv_override_free * common: add llama_model_kv_override_free common: free kv override if used after model loading * llama: finally move the string KV override value to the stack * llama : minor * no need to add a NUL to the std::vector, std::string can be initialized from a pair of iterators. Co-authored-by: slaren <slarengh@gmail.com> * kv override: ensure string termination --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: slaren <slarengh@gmail.com> |
||
---|---|---|
.. | ||
CMakeLists.txt | ||
imatrix.cpp | ||
README.md |
llama.cpp/examples/imatrix
Compute an importance matrix for a model and given text dataset. Can be used during quantization to enchance the quality of the quantum models. More information is available here: https://github.com/ggerganov/llama.cpp/pull/4861
Usage
./imatrix -m <some_fp_model> -f <some_training_data> [-o <output_file>] [--verbosity <verbosity_level>]
[-ofreq num_chunks] [-ow <0 or 1>] [other common params]
Here -m
with a model name and -f
with a file containing training data (such as e.g. wiki.train.raw
) are mandatory.
The parameters in square brackets are optional and have the following meaning:
-o
(or--output-file
) specifies the name of the file where the computed data will be stored. If missingimatrix.dat
is used.--verbosity
specifies the verbosity level. If set to0
, no output other than the perplexity of the processed chunks will be generated. If set to1
, each time the results are saved a message is written tostderr
. If>=2
, a message is output each time data is collected for any tensor. Default verbosity level is1
.-ofreq
(or--output-frequency
) specifies how often the so far computed result is saved to disk. Default is 10 (i.e., every 10 chunks)-ow
(or--output-weight
) specifies if data will be collected for theoutput.weight
tensor. My experience is that it is better to not utilize the importance matrix when quantizingoutput.weight
, so this is set tofalse
by default.
For faster computation, make sure to use GPU offloading via the -ngl
argument
Example
LLAMA_CUDA=1 make -j
# generate importance matrix (imatrix.dat)
./imatrix -m ggml-model-f16.gguf -f train-data.txt -ngl 99
# use the imatrix to perform a Q4_K_M quantization
./quantize --imatrix imatrix.dat ggml-model-f16.gguf ./ggml-model-q4_k_m.gguf q4_k_m