mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-09-23 05:26:19 +00:00
fc0c8d286a
This commit updates the surgery script to not remove the tensors from the model file. For this to work the `--skip-unknown` flag is added as an argument to the convert.py script in README.md. The motivation for this change is that the surgery script currently removes the projector tensors from the model file. If the model was checked out from a repository, the model file will have been updated and have to be checked out again to reset this effect. If this can be avoided I think it would be preferable. I did not perform this change for BakLLaVA models as I am not sure how that part works.
107 lines
4.5 KiB
Markdown
107 lines
4.5 KiB
Markdown
# LLaVA
|
|
|
|
Currently this implementation supports [llava-v1.5](https://huggingface.co/liuhaotian/llava-v1.5-7b) variants,
|
|
as well as llava-1.6 [llava-v1.6](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2) variants.
|
|
|
|
The pre-converted [7b](https://huggingface.co/mys/ggml_llava-v1.5-7b)
|
|
and [13b](https://huggingface.co/mys/ggml_llava-v1.5-13b)
|
|
models are available.
|
|
For llava-1.6 a variety of prepared gguf models are available as well [7b-34b](https://huggingface.co/cmp-nct/llava-1.6-gguf)
|
|
|
|
After API is confirmed, more models will be supported / uploaded.
|
|
|
|
## Usage
|
|
Build with cmake or run `make llava-cli` to build it.
|
|
|
|
After building, run: `./llava-cli` to see the usage. For example:
|
|
|
|
```sh
|
|
./llava-cli -m ../llava-v1.5-7b/ggml-model-f16.gguf --mmproj ../llava-v1.5-7b/mmproj-model-f16.gguf --image path/to/an/image.jpg
|
|
```
|
|
|
|
**note**: A lower temperature like 0.1 is recommended for better quality. add `--temp 0.1` to the command to do so.
|
|
**note**: For GPU offloading ensure to use the `-ngl` flag just like usual
|
|
|
|
## LLaVA 1.5
|
|
|
|
- Clone a LLaVA and a CLIP model ([available options](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md)). For example:
|
|
|
|
```sh
|
|
git clone https://huggingface.co/liuhaotian/llava-v1.5-7b
|
|
|
|
git clone https://huggingface.co/openai/clip-vit-large-patch14-336
|
|
```
|
|
|
|
2. Install the required Python packages:
|
|
|
|
```sh
|
|
pip install -r examples/llava/requirements.txt
|
|
```
|
|
|
|
3. Use `llava-surgery.py` to split the LLaVA model to LLaMA and multimodel projector constituents:
|
|
|
|
```sh
|
|
python ./examples/llava/llava-surgery.py -m ../llava-v1.5-7b
|
|
```
|
|
|
|
4. Use `convert-image-encoder-to-gguf.py` to convert the LLaVA image encoder to GGUF:
|
|
|
|
```sh
|
|
python ./examples/llava/convert-image-encoder-to-gguf.py -m ../clip-vit-large-patch14-336 --llava-projector ../llava-v1.5-7b/llava.projector --output-dir ../llava-v1.5-7b
|
|
```
|
|
|
|
5. Use `convert.py` to convert the LLaMA part of LLaVA to GGUF:
|
|
|
|
```sh
|
|
python ./convert.py ../llava-v1.5-7b --skip-unknown
|
|
```
|
|
|
|
Now both the LLaMA part and the image encoder is in the `llava-v1.5-7b` directory.
|
|
|
|
## LLaVA 1.6 gguf conversion
|
|
|
|
1) Backup your pth/safetensor model files as llava-surgery modifies them
|
|
2) Use `python llava-surgery-v2.py -C -m /path/to/hf-model` which also supports llava-1.5 variants pytorch as well as safetensor models:
|
|
- you will find a llava.projector and a llava.clip file in your model directory
|
|
3) Copy the llava.clip file into a subdirectory (like vit), rename it to pytorch_model.bin and add a fitting vit configuration to the directory (https://huggingface.co/cmp-nct/llava-1.6-gguf/blob/main/config_vit.json) and rename it to config.json.
|
|
4) Create the visual gguf model: `python ./examples/llava/convert-image-encoder-to-gguf.py -m ../path/to/vit --llava-projector ../path/to/llava.projector --output-dir ../path/to/output --clip-model-is-vision`
|
|
- This is similar to llava-1.5, the difference is that we tell the encoder that we are working with the pure vision model part of CLIP
|
|
5) Everything else as usual: convert.py the hf model, quantize as needed
|
|
**note** llava-1.6 needs more context than llava-1.5, at least 3000 is needed (just run it at -c 4096)
|
|
**note** llava-1.6 greatly benefits from batched prompt processing (defaults work)
|
|
|
|
## llava-cli templating and llava-1.6 prompting
|
|
|
|
llava-1.5 models all use the same vicuna prompt, here you can just add your image question like `-p "Provide a full description."`
|
|
For llava-1.5 models which are not vicuna (mistral and Yi) you need to adapt system prompt as well as user prompt, for this purpose llava-cli has a basic templating system:
|
|
|
|
**For Mistral and using llava-cli binary:**
|
|
Add this: `-p "<image>\nUSER:\nProvide a full description.\nASSISTANT:\n"`
|
|
The mistral template for llava-1.6 seems to be no system print and a USER/ASSISTANT role
|
|
|
|
**For the 34B this should work:**
|
|
Add this: `-e -p <|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nProvide a full description.<|im_end|><|im_start|>assistant\n`
|
|
|
|
|
|
## How to know if you are running in llava-1.5 or llava-1.6 mode
|
|
|
|
When running llava-cli you will see a visual information right before the prompt is being processed:
|
|
|
|
**Llava-1.5:**
|
|
`encode_image_with_clip: image embedding created: 576 tokens`
|
|
|
|
**Llava-1.6 (anything above 576):**
|
|
`encode_image_with_clip: image embedding created: 2880 tokens`
|
|
|
|
|
|
Alternatively just pay notice to how many "tokens" have been used for your prompt, it will also show 1000+ tokens for llava-1.6
|
|
|
|
|
|
|
|
|
|
## TODO
|
|
|
|
- [x] Support non-CPU backend for the image encoding part.
|
|
- [ ] Support different sampling methods.
|
|
- [ ] Support more model variants.
|