llama.cpp/examples/embd-input
Georgi Gerganov 0e89203b51
speculative : add tree-based sampling example (#3624)
* sampling : one sequence per sampling context

ggml-ci

* speculative : add tree-based sampling support

ggml-ci

* speculative : reuse the n_parallel CLI param

* speculative : refactor sampling

* examples : fix build after sampling refactoring

ggml-ci

* batched : fix n_seq_id

* sampling : fix malloc

ggml-ci

* swift : fix build

ggml-ci

* swift : try to fix build

ggml-ci

* prompts : add assistant.txt

* common : add llama_batch_add() and llama_batch_clear() helpers

* speculative : minor refactor

ggml-ci

* minor : comments + rename

ggml-ci

* speculative : fix off-by-one for n_drafted

* speculative : fix the n_drafted fix + p constants
2023-10-18 16:21:57 +03:00
..
.gitignore llama : support input embeddings directly (#1910) 2023-06-28 18:53:37 +03:00
CMakeLists.txt cmake : install targets (#2256) 2023-07-19 10:01:11 +03:00
embd_input.py chmod : make scripts executable (#2675) 2023-08-23 17:29:09 +03:00
embd-input-lib.cpp speculative : add tree-based sampling example (#3624) 2023-10-18 16:21:57 +03:00
embd-input-test.cpp llama.cpp : split llama_context_params into model and context params (#3301) 2023-09-28 22:42:38 +03:00
embd-input.h examples : add compiler version and target to build info (#2998) 2023-09-15 16:59:49 -04:00
llava.py chmod : make scripts executable (#2675) 2023-08-23 17:29:09 +03:00
minigpt4.py chmod : make scripts executable (#2675) 2023-08-23 17:29:09 +03:00
panda_gpt.py chmod : make scripts executable (#2675) 2023-08-23 17:29:09 +03:00
README.md examples : fixed path typos in embd-input (#2214) 2023-07-14 21:40:05 +03:00

Examples for input embedding directly

Requirement

build libembdinput.so run the following comman in main dir (../../).

make

LLaVA example (llava.py)

  1. Obtian LLaVA model (following https://github.com/haotian-liu/LLaVA/ , use https://huggingface.co/liuhaotian/LLaVA-13b-delta-v1-1/).
  2. Convert it to ggml format.
  3. llava_projection.pth is pytorch_model-00003-of-00003.bin.
import torch

bin_path = "../LLaVA-13b-delta-v1-1/pytorch_model-00003-of-00003.bin"
pth_path = "./examples/embd-input/llava_projection.pth"

dic = torch.load(bin_path)
used_key = ["model.mm_projector.weight","model.mm_projector.bias"]
torch.save({k: dic[k] for k in used_key}, pth_path)
  1. Check the path of LLaVA model and llava_projection.pth in llava.py.

PandaGPT example (panda_gpt.py)

  1. Obtian PandaGPT lora model from https://github.com/yxuansu/PandaGPT. Rename the file to adapter_model.bin. Use convert-lora-to-ggml.py to convert it to ggml format. The adapter_config.json is
{
  "peft_type": "LORA",
  "fan_in_fan_out": false,
  "bias": null,
  "modules_to_save": null,
  "r": 32,
  "lora_alpha": 32,
  "lora_dropout": 0.1,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"]
}
  1. Papare the vicuna v0 model.
  2. Obtain the ImageBind model.
  3. Clone the PandaGPT source.
git clone https://github.com/yxuansu/PandaGPT
  1. Install the requirement of PandaGPT.
  2. Check the path of PandaGPT source, ImageBind model, lora model and vicuna model in panda_gpt.py.

MiniGPT-4 example (minigpt4.py)

  1. Obtain MiniGPT-4 model from https://github.com/Vision-CAIR/MiniGPT-4/ and put it in embd-input.
  2. Clone the MiniGPT-4 source.
git clone https://github.com/Vision-CAIR/MiniGPT-4/
  1. Install the requirement of PandaGPT.
  2. Papare the vicuna v0 model.
  3. Check the path of MiniGPT-4 source, MiniGPT-4 model and vicuna model in minigpt4.py.