mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-12-25 19:04:35 +00:00
144 lines
6.2 KiB
Markdown
144 lines
6.2 KiB
Markdown
# llama.cpp
|
||
|
||
Inference of [Facebook's LLaMA](https://github.com/facebookresearch/llama) model in pure C/C++
|
||
|
||
**TEMPORARY NOTICE:**
|
||
If you observe garbage results, make sure to update to latest master. There was a bug and it was fixed here: https://github.com/ggerganov/llama.cpp/commit/70bc0b8b15b98dca23b28f0c8f5e34b27e424cda
|
||
|
||
## Description
|
||
|
||
The main goal is to run the model using 4-bit quantization on a MacBook.
|
||
|
||
- Plain C/C++ implementation without dependencies
|
||
- Apple silicon first-class citizen - optimized via Arm Neon and Accelerate framework
|
||
- Mixed F16 / F32 precision
|
||
- 4-bit quantization support
|
||
- Runs on the CPU
|
||
|
||
This was hacked in an evening - I have no idea if it works correctly.
|
||
|
||
So far, I've tested just the 7B model.
|
||
Here is a typical run:
|
||
|
||
```java
|
||
make -j && ./main -m ../LLaMA-4bit/7B/ggml-model-q4_0.bin -p "Building a website can be done in 10 simple steps:" -t 8 -n 512
|
||
I llama.cpp build info:
|
||
I UNAME_S: Darwin
|
||
I UNAME_P: arm
|
||
I UNAME_M: arm64
|
||
I CFLAGS: -I. -O3 -DNDEBUG -std=c11 -fPIC -pthread -DGGML_USE_ACCELERATE
|
||
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread
|
||
I LDFLAGS: -framework Accelerate
|
||
I CC: Apple clang version 14.0.0 (clang-1400.0.29.202)
|
||
I CXX: Apple clang version 14.0.0 (clang-1400.0.29.202)
|
||
|
||
make: Nothing to be done for `default'.
|
||
main: seed = 1678486056
|
||
llama_model_load: loading model from '../LLaMA-4bit/7B/ggml-model-q4_0.bin' - please wait ...
|
||
llama_model_load: n_vocab = 32000
|
||
llama_model_load: n_ctx = 512
|
||
llama_model_load: n_embd = 4096
|
||
llama_model_load: n_mult = 256
|
||
llama_model_load: n_head = 32
|
||
llama_model_load: n_layer = 32
|
||
llama_model_load: n_rot = 128
|
||
llama_model_load: f16 = 2
|
||
llama_model_load: n_ff = 11008
|
||
llama_model_load: ggml ctx size = 4529.34 MB
|
||
llama_model_load: memory_size = 512.00 MB, n_mem = 16384
|
||
llama_model_load: .................................... done
|
||
llama_model_load: model size = 4017.27 MB / num tensors = 291
|
||
|
||
main: prompt: 'Building a website can be done in 10 simple steps:'
|
||
main: number of tokens in prompt = 15
|
||
1 -> ''
|
||
8893 -> 'Build'
|
||
292 -> 'ing'
|
||
263 -> ' a'
|
||
4700 -> ' website'
|
||
508 -> ' can'
|
||
367 -> ' be'
|
||
2309 -> ' done'
|
||
297 -> ' in'
|
||
29871 -> ' '
|
||
29896 -> '1'
|
||
29900 -> '0'
|
||
2560 -> ' simple'
|
||
6576 -> ' steps'
|
||
29901 -> ':'
|
||
|
||
sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000
|
||
|
||
|
||
Building a website can be done in 10 simple steps:
|
||
1) Select a domain name and web hosting plan
|
||
2) Complete a sitemap
|
||
3) List your products
|
||
4) Write product descriptions
|
||
5) Create a user account
|
||
6) Build the template
|
||
7) Start building the website
|
||
8) Advertise the website
|
||
9) Provide email support
|
||
10) Submit the website to search engines
|
||
A website is a collection of web pages that are formatted with HTML. HTML is the code that defines what the website looks like and how it behaves.
|
||
The HTML code is formatted into a template or a format. Once this is done, it is displayed on the user's browser.
|
||
The web pages are stored in a web server. The web server is also called a host. When the website is accessed, it is retrieved from the server and displayed on the user's computer.
|
||
A website is known as a website when it is hosted. This means that it is displayed on a host. The host is usually a web server.
|
||
A website can be displayed on different browsers. The browsers are basically the software that renders the website on the user's screen.
|
||
A website can also be viewed on different devices such as desktops, tablets and smartphones.
|
||
Hence, to have a website displayed on a browser, the website must be hosted.
|
||
A domain name is an address of a website. It is the name of the website.
|
||
The website is known as a website when it is hosted. This means that it is displayed on a host. The host is usually a web server.
|
||
A website can be displayed on different browsers. The browsers are basically the software that renders the website on the user’s screen.
|
||
A website can also be viewed on different devices such as desktops, tablets and smartphones. Hence, to have a website displayed on a browser, the website must be hosted.
|
||
A domain name is an address of a website. It is the name of the website.
|
||
A website is an address of a website. It is a collection of web pages that are formatted with HTML. HTML is the code that defines what the website looks like and how it behaves.
|
||
The HTML code is formatted into a template or a format. Once this is done, it is displayed on the user’s browser.
|
||
A website is known as a website when it is hosted
|
||
|
||
main: mem per token = 14434244 bytes
|
||
main: load time = 1332.48 ms
|
||
main: sample time = 1081.40 ms
|
||
main: predict time = 31378.77 ms / 61.41 ms per token
|
||
main: total time = 34036.74 ms
|
||
```
|
||
|
||
And here is another demo of running both LLaMA-7B and [whisper.cpp](https://github.com/ggerganov/whisper.cpp) on a single M1 Pro MacBook:
|
||
|
||
https://user-images.githubusercontent.com/1991296/224442907-7693d4be-acaa-4e01-8b4f-add84093ffff.mp4
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# build this repo
|
||
git clone https://github.com/ggerganov/llama.cpp
|
||
cd llama.cpp
|
||
make
|
||
|
||
# obtain the original LLaMA model weights and place them in ./models
|
||
ls ./models
|
||
65B 30B 13B 7B tokenizer_checklist.chk tokenizer.model
|
||
|
||
# convert the 7B model to ggml FP16 format
|
||
python3 convert-pth-to-ggml.py models/7B/ 1
|
||
|
||
# quantize the model to 4-bits
|
||
./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2
|
||
|
||
# run the inference
|
||
./main -m ./models/7B/ggml-model-q4_0.bin -t 8 -n 128
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook
|
||
- Not sure if my tokenizer is correct. There are a few places where we might have a mistake:
|
||
- https://github.com/ggerganov/llama.cpp/blob/26c084662903ddaca19bef982831bfb0856e8257/convert-pth-to-ggml.py#L79-L87
|
||
- https://github.com/ggerganov/llama.cpp/blob/26c084662903ddaca19bef982831bfb0856e8257/utils.h#L65-L69
|
||
In general, it seems to work, but I think it fails for unicode character support. Hopefully, someone can help with that
|
||
- I don't know yet how much the quantization affects the quality of the generated text
|
||
- Probably the token sampling can be improved
|
||
- x86 quantization support [not yet ready](https://github.com/ggerganov/ggml/pull/27). Basically, you want to run this on Apple Silicon
|
||
|