mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-09-23 13:36:20 +00:00
f98eb31c51
* convert-hf : begin refactoring write_tensor * convert : upgrade to sentencepiece v0.2.0 * convert-hf : remove unused n_dims in extra_*_tensors * convert-hf : simplify MoE weights stacking * convert-hf : flake8 linter doesn't like semicolons * convert-hf : allow unusual model part names For example, loading `model-00001-of-00001.safetensors` now works. * convert-hf : fix stacking MoE expert tensors `torch.stack` and `torch.cat` don't do the same thing. * convert-hf : fix Mamba conversion Tested to work even with a SentencePiece-based tokenizer. * convert : use a string for the SentencePiece tokenizer path * convert-hf : display tensor shape * convert-hf : convert norms to f32 by default * convert-hf : sort model part names `os.listdir` is said to list files in arbitrary order. Sorting the file names should let "model-00009-of-00042.safetensors" be loaded before "model-00010-of-00042.safetensors". * convert-hf : use an ABC for Model again It seems Protocol can't be used as a statically type-checked ABC, because its subclasses also can't be instantiated. (why did it seem to work?) At least there's still a way to throw an error when forgetting to define the `model_arch` property of any registered Model subclasses. * convert-hf : use a plain class for Model, and forbid direct instantiation There are no abstract methods used anyway, so using ABC isn't really necessary. * convert-hf : more consistent formatting of cmdline args * convert-hf : align the message logged for converted tensors * convert-hf : fix Refact conversion * convert-hf : save memory with lazy evaluation * convert-hf : flake8 doesn't like lowercase L as a variable name * convert-hf : remove einops requirement for InternLM2 * convert-hf : faster model parts loading Instead of pre-loading them all into a dict, iterate on the tensors in the model parts progressively as needed in Model.write_tensors Conversion for some architectures relies on checking for the presence of specific tensor names, so for multi-part models, the weight map is read from the relevant json file to quickly get these names up-front. * convert-hf : minor changes for consistency * gguf-py : add tqdm as a dependency It's small, and used for a progress bar in GGUFWriter.write_tensors_to_file |
||
|---|---|---|
| .. | ||
| features | ||
| README.md | ||
| requirements.txt | ||
| tests.sh | ||
Server tests
Python based server tests scenario using BDD and behave:
- issues.feature Pending issues scenario
- parallel.feature Scenario involving multi slots and concurrent requests
- security.feature Security, CORS and API Key
- server.feature Server base scenario: completion, embedding, tokenization, etc...
Tests target GitHub workflows job runners with 4 vCPU.
Requests are using aiohttp, asyncio based http client.
Note: If the host architecture inference speed is faster than GitHub runners one, parallel scenario may randomly fail.
To mitigate it, you can increase values in n_predict, kv_size.
Install dependencies
pip install -r requirements.txt
Run tests
- Build the server
cd ../../..
mkdir build
cd build
cmake -DLLAMA_CURL=ON ../
cmake --build . --target server
- Start the test:
./tests.sh
It's possible to override some scenario steps values with environment variables:
| variable | description |
|---|---|
PORT |
context.server_port to set the listening port of the server during scenario, default: 8080 |
LLAMA_SERVER_BIN_PATH |
to change the server binary path, default: ../../../build/bin/server |
DEBUG |
"ON" to enable steps and server verbose mode --verbose |
SERVER_LOG_FORMAT_JSON |
if set switch server logs to json format |
N_GPU_LAYERS |
number of model layers to offload to VRAM -ngl --n-gpu-layers |
Run @bug, @wip or @wrong_usage annotated scenario
Feature or Scenario must be annotated with @llama.cpp to be included in the default scope.
@bugannotation aims to link a scenario with a GitHub issue.@wrong_usageare meant to show user issue that are actually an expected behavior@wipto focus on a scenario working in progress@slowheavy test, disabled by default
To run a scenario annotated with @bug, start:
DEBUG=ON ./tests.sh --no-skipped --tags bug --stop
After changing logic in steps.py, ensure that @bug and @wrong_usage scenario are updated.
./tests.sh --no-skipped --tags bug,wrong_usage || echo "should failed but compile"