Commit Graph

2385 Commits

Author SHA1 Message Date
Georgi Gerganov
b3f460e941
Disable BLAS branch in mul_mat - seems there is a bug 2023-03-24 23:39:17 +02:00
Georgi Gerganov
04c6f5ed6f
Immediately start processing the prompt before user input has been provided (#476) 2023-03-24 23:17:58 +02:00
Georgi Gerganov
7a9b6c3a8b
Reduce memory usage and allocate enough memory for largest context (#473)
* Reduce memory usage and allocate enough memory for large contexts

* Simpler scratch buffer usage

* Reenable BLAS for quantized mul_mat

* Fix number of layers in 30B and 65B

* Fix KV cache size for F32
2023-03-24 23:17:37 +02:00
Georgi Gerganov
31572d9665
Temporary bump the memory buffer size - hopefully fix issues from 483bab2e 2023-03-24 18:23:56 +02:00
Gary Mulder
f4f5362edb
Update README.md (#444)
Added explicit **bolded** instructions clarifying that people need to request access to models from Facebook and never through through this repo.
2023-03-24 15:23:09 +00:00
rabidcopy
863f65e2e3
fix instruct mode (#445)
changes to EOS behavior in interactive and reverse prompt handling broke instruct mode by erroneously injecting instruct mode's reverse prompt and an extra newline.
2023-03-24 17:22:39 +02:00
Georgi Gerganov
afd220d9c6
Properly free llama_context on failure 2023-03-24 17:21:01 +02:00
Cameron Kaiser
481044d50c
additional optimizations for POWER9 (#454) 2023-03-24 17:19:26 +02:00
comex
563cdc391d
Support calling mlock() on loaded model data on Linux and macOS (#453)
* Support calling mlock() on loaded model data on Linux and macOS

This is enabled by a new --mlock command line option.

Using mlock() disables swapping and memory compression for the model
data.  Doing so can be useful on systems where the model takes up a
large fraction of system RAM.  In my experience, macOS is quite eager to
start compressing llama.cpp's memory, which then makes it halt for a few
seconds while it decompresses, even with a model that uses "only" 25GB
out of 32GB.

Of course, this comes at the cost of forcing the system to swap or
compress other processes' memory instead, so it needs to be used with
care and shouldn't be enabled by default.

In theory it should be possible to support this on Windows as well using
VirtualLock(), but I'm not much of a Windows user.

* Update llama.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-03-24 17:19:05 +02:00
Luciano
8d4a855c24
Add embedding mode with arg flag. Currently working (#282)
* working but ugly

* add arg flag, not working on embedding mode

* typo

* Working! Thanks to @nullhook

* make params argument instead of hardcoded boolean. remove useless time check

* start doing the instructions but not finished. This probably doesnt compile

* Embeddings extraction support

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-03-24 17:05:13 +02:00
Georgi Gerganov
b6b268d441
Add link to Roadmap discussion 2023-03-24 09:13:35 +02:00
Georgi Gerganov
3cd8dde0d1 Revert "Fix memory allocation issues and seg faults"
This reverts commit 4870e455b3.

Will provide the correct fix later
2023-03-24 06:22:28 +02:00
Georgi Gerganov
4870e455b3
Fix memory allocation issues and seg faults 2023-03-24 00:11:53 +02:00
Georgi Gerganov
483bab2e3d
Avoid the transposed X branch in the Z = X * Y matrix multiplication (#439)
Should make results reproducible for different number of threads and batch sizes
2023-03-23 23:22:01 +02:00
Jed Fox
404e1da38e
Fix quantize script not finding models in parent directory (#428) 2023-03-23 22:42:52 +02:00
Georgi Gerganov
4cc053b6d5
Remove oboslete command from Docker script 2023-03-23 22:39:44 +02:00
Georgi Gerganov
0ba5a3a9a5
Obsolete 2023-03-23 22:32:21 +02:00
rabidcopy
2e17dfd80a
Replace EOS with newline to prevent context/memory being flushed by EOS in interactive mode (#333)
* Improve interactive mode's coherence after EOS

Aims to improve coherence and ability to resume the interactive session when the user is given input back after an end of text token is reached.
Not sure what token 13 is or why it seems to help. See conversation for examples.

* Make newline token a constant

* dynamically determine newline token

* relocate previous newline token const

* cleanup whitespace

* print a new line on end of text in interactive

this may need to be looked into further when not using a reverse prompt

* only print manual newline with reverse prompt

fix formatting of reverse prompts so they don't end up at the end of the current line while not introducing unnecessary new lines otherwise

* alternate approach to replace end of text tokens

* Inject the reverse prompt again after eos in interactive mode

* tokenize reverse prompt when needed

makes this PR compatible with https://github.com/ggerganov/llama.cpp/pull/330

* tokenize and inject only first reverse prompt

thanks to tjohnman

* tokenize first reverse prompt once

* add newline token

* add newline token

* tokenize/inject reverse prompt for refactor

this doesn't seem right though

* tokenize nothing for antiprompt if no reverse

* Update main.cpp

* Update main.cpp

* tokenize and inject reverse prompt as needed

this doesn't seem to work if the reverse prompt is tokenized outside earlier on

* not needed

* remove newline token

* remove newline token

* tokenize newline token

* add space to comment

* Update main.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: Slaren <2141330+slaren@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-03-23 22:22:47 +02:00
Timmy Knight
20a1a4e09c
Fix GPTQ converter (#423)
* Fix GPTQ converter

* Fix comment

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-03-23 22:18:13 +02:00
nusu-github
ad072fc5ad
Generate library with CMake (#430)
* Generate library with CMake

BUILD_SHARED_LIBS to allow llama library to be generated.

* Turn ON PIC when BUILD_SHARED_LIBS is ON
2023-03-23 21:16:48 +01:00
anzz1
ea10d3ded2
Command line args bounds checking (#424)
* command line args bounds checking

* unknown and invalid param exit codes 0 -> 1
2023-03-23 19:54:28 +02:00
Ben Siraphob
a18c19259a Fix Nix build 2023-03-23 17:51:26 +01:00
Stephan Walter
a50e39c6fe
Revert "Delete SHA256SUMS for now" (#429)
* Revert "Delete SHA256SUMS for now (#416)"

This reverts commit 8eea5ae0e5.

* Remove ggml files until they can be verified
* Remove alpaca json
* Add also model/tokenizer.model to SHA256SUMS + update README

---------

Co-authored-by: Pavol Rusnak <pavol@rusnak.io>
2023-03-23 15:15:48 +01:00
Kerfuffle
a140219e81
Fix Makefile echo escape codes (by removing them). (#418) 2023-03-23 12:41:32 +01:00
Gary Mulder
8a3e5ef801
Move model section from issue template to README.md (#421)
* Update custom.md

* Removed Model section as it is better placed in README.md

* Updates to README.md model section

* Inserted text that was removed from  issue template about obtaining models from FB and links to papers describing the various models

* Removed IPF down links for the Alpaca 7B models as these look to be in the old data format and probably shouldn't be directly linked to, anyway

* Updated the perplexity section to point at Perplexity scores #406 discussion
2023-03-23 11:30:40 +00:00
anzz1
8eea5ae0e5
Delete SHA256SUMS for now (#416)
Delete this for now to avoid confusion since it contains some wrong checksums from the old tokenizer format
Re-add after #374 is resolved
2023-03-23 11:26:19 +01:00
Georgi Gerganov
93208cfb92
Adjust repetition penalty .. 2023-03-23 10:46:58 +02:00
Georgi Gerganov
03ace14cfd
Add link to recent podcast about whisper.cpp and llama.cpp 2023-03-23 09:48:51 +02:00
anzz1
e4412b45e3
CI: CMake: Separate build and test steps (#376)
* CI: Separate Build and Test steps (CMake)

* CI: Make sure build passes before running tests (CMake)

* CI: Standardise step id names
2023-03-23 04:20:34 +02:00
tjohnman
f7dc43bc0d
Fix instruct mode broken by PR #354 (#409)
Co-authored-by: Johnman <tjohnman@github>
2023-03-23 01:30:23 +01:00
Gary Mulder
ee8a788786
Update issue template so people will use it (#404) 2023-03-22 19:06:18 +00:00
Stephan Walter
69c92298a9
Deduplicate q4 quantization functions (#383)
* Deduplicate q4 quantization functions

* Use const; add basic test

* Re-enable quantization test

* Disable AVX2 flags in CI

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-03-22 19:29:06 +02:00
Valentyn Bezshapkin
97940520e8
fix: add POSIX functionality for Linux compilation (#51)
* fix: add POSIX functionality for Linux compilation

* fix: older standard for compatibility
2023-03-22 19:20:25 +02:00
tjohnman
305ba6f0e6
Don't force immediate interactive without -i (#354)
* Don't force immediate interactive without -i

Sometimes we might want to use a reverse prompt but we want to let the
model generate tokens right after the initial prompt. So we don't force
user input mode if the -i flag wasn't specified and instead let it run
until we encounter the reverse prompt.

This gives use some more flexibility, since it doesn't force the user to
enter a newline if they want to let the model generate text right after
the initial prompt and only be asked for input if the reverse prompt is
encountered.

The `--interactive-first` flag is reintroduced to force the old
behavior. `-r` behaves like `-i` plus introduces a reverse prompt (it
can be specified more than once).

* Update help output.

---------

Co-authored-by: Johnman <tjohnman@github>
2023-03-22 19:16:35 +02:00
Erik Scholz
4122dffff9
cmake: make llama an actual library (#392) 2023-03-22 18:37:10 +02:00
Erik Scholz
56e659a0b2
fix perplexity after c-api refactor (#390)
* preallocate a buffer of fitting size for tokenization (utils.cpp)

* don't create a new std::string (especially here, where it's usually large)
2023-03-22 18:09:38 +02:00
Gary Linscott
40ea807a97
Add details on perplexity to README.md (#395) 2023-03-22 08:53:54 -07:00
Yusuf Kağan Hanoğlu
d5850c53ca
Add missing header for memcpy (#386)
fixed: memcpy is not defined
2023-03-22 10:55:45 +02:00
Georgi Gerganov
ae44e23ee3
When seed <= 0 - use the clock to generate one 2023-03-22 07:47:15 +02:00
Georgi Gerganov
928480ef5b
Init llama_context_params properly from CLI (#370) 2023-03-22 07:45:14 +02:00
Georgi Gerganov
56817b1f88
Remove temporary notice and update hot topics 2023-03-22 07:34:02 +02:00
Georgi Gerganov
f5a77a629b
Introduce C-style API (#370)
* Major refactoring - introduce C-style API

* Clean up

* Add <cassert>

* Add <iterator>

* Add <algorithm> ....

* Fix timing reporting and accumulation

* Measure eval time only for single-token calls

* Change llama_tokenize return meaning
2023-03-22 07:32:36 +02:00
Gary Mulder
da0e9fe90c Add SHA256SUMS file and instructions to README how to obtain and verify the downloads
Hashes created using:

sha256sum models/*B/*.pth models/*[7136]B/ggml-model-f16.bin* models/*[7136]B/ggml-model-q4_0.bin* > SHA256SUMS
2023-03-21 23:19:11 +01:00
anzz1
e6c9e0986c Fix bin dir for win ci 2023-03-22 00:01:08 +02:00
Erik Scholz
01a297b099
specify build type for ctest on windows (#371) 2023-03-21 23:34:25 +02:00
Georgi Gerganov
3366853e41
Add notice about pending change 2023-03-21 22:57:35 +02:00
Mathieu Nayrolles
3f9c6135e4
fix typo in chatLLaMa (#368)
The prompt contains a typo where 'alound' is used instead of 'aloud'.
2023-03-21 22:52:27 +02:00
Georgi Gerganov
0f61352708
Update issue templates 2023-03-21 19:47:27 +02:00
Fabio R. Sluzala
353ec251a4
We could use std::unordered_map over std::map (#305)
* Improve performance by changing std::map to std::unordered_map and std::map<id, token> id_to_token; to std::vector<token> id_to_token;

* fix last commit on gpt_vocab_init add vocab.id_to_token.resize(vocab.token_to_id.size());

* Removed include <map>

* Nest struct token score inside gpt_vocab

* renamed token to tok
2023-03-21 19:21:50 +02:00
Matvey Soloviev
89d5d90f3b
Fix color codes emitting mid-UTF8 code. (#312) 2023-03-21 19:11:01 +02:00