370 Commits

Author	SHA1	Message	Date
Olivier Chafik	db4bf93812	Merge remote-tracking branch 'origin/master' into tool-call	2024-10-22 14:37:30 +01:00
Olivier Chafik	7f2429e6b0	tool-calls: fix grammar regression	2024-10-22 11:49:50 +01:00
ochafik	9f5ab97756	tool-calls: add generic tool call style as default	2024-10-22 10:53:21 +01:00
ochafik	fa8462ffd3	fix root	2024-10-22 10:53:01 +01:00
ochafik	75764871e6	tool-call: fix grammar roots	2024-10-22 10:50:52 +01:00
Daniel Bevenius	674804a996	arg : fix typo in embeddings argument help [no ci] (#9994) ... This commit fixes two typos in the help text for the `--embd-normalize` and `--embd-separator` arguments. It also updates common.h which contain the same typo in two comments.	2024-10-22 10:40:02 +03:00
Daniel Bevenius	94008cc760	arg : fix attention non-causal arg value hint (#9985) ... This commit updates the argument value hint for the `--attention` argument to `non-causal`. The motivation for this change is that the only values for this argument are `causal` and `non-causal`.	2024-10-21 21:12:52 +03:00
Georgi Gerganov	f594bc80ba	ggml : add asserts for type conversion in fattn kernels (#9971) ... ggml-ci	2024-10-21 16:20:46 +03:00
Georgi Gerganov	55e47786e3	llama : default sampling changes + greedy update (#9897) ... * llama : deprecate softmax sampler + fix dist sampler ggml-ci * tests : replace macros with functions ggml-ci * sampling : change temperature sampler logic For t <= 0.0f, keep the max logit intact and set the rest to -inf * cont : no need for special "greedy" logic top-k == 1 is the same * tests : init prob correctly * llama : handle temp <= 0.0 in the temp_ext sampler too ggml-ci * cont : avoid extra loop in temperature sampler for sub-zero temp ggml-ci	2024-10-21 09:46:40 +03:00
Xuan Son Nguyen	cda0e4b648	llama : remove all_pos_0, all_pos_1, all_seq_id from llama_batch (#9745) ... * refactor llama_batch_get_one * adapt all examples * fix simple.cpp * fix llama_bench * fix * fix context shifting * free batch before return * use common_batch_add, reuse llama_batch in loop * null terminated seq_id list * fix save-load-state example * fix perplexity * correct token pos in llama_batch_allocr	2024-10-18 23:18:01 +02:00
Joe Eli McIlvain	66c2c93082	grammar : fix JSON Schema for string regex with top-level alt. (#9903) ... Prior to this commit, using a JSON Schema containing a string with `pattern` regular expression that uses top-level alternation (e.g. `"pattern": "^A|B|C|D$"`) would result in invalid JSON output from the constrained sampling grammar, because it ended up creating a grammar rule like this for the string: ``` thing ::= "\"" "A" | "B" | "C" | "D" "\"" space ``` Note that this rule will only match a starting quote for the "A" case, and will only match an ending quote for the "D" case, so this rule will always produce invalid JSON when used for sampling (that is, the JSON will always be lacking the starting quote, the ending quote, or both). This was fixed in a simple way by adding parentheses to the generated rule (for all string pattern rules, to keep it simple), such that the new generated rule looks like this (correct): ``` thing ::= "\"" ("A" | "B" | "C" | "D") "\"" space ```	2024-10-16 19:03:24 +03:00
Georgi Gerganov	755a9b2bf0	llama : add infill sampler (#9896) ... ggml-ci	2024-10-15 16:35:33 +03:00
MaggotHATE	fbc98b748e	sampling : add XTC sampler (#9742) ... * Initial XTC commit Adds XTC sampler, not activated by default, but recommended settings by default. * Cleanup * Simplified chances calculation To be more inline with the original implementation, chance is calculated once at the beginning. * First fixes by comments Still need to look into sorting * Fixed trailing backspaces * Fixed RNG to be reproduceable Thanks to @slaren for directions * Fixed forgotten header * Moved `min_keep` Moved from conditions to a simple check at the end. * Fixed broken randomization Thanks to @slaren for explanation * Swapped sorting for a custom algorithm Shifts tokens to remove the penalized ones, then puts the penalized at the back. Should make `min_keep` still viable. * Algorithm rework 1. Scan token from top till the first non-penalizable 2. Remove the last captured token (the least probable above threshold) 3. Shift all tokens to override the remaining penalizable 4. Penalize and put them at the the bottom. * Added XTC to `test-sampling` * Simplified algorithm and more tests * Updated info in common and args * Merged back lost commits in common and arg * Update dump info in common * Fixed incorrect min_keep check * Added XTC to README * Renamed parameters, fixed info and defaults * probability is at 0 by default, but XTC is included in sampling queue * threshold higher than 0.5 switches XTC off * Initial server support * Added XTC to server UIs * Fixed labels in old server UI * Made algorithm safer and more readable * Removed xtc_threshold_max * Fixed arg after update * Quick fixes by comments * Simplified algorithm since threshold_max is removed * Renamed random distribution * Fixed tests and outdated README * Small fixes	2024-10-15 12:54:55 +02:00
Georgi Gerganov	c7181bd294	server : reuse cached context chunks (#9866) ... ggml-ci	2024-10-13 18:52:48 +03:00
Georgi Gerganov	1bde94dd02	server : remove self-extend features (#9860) ... * server : remove self-extend ggml-ci * server : fix context limit check to use slot.n_past ggml-ci	2024-10-12 16:06:31 +03:00
Georgi Gerganov	95c76e8e92	server : remove legacy system_prompt feature (#9857) ... * server : remove legacy system_prompt feature ggml-ci * readme : update [no ci] * server : fix non-transformer logic + remove response from /props	2024-10-12 14:51:54 +03:00
Georgi Gerganov	11ac9800af	llama : improve infill support and special token detection (#9798) ... * llama : improve infill support ggml-ci * llama : add more FIM token strings ggml-ci * server : update prompt on slot restore (#9800) * gguf : deprecate old FIM token KVs	2024-10-12 08:21:51 +03:00
Diego Devesa	7eee341bee	common : use common_ prefix for common library functions (#9805) ... * common : use common_ prefix for common library functions --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-10-10 22:57:42 +02:00
Diego Devesa	0e9f760eb1	rpc : add backend registry / device interfaces (#9812) ... * rpc : add backend registry / device interfaces * llama : add llama_supports_rpc API * ggml_backend_rpc_start_rpc_server -> ggml_backend_rpc_start_server	2024-10-10 20:14:55 +02:00
Xuan Son Nguyen	458367a906	server : better security control for public deployments (#9776) ... * server : more explicit endpoint access settings * protect /props endpoint * fix tests * update server docs * fix typo * fix tests	2024-10-08 13:27:04 +02:00
ochafik	332506910f	tool-call: accept {"type": "function", "name": "fn" for llama 3.x	2024-10-07 02:23:37 +01:00
Georgi Gerganov	8c475b97b8	rerank : use [SEP] token instead of [BOS] (#9737) ... * rerank : use [SEP] token instead of [BOS] ggml-ci * common : sanity check for non-NULL tokens ggml-ci * ci : adjust rank score interval ggml-ci * ci : add shebang to run.sh ggml-ci	2024-10-05 15:55:04 +03:00
Daniel Kleine	133c7b46b3	Fixed RNG seed docs (#9723) ... * Update README.md fixed RNG seed info * changed print format to unsigned	2024-10-04 10:54:44 +02:00
Olivier Chafik	366efc8a18	tool-call: fix llama 3.x tc parsing when there are spaces before "name"	2024-10-03 21:46:41 +01:00
Olivier Chafik	b4fc1e8ba7	tool-call: adjust triggers to most common tool call variations from Llama-3.1-8B and Llama-3.2-3B	2024-10-03 19:23:08 +01:00
Olivier Chafik	ece12b074f	antiprompts: ensure partial match is at end of string (or else server stops sending replies)	2024-10-03 19:23:08 +01:00
Olivier Chafik	c36a196f53	tool-call: prepare possible externalization of minja + factor tool call style out of template	2024-10-01 23:12:24 +01:00
Ruchira Hasaranga	8277a817f1	console : utf-8 fix for windows stdin (#9690) ... * utf-8 fix for windows stdin * Update common/console.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-09-30 11:23:42 +03:00
ochafik	277f38536c	minja: attempt to handle windows' crlf	2024-09-30 03:45:50 +01:00
matiaslin	faac0bae26	common : ensure llama_batch size does not exceed max size (#9668) ... A crash was observed when the number of tokens added to a batch exceeds llama_batch size. An assertion in llama_batch_add was added to protect against llama_batch size overflow.	2024-09-29 15:25:00 +03:00
ochafik	9ac4b04aa2	tool-call: add fs_list_files to common, w/ win32 impl for msys2 build	2024-09-29 00:42:52 +01:00
ochafik	8738d94bbd	minja: qualify std::nullptr_t type for msys2 build	2024-09-29 00:18:22 +01:00
ochafik	bc3e0c0830	tool-call: Qwen 2.5 Instruct also requires object arguments	2024-09-28 23:05:35 +01:00
ochafik	b10ef04d8d	chat-template: tweak --chat-template error message when --jinja is set	2024-09-28 22:36:38 +01:00
ochafik	dbda025f87	tool-call: test messages -> template -> grammar -> tool call parser	2024-09-28 22:32:47 +01:00
ochafik	1b32ac129f	chat-template: fix test-arg	2024-09-28 20:06:10 +01:00
ochafik	e6be59c2a0	antiprompts: fix gcc8 build (avoid recursive struct)	2024-09-28 19:39:52 +01:00
ochafik	05bbba9f8a	tool-call: only match json eagerly for Llama 3.2	2024-09-28 19:05:10 +01:00
ochafik	7cef90cf9c	tool-call: more eager function call parsing for Functionary & Llama (give a chance to 3B model)	2024-09-28 18:33:40 +01:00
ochafik	d983516f40	tool-call: let the tool call handler expand chat template, moving builtin_tools down as extra_context	2024-09-28 17:46:36 +01:00
ochafik	0c85bc7a8f	tool-call: test tool call style detection	2024-09-28 17:43:09 +01:00
Georgi Gerganov	f4d2b8846a	llama : add reranking support (#9510) ... * py : add XLMRobertaForSequenceClassification [no ci] * py : fix scalar-tensor conversion [no ci] * py : fix position embeddings chop [no ci] * llama : read new cls tensors [no ci] * llama : add classigication head (wip) [no ci] * llama : add "rank" pooling type ggml-ci * server : add rerank endpoint ggml-ci * llama : aboud ggml_repeat during classification * rerank : cleanup + comments * server : accept /rerank endpoint in addition to /v1/rerank [no ci] * embedding : parse special tokens * jina : support v1 reranker * vocab : minor style ggml-ci * server : initiate tests for later ggml-ci * server : add docs * llama : add comment [no ci] * llama : fix uninitialized tensors * ci : add rerank tests ggml-ci * add reranking test * change test data * Update examples/server/server.cpp Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com> * add `--reranking` argument * update server docs * llama : fix comment [no ci] ggml-ci --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>	2024-09-28 17:42:03 +03:00
ochafik	701b664551	minja: add indent filter to support command-r-plus's chat templates	2024-09-27 19:00:14 +01:00
ochafik	0093a5e527	minja: fix identifiers parsing (when start w/ not/is/etc) and lstrip_blocks corner case (needed by DeepSeek-V2.5	2024-09-27 18:30:44 +01:00
ochafik	e62b5de3cf	tool-call: fix functionary-small-3.2 (first tool starts w/ name\n, subsequent are >>>name\n)	2024-09-27 07:06:33 +01:00
ochafik	e33b342da7	tool-call: fix passing of tools to template + allow agent to finish	2024-09-27 06:24:22 +01:00
ochafik	f62e688387	tool-call: fix crash / test non-tool call case (added llama_sampler_is_grammar_empty)	2024-09-27 06:04:41 +01:00
ochafik	27cd07a056	json: fix grammar conversion typo	2024-09-27 03:57:48 +01:00
ochafik	f9c1743bb5	minja: fix iterables	2024-09-27 03:36:49 +01:00
ochafik	c88c932d98	fix gcc error + lint	2024-09-26 19:18:40 +01:00