1089 Commits

Author	SHA1	Message	Date
Xuan Son Nguyen	afbbfaa537	server : add more env vars, improve gen-docs (#9635) ... * server : add more env vars, improve gen-docs * update server docs * LLAMA_ARG_NO_CONTEXT_SHIFT	2024-09-25 14:05:13 +02:00
Georgi Gerganov	cea1486ecf	log : add CONT level for continuing previous log entry (#9610)	2024-09-24 10:15:35 +03:00
StrangeBytesDev	0aa15011e3	server : add newline after chat example (#9616)	2024-09-24 09:04:39 +03:00
Georgi Gerganov	b0f27361f3	sampling : avoid expensive softmax during greedy sampling (#9605) ... * sampling : avoid expensive softmax during greedy sampling ggml-ci * speculative : fix default RNG seed + set sparams.n_probs * Update tests/test-sampling.cpp Co-authored-by: slaren <slarengh@gmail.com> * sampling : add clarifying comment [no ci] --------- Co-authored-by: slaren <slarengh@gmail.com>	2024-09-24 09:03:17 +03:00
Xuan Son Nguyen	0b3bf966f4	server : add --no-context-shift option (#9607) ... * server : add --no-context-shift option * small fix * Update examples/server/tests/features/embeddings.feature Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * tests : minor fix * revert usage of GGML_ASSERT * update server documentation --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-09-23 22:23:54 +02:00
Georgi Gerganov	37f8c7b4c9	perplexity : remove extra new lines after chunks (#9596)	2024-09-23 11:28:02 +03:00
slaren	63351143b2	quantize : improve type name parsing (#9570) ... quantize : do not ignore invalid types in arg parsing quantize : ignore case of type and ftype arguments	2024-09-20 20:55:36 +02:00
Georgi Gerganov	d39e26741f	examples : flush log upon ctrl+c (#9559)	2024-09-20 11:46:56 +03:00
Sigbjørn Skjæret	722ec1eb51	perplexity : do not escape input data by default (#9548)	2024-09-20 09:38:10 +03:00
Georgi Gerganov	6026da52d6	server : clean-up completed tasks from waiting list (#9531) ... ggml-ci	2024-09-19 12:44:53 +03:00
Sigbjørn Skjæret	eca0fab44e	imatrix : disable prompt escape by default (#9543)	2024-09-19 10:58:14 +03:00
Vinesh Janarthanan	8a308354f6	server : match OAI structured output response (#9527)	2024-09-18 09:50:34 +03:00
Eric Zhang	f799155ab8	server : fix OpenSSL build (remove obsolete LOG_INFO) (#9529)	2024-09-18 09:28:20 +03:00
Neo Zhang Jianyu	faf67b3de4	[SYCL]set context default value to avoid memory issue, update guide (#9476) ... * set context default to avoid memory issue, update guide * Update docs/backend/SYCL.md Co-authored-by: Meng, Hengyu <hengyu.meng@intel.com> --------- Co-authored-by: arthw <14088817+arthw@users.noreply.github.com> Co-authored-by: Meng, Hengyu <hengyu.meng@intel.com>	2024-09-18 08:30:31 +08:00
Michael Podvitskiy	7be099fa81	llama-bench: correct argument parsing error message (#9524)	2024-09-17 22:41:38 +02:00
Bert Wagner	8b836ae731	arg : add env variable for parallel (#9513) ... * add env variable for parallel * Update README.md with env: LLAMA_ARG_N_PARALLEL	2024-09-17 16:35:38 +03:00
Vinesh Janarthanan	441b72b91f	main : option to disable context shift (#9484) ... * added cli arg to disable context shift * reverted precommit * updated README.md for main * white space * allow disabling context shift in the server * Update common/arg.cpp no-context-shift only works for main example Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * added server example to --no-context-shift args * removed server changes * white space --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-09-16 09:20:01 +03:00
Georgi Gerganov	6262d13e0b	common : reimplement logging (#9418) ... https://github.com/ggerganov/llama.cpp/pull/9418	2024-09-15 20:46:12 +03:00
slaren	e6deac31f7	gguf-split : add basic checks (#9499) ... * gguf-split : do not overwrite existing files when merging * gguf-split : error when too many arguments are passed	2024-09-15 19:02:27 +02:00
VoidIsVoid	dcdcee3a74	server: add data: [DONE] to /chat/completions stream response (#9459)	2024-09-14 11:36:44 +02:00
Xuan Son Nguyen	feff4aa846	server : add loading html page while model is loading (#9468) ... * Adding loading page for '/' server requests * set content when model is loading * removed loading html file * updated cmakelist * updated makefile * cleaned up whitespace * cleanup for PR removed error * updated server test to handle 503 HTML * updated server test to handle 503 HTML * ca†ch 503 before parsing json * revert test * account for both api and web browser requests * precommit corrections * eol fix * revert changes to pre-commit * removed print statement * made loading message more descriptive * also support .html files --------- Co-authored-by: VJHack <flymyplane21@gmail.com> Co-authored-by: Vinesh Janarthanan <36610342+VJHack@users.noreply.github.com>	2024-09-13 14:23:11 +02:00
Georgi Gerganov	0abc6a2c25	llama : llama_perf + option to disable timings during decode (#9355) ... * llama : llama_perf + option to disable timings during decode ggml-ci * common : add llama_arg * Update src/llama.cpp Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com> * perf : separate functions in the API ggml-ci * perf : safer pointer handling + naming update ggml-ci * minor : better local var name * perf : abort on invalid sampler pointer ggml-ci --------- Co-authored-by: Xuan Son Nguyen <thichthat@gmail.com>	2024-09-13 09:53:38 +03:00
Mathijs Henquet	78203641fe	server : Add option to return token pieces in /tokenize endpoint (#9108) ... * server : added with_pieces functionality to /tokenize endpoint * server : Add tokenize with pieces tests to server.feature * Handle case if tokenizer splits along utf8 continuation bytes * Add example of token splitting * Remove trailing ws * Fix trailing ws * Maybe fix ci * maybe this fix windows ci? --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>	2024-09-12 22:30:11 +02:00
fengerhu1	e665744317	llava : fix the script error in MobileVLM README (#9054) ... Signed-off-by: Erhu Feng <2748250768@qq.com>	2024-09-12 14:34:22 +03:00
Michael Podvitskiy	ff76e18516	cmake : fixed the order of linking libraries for llama-quantize (#9450)	2024-09-12 14:27:14 +03:00
Georgi Gerganov	d6a04f872d	ggml : hide ggml_object, ggml_cgraph, ggml_hash_set (#9408) ... * ggml : hide ggml_object, ggml_cgraph, ggml_hash_set ggml-ci * ggml : add ggml-impl.h to backends * ggml : fix compiler warnings ggml-ci * ggml : add assert upon adding nodes	2024-09-12 14:23:49 +03:00
Neo Zhang Jianyu	c9c8575a1a	enhance run script to be easy to change the parameters (#9448) ... Co-authored-by: arthw <14088817+arthw@users.noreply.github.com>	2024-09-12 17:44:17 +08:00
Xuan Son Nguyen	0996c5597f	llava : correct args for minicpmv-cli (#9429)	2024-09-11 12:59:13 +02:00
Georgi Gerganov	d2b496bff4	batched-bench : remove unused code (#9305)	2024-09-11 10:03:54 +03:00
slaren	49006c67b4	llama : move random seed generation to the samplers (#9398) ... * llama_sampler_penalties : clamp penalty_last_n to zero	2024-09-10 18:04:25 +02:00
Xuan Son Nguyen	bfe76d4a17	common : move arg parser code to arg.cpp (#9388) ... * common : move arg parser to arg.cpp * better categorize args * add cmake * missing climits * missing cstdarg * common : more explicit includes * fix build * refactor gpt_params_parse * update server readme * fix test --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-09-09 23:36:09 +02:00
slaren	5fb5e24811	llama : minor sampling refactor (2) (#9386)	2024-09-09 17:10:46 +02:00
Antonis Makropoulos	5ed087573e	readme : add LLMUnity to UI projects (#9381) ... * add LLMUnity to UI projects * add newline to examples/rpc/README.md to fix editorconfig-checker unit test	2024-09-09 14:21:38 +03:00
Radoslav Gerganov	54f376d0b9	rpc : update README [no ci] (#9320) ... Update README with instructions how to offload model layers to both local and remote devices	2024-09-09 11:04:39 +03:00
Dan Johansson	b2e89a3274	Arm AArch64: Documentation updates (#9321) ... * Arm AArch64: Documentation updates * Update docs/build.md to include information on how to enable the Arm optimized gemm/gemv kernels * Update examples/quantize/README.md with information on the Q4_0_4_4, Q4_0_4_8 and Q4_0_8_8 formats * Add newline to the end of docs/build.md	2024-09-09 10:02:45 +03:00
Xuan Son Nguyen	3f7ccfd649	common : bring back missing args, add env var duplication check (#9375) ... * common : bring back missing args * move duplication check to test-arg-parser * add check for duplicated env var * correct default values	2024-09-08 18:08:55 +02:00
Xuan Son Nguyen	00b02bb249	imatrix : fix arg parser for imatrix (#9366) ... * imatrix : fix arg parser * beautify printing first arg	2024-09-08 12:12:17 +02:00
Georgi Gerganov	a5b5d9a101	llama.android : fix build (#9350)	2024-09-08 00:33:50 +03:00
Georgi Gerganov	faf69d4237	llama : sanitize invalid tokens (#9357) ... * common : do not add null tokens during warmup ggml-ci * llama : check that the input tokens are valid ggml-ci * tests : fix batch size of bert model ggml-ci	2024-09-08 00:33:13 +03:00
Xuan Son Nguyen	1b9ae5189c	common : refactor arg parser (#9308) ... * (wip) argparser v3 * migrated * add test * handle env * fix linux build * add export-docs example * fix build (2) * skip build test-arg-parser on windows * update server docs * bring back missing --alias * bring back --n-predict * clarify test-arg-parser * small correction * add comments * fix args with 2 values * refine example-specific args * no more lamba capture Co-authored-by: slaren@users.noreply.github.com * params.sparams * optimize more * export-docs --> gen-docs	2024-09-07 20:43:51 +02:00
Georgi Gerganov	df270ef745	llama : refactor sampling v2 (#9294) ... - Add `struct llama_sampler` and `struct llama_sampler_i` - Add `llama_sampler_` API - Add `llama_sampler_chain_` API for chaining multiple samplers - Remove `LLAMA_API_INTERNAL` - Add `llama_perf_` API and remove old `llama_print_timings` and `llama_reset_timings`	2024-09-07 15:16:19 +03:00
Xuan Son Nguyen	9b2c24c099	server : simplify state machine for slot (#9283) ... * server : simplify state machine for slot * add SLOT_STATE_DONE_PROMPT * pop_deferred_task * add missing notify_one * fix passkey test * metrics : add n_busy_slots_per_decode * fix test step * add test * maybe fix AddressSanitizer? * fix deque ? * missing lock * pop_deferred_task: also notify * Update examples/server/server.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>	2024-09-06 23:21:29 +02:00
Aarni Koskela	134bc38ecf	llama-bench : log benchmark progress (#9287) ... * llama-bench : add optional progress messages	2024-09-06 23:03:01 +02:00
Aarni Koskela	815b1fb20a	batched-bench : add --output-format jsonl option (#9293) ... `--output-format` is modeled after `llama-bench`'s options	2024-09-06 17:59:58 +02:00
Xuan Son Nguyen	4a1411b4f1	server : fix missing lock (#9334)	2024-09-06 14:06:04 +02:00
compilade	9bc6db28d0	ggml-quants : ternary packing for TriLMs and BitNet b1.58 (#8151) ... * ggml-quants : 1.625 bpw ternary packing for BitNet 1.58b * ggml-quants : faster 1.625 bpw AVX2 vec_dot Not using a lookup table anymore makes it match q4_0 speed. * gguf-py : fix formatting * llama : remove spaces on empty line * ggml-quants : subtract 1 when back in epi8 This makes the 1.625 bpw type go faster than q4_0. Still not the fastest. * ggml-quants : Q2_2 now faster than Q4_K on with AVX2 * ggml-quants : cleanup Q1_3 code formatting * ggml-quants : ARM NEON vec_dot for q2_2 and q1_3 * ggml-quants : use ceiling division when quantizing q1_3 * convert-hf : simplify BitNet pre-quantization This still results in the exact same tensor weights and scales, but it reveals some weirdness in the current algorithm. * convert-hf : allow converting the weird BitNet 1.3B Its FFN size is 5460 which is not convenient. The offending tensors are kept in F16, which makes the final model 5.01 bpw. * bitnet : replace 1.58b with b1.58, as in the paper * ggml-quants : fix build failure on Windows * ggml-quants : attempt to fix Arm 32-bit support * ggml : add some informative comments in q1_3 vec_dot * ggml : add TQ1_0 and TQ2_0 ternary quantization types * ggml : even faster TQ2_0 * ggml : also faster TQ1_0 Same optimization as for TQ2_0 by offsetting the sum instead of the weights. This makes TQ1_0 almost as fast as Q8_0 on AVX2. * ggml : fix build issues in certain environments * ggml : add NEON vec_dot implementation for TQ1_0 and TQ2_0 * ggml : avoid directly using vmlal_high_s8, for 32-bit ARM compat The compiler seems smart enough to use the same instruction even when using vget_high_s8 instead. * ggml : remove q1_3 and q2_2 No more 1.625 bpw and 2.000 bpw, now instead using 1.6875 bpw and 2.0625 bpw with TQ1_0 and TQ2_0, respectively. * llama : remove the separate scale tensors of BitNet b1.58 They won't be needed, since the remaining ternary quant types have built-in scales. * ggml-quants : rename fields of TQ1_0 and TQ2_0 structs for consistency * ggml-quants : allow using vdotq_s32 in TQ2_0 vec_dot Not yet tested on hardware which supports it, might not work or might not even compile. But also it might. It should make the performance better on recent ARM CPUs. * ggml-quants : remove comment about possible format change of TQ2_0 Making it slightly more convenient for AVX512 but less convenient for everything else is not worth the trouble. * gguf-py : Numpy (de)quantization for TQ1_0 and TQ2_0 * ggml-quants : use roundf instead of nearest_int for TQ1_0 and TQ2_0 This does not change anything for ternary models, since their values should never end up being in halfway cases anyway. * convert : allow direct conversion to TQ1_0 and TQ2_0 The token embeddings and output tensors are kept in F16 to allow quantizing them to Q4_K and Q6_K with llama-quantize. * llama : handle fallback for TQ1_0 and TQ2_0 with Q4_0 Q4_0 is not completely symmetric (so not lossless for ternary models), but it should be good enough. * ggml-quants : allow using ARM dot product instructions for TQ1_0 * ggml-quants : deduplicate TQ1_0 and TQ2_0 __ARM_FEATURE_DOTPROD support * ggml : remove unused ggml_mul special case It would otherwise conflict with the more general optimization coming with Mamba-2. * ggml : handle TQ1_0 and TQ2_0 in dequantization-based operators * test-backend-ops : add TQ1_0 and TQ2_0 comments for later Not yet adding uncommented, because some backends like SYCL and Metal do not properly handle unknown types in supports_op for GGML_OP_MUL_MAT. (and Metal also doesn't handle it with GGML_OP_GET_ROWS) Support for TQ1_0 and TQ2_0 for other backends than CPU will be added in follow-up pull requests.	2024-09-05 21:48:47 -04:00
slaren	bdf314f38a	llama-bench : fix NUL terminators in CPU name (#9313)	2024-09-05 02:19:39 +02:00
Radoslav Gerganov	82e3b03c11	rpc : make RPC servers come first in the device list (#9296) ... * rpc : make RPC servers come first in the device list * rpc : disable options for non-RPC builds * rpc : rpc_count always zero for non-RPC builds	2024-09-04 11:08:32 +03:00
Aarni Koskela	8962422b1c	llama-bench : add JSONL (NDJSON) output mode (#9288) ... * llama-bench : add JSONL (NDJSON) output mode * llama-bench : update usage docs	2024-09-03 19:58:54 +02:00
Xuan Son Nguyen	48baa61ecc	server : test script : add timeout for all requests (#9282)	2024-09-02 22:08:38 +02:00