9 Commits

Author	SHA1	Message	Date
Georgi Gerganov	4b5f3cd6bf	parallel : process system prompt once + configurable paramters + llama API	2023-09-19 17:00:42 +03:00
Georgi Gerganov	82e20e9ba0	parallel : remove new line from prompt	2023-09-19 13:54:41 +03:00
Georgi Gerganov	16090a5dde	parallel : fix sequence termination criteria	2023-09-19 13:29:29 +03:00
Georgi Gerganov	806d397c1a	parallel : try smaller batches when the KV cache is fragmented	2023-09-19 13:21:36 +03:00
Georgi Gerganov	36714e16d0	parallel : various improvements	2023-09-19 12:29:37 +03:00
Georgi Gerganov	fa0e677820	llama : extend batch API to select which logits to output	2023-09-19 00:24:13 +03:00
Georgi Gerganov	897caccdf4	fixes : speculative KV cache + llama worst-case graph	2023-09-18 22:32:28 +03:00
Georgi Gerganov	466b513851	parallel : disable hot-plug to avoid cache fragmentation	2023-09-18 21:34:20 +03:00
Georgi Gerganov	0161372b9a	parallel : example for serving multiple users in parallel	2023-09-18 20:37:28 +03:00