14 Commits

Author	SHA1	Message	Date
Georgi Gerganov	a1327c71c6	parallel : rename hot-plug to continuous-batching	2023-09-20 09:24:41 +03:00
Georgi Gerganov	7b7472ee26	parallel : minor	2023-09-20 00:35:10 +03:00
Georgi Gerganov	6028879f56	parallel : print misses on each request	2023-09-19 23:50:05 +03:00
Georgi Gerganov	eed3fd4234	parallel : count cache misses	2023-09-19 23:47:47 +03:00
Georgi Gerganov	8a9aca37c1	parallel : remove question with short answers	2023-09-19 23:34:30 +03:00
Georgi Gerganov	4b5f3cd6bf	parallel : process system prompt once + configurable paramters + llama API	2023-09-19 17:00:42 +03:00
Georgi Gerganov	82e20e9ba0	parallel : remove new line from prompt	2023-09-19 13:54:41 +03:00
Georgi Gerganov	16090a5dde	parallel : fix sequence termination criteria	2023-09-19 13:29:29 +03:00
Georgi Gerganov	806d397c1a	parallel : try smaller batches when the KV cache is fragmented	2023-09-19 13:21:36 +03:00
Georgi Gerganov	36714e16d0	parallel : various improvements	2023-09-19 12:29:37 +03:00
Georgi Gerganov	fa0e677820	llama : extend batch API to select which logits to output	2023-09-19 00:24:13 +03:00
Georgi Gerganov	897caccdf4	fixes : speculative KV cache + llama worst-case graph	2023-09-18 22:32:28 +03:00
Georgi Gerganov	466b513851	parallel : disable hot-plug to avoid cache fragmentation	2023-09-18 21:34:20 +03:00
Georgi Gerganov	0161372b9a	parallel : example for serving multiple users in parallel	2023-09-18 20:37:28 +03:00