root/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggerganov/llama.cpp.git synced 2024-12-26 19:34:35 +00:00
Code Issues Actions 4 Packages Projects Releases Wiki Activity
3,220 Commits 359 Branches 2,870 Tags 389 MiB
f702a90e24
Commit Graph

3 Commits

Author SHA1 Message Date
Johannes Gäßler
9a590c8226
CUDA: optimize MMQ int8 tensor core performance (#8062) 
* CUDA: optimize MMQ int8 tensor core performance

* only a single get_mma_tile_x_k function

* simplify code, make functions constexpr
 2024-06-24 12:41:23 +02:00
Johannes Gäßler
bdcb8f4222
CUDA: int8 tensor cores for MMQ (q4_K, q5_K, q6_K) (#7860) 2024-06-11 08:26:07 +02:00
Johannes Gäßler
1f0dabda8d
CUDA: use tensor cores for MMQ (#7676) 
* CUDA: int8 tensor cores for MMQ (legacy quants)

* fix out-of-bounds writes

* __builtin_assume -> GGML_CUDA_ASSUME

* fix writeback returning too early
 2024-06-10 11:45:13 +02:00