llama.cpp/fattn-vec-f16-instance-hs128-q4_0-q5_0.cu at 9b596417af11c9ac44fcae0fcfbc6f3665089083 - llama.cpp - Gitea: Git with a cup of tea

root/llama.cpp

mirror of https://github.com/ggerganov/llama.cpp.git synced 2024-11-14 23:09:53 +00:00

Johannes Gäßler 9b596417af

CUDA: quantized KV support for FA vec (#7527 )

* CUDA: quantized KV support for FA vec

* try CI fix

* fix commented-out kernel variants

* add q8_0 q4_0 tests

* fix nwarps > batch size

* split fattn compile via extern templates

* fix flake8

* fix metal tests

* fix cmake

* make generate_cu_files.py executable

* add autogenerated .cu files

* fix AMD

* error if type_v != FP16 and not flash_attn

* remove obsolete code

2024-06-01 08:44:14 +02:00

6 lines

179 B

Plaintext

Raw Blame History

 // This file has been autogenerated by generate-variants.py, do not edit manually.
 #include "../fattn-vec-f16.cuh"
 DECL_FATTN_VEC_F16_CASE(128, GGML_TYPE_Q4_0, GGML_TYPE_Q5_0);