llama.cpp/ggml-cuda.h

#pragma once

#include "ggml.h"

#ifdef  __cplusplus
extern "C" {
#endif

void * ggml_cuda_host_malloc(size_t size);
void   ggml_cuda_host_free(void * ptr);

// backend API

struct ggml_backend ggml_backend_cuda_init();


#ifdef  __cplusplus
}
#endif
Multi GPU support, CUDA refactor, CUDA scratch buffer (#1703) * CUDA multi GPU + scratch ggml_cuda_compute_forward Tensor parallelism ggml_cuda_add ggml_cuda_rms_norm ggml_cuda_silu CUDA scratch buffer --main-gpu CLI option 2023-06-06 19:33:23 +00:00			`#pragma once`

cuBLAS: non-contiguous tensor support (#1215) * Cuda: non-contiguous tensor support * remove extra stuff * rename * fix error * more fixes, now OpenBLAS and CLBlast build too * now then? 2023-04-28 23:31:56 +00:00			`#include "ggml.h"`
Improve cuBLAS performance by using a memory pool (#1094) * Improve cuBLAS performance by using a memory pool * Move cuda specific definitions to ggml-cuda.h/cu * Add CXX flags to nvcc * Change memory pool synchronization mechanism to a spin lock General code cleanup 2023-04-21 19:59:17 +00:00
Improve cuBLAS performance by dequantizing on the GPU (#1065) 2023-04-20 01:14:14 +00:00			`#ifdef __cplusplus`
			`extern "C" {`
			`#endif`

cuBLAS: use host pinned memory and dequantize while copying (#1207) * cuBLAS: dequantize simultaneously while copying memory * cuBLAS: use host pinned memory * cuBLAS: improve ggml_compute_forward_mul_mat_f16_f32 with pinned memory * cuBLAS: also pin kv cache * fix rebase 2023-04-29 00:04:18 +00:00			`void * ggml_cuda_host_malloc(size_t size);`
			`void ggml_cuda_host_free(void * ptr);`

ggml backend interface wip refactor ggml-cuda 2023-07-10 15:32:06 +00:00			`// backend API`

			`struct ggml_backend ggml_backend_cuda_init();`
Leverage mmap for offloading tensors to GPU (#1597) * Rebase to latest * Show progress * Add assert to make sure we only allocate temp buffer for non-CPU backend tensor Co-authored-by: Johannes Gäßler <johannesg@5d6.de> --------- Co-authored-by: Johannes Gäßler <johannesg@5d6.de> 2023-06-12 12:44:16 +00:00
ggml : GPU-accelerated token generation (#1412) * CUDA kernel for q4_0 dequant. + mat. vec. mult. * Added q4_1 via template * Added missing __syncthreads(); * --gpu_layers -> --gpu-layers * Shorter dequantize_mul_mat_vec line * q5_0 dequantize_mul_mat kernel * More readable dequantize_mul_mat_vec logic * dequantize_mul_mat_vec kernels for q5_1, q8_0, f16 * llama : offload "output" tensor to GPU too + coding style fixes --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> 2023-05-13 13:38:36 +00:00
Improve cuBLAS performance by dequantizing on the GPU (#1065) 2023-04-20 01:14:14 +00:00			`#ifdef __cplusplus`
			`}`
			`#endif`