mirror of
https://github.com/ggerganov/llama.cpp.git
synced 2024-11-11 21:39:52 +00:00
0e76a8992c
* fix track_max_mem in forward_batch_wo_cache_flash_attn_train * remove unnecessary Adam(W) optimizer tensors. reduces optimizer memory overhead from 7*modelsize to 2*modelsize. additionally allows to optimize models with more than 2^31 parameters by replacing int with int64_t. bumps training checkpoint file version, but old checkpoints can still be read. new version with less tensors is saved. * add gradient clipping to AdamW * Fix reset of unused g->nodes and g->grads to NULL * implement gradient checkpointing for training reduces memory overhead from O(n_layer) to O(sqrt(n_layer)) as explained in readme of https://github.com/cybertronai/gradient-checkpointing * remove unused compute buffer 3 * add and use function ggml_build_backward_expand to avoid stack overflows with large maximum number of nodes GGML_API void ggml_build_backward_expand(struct ggml_context * ctx, struct ggml_cgraph * gf, struct ggml_cgraph * gb, bool keep); * change AdamW decay parameter to work like the torch AdamW decay parameter It is now relative to Adam learning rate `alpha*sched`. Before that it was relative to `sched` only. `alpha` being the maximum learning rate and `sched` being a scaling parameter in [0..1] * change default AdamW weight decay parameter used in training to 0.1 as used in nanoGPT * change default AdamW weight decay parameter defined in ggml to 0.0, making Adam default instead of AdamW btw: the default weight decay parameter for torch.optim.AdamW is 0.01 * bug fixes for cross entropy loss ggml_cross_entropy_loss: sums where not correctly added in workload of each thread ggml_cross_entropy_loss_back: simplify backward process, reducing numerical issues guard usage of exp f16 lookup in cross entropy by #define GGML_CROSS_ENTROPY_EXP_FP16 cross entropy loss is only used once during training, but it is quite sensitive to numerical errors introduced by exp-f16-lookup. so exp-f16-lookup for cross entropy loss is disabled by default, trading better gradients for very slightly worse runtime performance. * fix test-grad0 for cross_entropy_loss the second argument to cross_entropy_loss must sum up to 1 for each row * fix test-grad0 for soft_max dont use only sum as aggregation, because sum of softmax is always 1 -> finite differences should not work instead use sum(log(soft_max()*(1-eps)+eps)); use eps to avoid log(0) * improve finite differences of test-grad0 by using double instead of float * change cross_entropy_loss to output average over all rows this helps keeping the loss and gradients in a sane range * improve gradient checkpointing sqrt(n_layers) is only the best checkpoint step when mem size of checkpoints and mem size of layers are equal. since layers require more memory than the single-tensor-checkpoint we use, the optimal values are compute different: ``` given: n, u, v objective: minimize(a*u+b*v) where a*b=n, a>0, b>0 b=n/a minimize(a*u+v*n/a) diff(a*u+v*n/a, a) = u - (v*n/a)/a diff(a*u+v*n/a, a) == 0 u - (v*n/a)/a == 0 u == v*n/(a*a) u*a*a = v*n a*a = v*n/u a = sqrt(n*v/u) ``` this change results in more checkpoints, requiring less layers to store between checkpoints, overall improving memory usage. * disable gradient checkpointing debug output * llama : fix rope usage in train-text-from-scratch after ChatGLM change * add more training parameters: --enable-restart N Only for Adam optimizer. Enable restarts of cos-decay --disable-restart N Only for Adam optimizer. Disable restarts of cos-decay --opt-past N Number of optimization iterations to track for delta convergence test. Disabled when zero. --opt-delta N Maximum delta for delta convergence test. Disabled when <= zero. --opt-max-no-improvement N Maximum number of optimization iterations with no improvement. Disabled when <= zero. --adam-epsf N AdamW epsilon for convergence test. Disabled when <= zero. --adam-min-alpha N Adam minimum learning rate alpha, usually 0.1 * alpha * replace memcpy with reshape operation so that the graph is not cut at the input this makes it possible to store other values into the input tensor and then simply recompute the graph without rebuilding it * remove unused function argument from get_example_targets_batch * measure and print total training time * add optimization callback to ggml_opt_resume_g this callback is called before each iteration with custom data and pointer to learning schedule parameter (only used in Adam(W)). can be used for dynamic learning schedule and setting input data for batches before each iteration * use optimization callback in training allows dynamic learning schedule and different batch data for each iteration without relying on low n_iter and high n_examples parameters reduces runtime by avoiding restart of optimization function and improves training convergence by providing a different batch for each iteration * add minimum number of tensor dimensions to apply weight decay (default 2) this allows to not apply weight decay to bias parameters * rename training parameter cos-decay-alpha to cos-decay-min and clarify that adam-min-alpha also applies to warmup * fix increase of model.train_samples and model.train_tokens now that each optimizer iteration gets its own batch we need to multiply by number of opt iterations * change sampling parameters for prediction after training to defaults of common.h and clarify what is context for prediction and what are generated tokens * tighten abs error bounds for cross_entropy_loss in test-grad0 * add conditional compilation of using F16 exp in flash attention uncomment `// #define GGML_FLASH_ATTN_EXP_FP16` to enable usage of f16 exp in flash attention * tighten abs error bounds for flash_attn in test-grad0 * tighten abs error bounds for sqrt in test-grad0 * remove out-commented vectorized code of opt_adam the vectorized code might be bit faster for low number of parameters, but it had a big memory usage overhead * ggml : update ggml_rms_norm_back with configurable eps * llama training : fix ggml_rms_norm_back calls to pass configurable eps * remove trailing whitespace * add train function using automatic gradient checkpointing backward pass and allocator * in train function replace add_inplace by regular add because using add_inplace seems to result in different gradients * don't use allocate hash_map on context because the context has no_alloc=True when using memory allocator resulting in NULL data pointers * correctly clone reshape and permute operations by also cloning tensor->nb values * fix variable name and add missing type cast * terminate recursive tensor cloning when reaching tensor without src tensors * correctly clone view tensors by setting data pointers without this the checkpointing would only work when being used together with memory allocator * fix variable names * swap arguments to commutative ops to be the same as in `forward_batch_wo_cache_flash_attn` * add input tensors as checkpoints so that recursive tensor cloning of gradient checkpointing terminates on input tensors * fix variable name and add missing boolean negation * make sure some tensors are not reallocated by inserting new temporary nodes depending on them: output and parameter gradient tensors need to be available at the end of the graph execution parameter gradient tensors also need to be available before the graph execution because they are set to zero before each optimizer iteration checkpoint tensors are allocated all together to reduce memory allocator fragmentation afterwards, in addition to the temporary nodes, we also need to reset the temporary leafs * fix ASSERT to work with zero layers * add training options whether to use allocator and/or unified training function * integrate unified training function which may use memory allocator the unified training function also supports arguments whether to use flash attention and/or gradient checkpointing * format name of cloned tensors with " (clone)" suffix * set names for tensors in unified train function for easier debugging * allocate graph on context using ggml_new_graph * remove handwritten training functions * remove unused training parameters "use_scratch" and "use_unified" * remove trailing whitespace * remove unused train params: mem_compute1_gb & mem_compute2_gb mem_compute_gb is used for compute when automatic memory allocator is not enabled, otherwise it can be very small to only hold the tensor definitions mem_compute0_gb is used for automatic memory allocator (as long as measurement of max required size is not implemented) * remove unused forward_batch function * add debug asserts in ggml_allocr_alloc to some common pitfalls when using this function directly * only use ggml_allocr_alloc when tensor has NULL data and is no view * fix test when to create temporary backward graph temporary backward graph is only necessary when using checkpointing * fix memory "leak" in optimizers each iteration a new cplan with new memory for work data was allocated. now cplan creation only happens at the start of optimization, with each iteration reusing the cplan and its work data. * reverse order of for loop in ggml_build_backward_expand to save memory when using gradient checkpointing and allocator with this loop order gradient checkpointing with allocator on 16 layer model saves 13% memory; 2 layer memory it saves 2% memory. the computation results are the same * add API functions to access llama model tensors * add stub example for finetuning, based on train-text-from-scratch * move and remove code * add API functions to access remaining model parameters: mult, head and rot * first draft for LORA finetune training * remove const model and layer arguments in API functions for accessing model tensors * bug fixes to make finetune compile automatic allocator does not work yet * add debug prints for training memory improvements * fix names of lora tensors * avoid stack overflow resulting from big ggml_cgraph replace stack allocation and ggml_build_forward by ggml_new_graph in combination with ggml_build_forward_expand * replace llama API functions to get model tensors by one function to get model tensor by name LLAMA_API struct ggml_tensor * llama_get_model_tensor(struct llama_model * model, const char * name); * remove unused call to not existing llama_get_layer_from_model * implement ggml_compute_forward_out_prod_q_f32 * remove trailing whitespace * add lora finetune support on quantized base model tensors * add ggml_add_cast API function this function works like ggml_add, but accepts a data type for the resulting tensor. only supported for quantized src0 input. * use ggml_add_cast in finetuning lora-applied weights will now have data type F32, which improves gradients when finetuning quantized base models * bug fix: actually use result type passed to ggml_add_cast * make sure base model tensors data cannot be used in viewable operations memory allocator would try to make lora application inplace on base model tensors. since those are memory mapped this will result in memory access violations * fix bug in ggml_out_prod which resulted in wrong n_dims of result tensors * avoid keeping in memory ALL of the gradients The problem here stems from ggml_graph_reset. This function is called in the optimization function, before each graph computation, to reset the gradients to zero. This required a unique memory slot for each gradient: allocating memory from a previosly freed memory location might lead to non-zero input gradients. During ggml_compute_backward the gradients are build stepwise by adding or substracting new values, starting from a OP_NONE tensor which needs to contain zero-values. This requires the graph reset. To avoid this I now remember in ggml_build_backward_expand the original OP_NONE gradient tensors in a hash table, which is passed to ggml_compute_backward. There instead of using add (or sub or similar) I test whether the existing gradient to be changed is a zero-valued-tensor by looking up its existence in the hash table. When it is such a zero-tensor it will not be modified, but replaced by the value to be added, otherwise the regular add (not inplace, allocator will take care of this) will be used. This way none of those zero-tensor values will be necessary in the final backward graph and more importantly they won't need a unique memory slot, just to make them zero. * remove trailing whitespace * remove debug prints and function to compute tensor data hash * improve optimization iteration prints * adjust maximal values to support finetuning 3B models * change default finetune params lora_r and lora_alpha to match the n_rank parameters of 4 * bug fix: make sure finetune input gradient is allocated at begin and kept until end * remove unnecessary src tensor from ggml_get_rows_back we don't need data of src[2] for computation, only to setup the correct output shape. remove dependency on src[2], so that allocator can work more freely. the computational graph is still completely determined, because the output shape is naturally included. this is similar to how ggml_reshape does it. * remove unnecessary src tensor from ggml_repeat & ggml_repeat_back we don't need data of src[1] for computation, only to setup the correct output shape. remove dependency on src[1], so that allocator can work more freely. the computational graph is still completely determined, because the output shape is naturally included * resolve todo allocator will only make it inplace when they are of the same type * mixing multiple LORA adapters is now possible pass more than one '--lora FNAME' argument to apply more than one LORA. use '--lora-scaled FNAME S' when you want to specify a user-defined scale for an adapter. * add option to save finetune output every N iterations * also save latest finetune output with ITERATION="LATEST" and print where files are saved saving with LATEST makes it easier to resume training from the latest checkpoint the string "LATEST" can be configured with command line option "--fn-latest STR" * update checkpoint train stats before saving via "--save-every" * add command line option `--rank-wo N` for rank of wo tensor * update finetune README * fix dump_non_result_info_yaml to output multiple lora adapters * bug fix: replace GGML_TYPE_SIZE[t] by ggml_type_size(t) * replace llama_n_mult by llama_n_ff * finetune bug fixes to compile with merged in code from master * remove prediction related code to reduce duplicated code with main use main instead * reduce large memory overhead in train-text-from-scratch all gradients had to be pinned so that graph_reset works correctly. this is no longer necessary with the changes to ggml_compute_backward introduced in this PR. * add comment explaining why finetune checkpoints are allocated in one block * make default value of float member a float literal * handle rms_norm and rope parameters the same as in train-text-from-scratch * remove unused code * remove vocab related code as it is unnecessary * add LLM_KV_TRAINING_TYPE to train-text-from-scratch checkpoints so that they can be differentiated from lora finetune checkpoints * add gguf constants and load/save functions from train-text-from-scratch * add load & save lora finetune checkpoints via gguf * add python script to convert old finetune checkpoint files to gguf * remove old checkpoint save & load code * remove code to print data checksums which was used to verify correctness of new gguf code * omit tokenization when training is disabled, only save llama lora adapter training can be disabled by passing '-n 0' to finetune * remove trailing whitespace * update README.md * implement ggml_compute_forward_repeat_f16 * avoid stack overflow of large cgraphs in test-grad0 * add ggml API functions ggml_unravel_index, ggml_get_i32_nd and its analogs for set and for f32 ggml_get_i32_1d, ggml_set_i32_1d, ggml_get_f32_1d, ggml_set_f32_1d now support non-contiguous tensors. in case of non-contiguous tensor, the 1d index is unraveled into a multi index using ggml_unravel_index to be passed to '_nd' function equivalent. this fixes a bug in test-grad0 which happens due to ggml_build_backward not building purely contiguous tensors anymore * increase test-grad0 context mem size to accommodate for bigger cgraph * add sanity check to ggml_compute_backward, asserting the correct shape of gradients * fix ggml_acc_or_set to return tensor of correct shape * remove unused 'inplace' argument from ggml_compute_backward function inplace operations to add gradients are no longer created by ggml_compute_backward use allocator to automatically make inplace operations * add missing argument 'int i0' to ggml_get_i32_nd & ggml_set_i32_nd header declarations * fix error message in ggml_allocr_alloc to display actual max_avail * fix check_gradient ggml_build_backward_expand was previously replaced by ggml_build_backward, but the assignment of forward graph to backward graph missing * use tensor->view_src instead of ggml_is_view and get_view_source * move gradient checkpointing code into ggml, new API function: // build gradient checkpointing backward graph gb for gf using provided checkpoints // gb_tmp will contain original backward graph with rewritten backward process nodes, // but without the second forward pass nodes. GGML_API void ggml_build_backward_gradient_checkpointing( struct ggml_context * ctx, struct ggml_cgraph * gf, struct ggml_cgraph * gb, struct ggml_cgraph * gb_tmp, struct ggml_tensor * * checkpoints, int n_checkpoints); * replace custom data getters and setters by ggml functions * train-text-from-scratch can train (full finetune) gguf models just pass the gguf model via `--checkpoint-in FN`. after this, to continue training, pass the generated checkpoint instead of the original gguf model. tested with smaller models, bigger models may exceed available memory. use (LORA) finetune for those. * remove trailing whitespace * add option to save train-text-from-scratch output every N iterations * update README.md * fix warnings * fix warnings * remove finetune option to disable allocator the allocator should always be used. by making sure that it is always used it gets easier to implement automatic memory requirements computation * add tensor checkpoints only when gradient checkpointing is enabled * initialize opt ggml context if none was provided * add ggml-alloc API function 'ggml_allocr_max_size' to get max size of alloc GGML_API size_t ggml_allocr_max_size(struct ggml_allocr * alloc); * finetune: automatically allocate all memory and changes to command line options remove '--n_examples N' parameter, as it no longer makes sense to call optimization process multiple times in a loop. add '--only_write_lora' command line option: will skip tokenization and training, to only write a llama.cpp comptabile LORA adapter. remove memory buffer related command line options. improve iteration console output. * add finetune to Makefile * update README.md * print time per iteration and estimate remaining time * increase measured alloc size by tensor_alignment ggml_allocr_reset will reduce the given size by up to tensor_alignment-1 * fix README.md * add some more allocator debug prints * bug fix, probably solves the 'ggml_allocr_alloc: not enough space in the buffer' issue * revert last commit "bug fix, probably solves the 'ggml_allocr_alloc: not enough space in the buffer' issue" "alloc was freeing an externally allocated tensor, because it calculated the end of allocator memory as alloc->data + alloc->max_size instead of alloc->data + alloc->size." This is intentional to reduce the risk of freeing external tensors when measuring. Unless max_size is not properly calculated, I don't see why this is an issue. * remove unnecessary "0x" before "%p" output * move measurement memory segment to upper region of the address space * update README.md * fix printf format warnings * add missing gguf_free in load_checkpoint_lora_file * load default rms_norm and rope parameters from base model * add gradient accumulation specify number accumulation steps with '--grad-acc N'. this will simulate a bigger batch size of grad_acc*batch. * fix tracking of train_samples and train_tokens * build : fix compile warnings * ggml : fix L-BFGS linesearch loop * improve finetune time measurement fix printf warnings on system where int64_t is (long int). change time datatypes to double because values get big with long training times. exclude file saving from time measurement. converge faster to actual time per iteration by removing very small first duration before first iteration was performed. fix bug in output of total training time, the reported value was 1000 times to small. * specify default lora rank with '--lora-r N' '--lora-r N' will specify default rank for all tensors '--rank-wq N', etc. will override this default rank for specific tensor types. * fix gradient accumulation bug where the same batch was used for each microstep * fix gradient accumulation bug where the same batch was used for each microstep * support grouped-query-attention in ggml_flash_attn and ggml_flash_attn_back k and v can now be repeated in q along ne[2] in forward pass just use modulo to compute k and v indices, like ik2 = iq2 % nek2. in backard pass this won't work as easy, because multiple threads will compete to accumulate to the same k->grad[:,ik1,ik2,ik3] and v->grad[:,iv1,iv2,iv3]. so we change the parallelization over q rows to be over k rows. this ensures non-overlapping (ik2,ik3) across threads. in each thread we then iterate over the number of repetitions of k/v in q to compute iq2 as iq2 = ik2 + irep*nek2. since ne2 is not the same for q,k and v we also change how the gradients are concatenated into the result tensor. additionally the offsets of gradq, gradk and gradv in the result tensor are now memory aligned. we also simplify the compute_backward part of flash_attn to use ggml_reshape instead of switching over the number of dimensions. this needs a small change to ggml_reshape, removing the assertion of second argument to be contiguous. since only the shape (ne) of the second reshape argument is of relevance, its memory layout (nb) is irrelevant -> it can very well be non-contiguous. change test-grad0 to also test for repeated k/v in q. this changes the rng and now results in small gradient differences in softmax. these solely come from using f16 exp table lookup in forward softmax: when temporarily changing softmax to use actual exp function, the reported gradient differences go away. gradient differences coming solely from f16 table lookup are acceptable. added a note to explain this. * add llama API functions to get grouped-query-attention n_head parameter 'n_head_kv'. * fix finetune to support grouped-query-attention (using flash-attention) note: ggml changes to ggml_out_prod are necessary to support grouped-query-attention without flash-attention. * support broadcastable a in out_prod(a, b) and backward pass of broadcasting mul_mat(a, b) * test broadcasting mul_mat backward pass * decouple random number generator of each operation test when changing one test the rng of others tests is not influenced anymore * add comment briefly describing what ggml_repeat_back does * simplify broadcasting mul_mat backward using ggml_repeat_back * add cgraph evaluation order member and corresponding enum type this controls in which order ggml_build_forward visits source nodes. by default the nodes are visited left to right, i.e. src[0] first. in some cases it is beneficial for ggml-alloc to visit in a different order. two possible orders are supported: left-to-right (src[0] first) and right-to-left (src[0] last). * measure max compute size for each cgraph eval order and use best order this can bring huge memory savings: e.g. codellama-34b with n_ctx=64, n_batch=1 goes from 92927.8mb down to 4627.6 MB * remove unused command line options * add sample start patterns and options to force new or by default resume last shuffling * update shuffle rng state on reshuffle * exclude known zero values from computations in flash_attn_f32 & flash_attn_back_f32 * remove probably unnecessary exception type flags from stringstream * pass correct max number of tokens to llama_tokenize * account for possible leading whitespace that will be added by tokenizer e.g. '\t' will be tokenized by llama spm tokenizer to [29871, 12] * use unrolled vec_mad in out_prod y is vec_mad result vec. x is vec_mad input vec. v is vec_mad input scalar. ggml_vec_mad_f32_unroll will internally loop over x and v with same y. GGML_VEC_MAD_UNROLL is by default defined to 32. This value is empirical optimized using performance test runs of out-prod in openllama-3b finetune with 256 context length and batch size 1. It gives 23% performance boost for out_prod. Full measurements of out-prod runtime in ms: unroll_xv unroll_yv 1 67014.643 87826.469 2 77117.552 89077.656 4 72091.311 109121.657 8 61077.543 88678.334 16 56914.67 79514.947 24 59024.595 84350.254 28 55952.446 83368.73 32 51476.658 85177.745 36 55973.792 84659.92 40 55139.616 93844.738 48 60736.392 93330.267 64 99856.878 116994.99 Second column is when unrollying yv instead of xv * set lora_alpha to value of lora_r if it is not set via command line otherwise only changing lora_r will change scaling of lora adapter used in prediction * reshuffle original sample order instead of the previous shuffled order otherwise resumed reshuffle will not result in same sample order * block tiling for out-prod inspired by mul-mat block sizes are empirically optimized roughly doubles the flops of out-prod * exclude some more known zero values from computations in flash_attn_f32 & flash_attn_back_f32 * add static keywords * remove outcommented old code * update train-text-from-scratch with tokenization, sample selection and shuffling from finetune * remove lbfgs related train parameters * move common train functions into common/train.[h|cpp] * move train state into struct train_state * move train data saving code into callback to unify code of opt_callback train_params are still different in finetune and train-text-from-scratch, so it can't yet be moved to train.h|cpp * move common train params into common/train * move common opt_callback into common/train * fix consume_common_train_arg * save and load head_count_kv in lora checkpoints * increase train_samples by used_samples instead of number of batches on batch can contain more than one sample when option "fill_with_next_samples" is used * fix usage of llama_tokenize * remove static from process_escape since we need it exposed in header * fix code formating of long function declarations * fix condition in load_train_state_gguf * use die("msg") instead of replace GGML_ASSERT(!"msg") or throw std::runtime_error("msg") * fix saving and loading of training type * remove terminating '\0' from tokenization (llama_tokenize is now passed the string length instead of relying on terminating '\0') * fix compile warnings * fix compile warnings * use new/delete for train_state instead of malloc/free using malloc may result in seg faults when trying to assign string fields * assert that sample_count > 0, avoiding division by zero * fix frand to return value in interval [0,1) * add train option "--sample-random-offsets" Use samples beginning at random offsets. The offset is only applied to the first sample in each batch context window. Together with "--fill-with-next-samples" this may help for training endless text generation. For example given a dataset containing samples "abcd", "ABCD", "0123". With context size of 8 and options "--fill-with-next-samples", "--no-separate-with-eos", "--no-separate-with-bos", the context windows of batches could only be filled with "abcdABCD", "ABCDabcd", "0123abcd", etc. With "--sample-random-offsets" it can also be filled with "23abcdAB", "bcd0123A", etc. * deduplicate code into function * remove n_rot hparam, as it must always be hparam.n_embd_head() * align code * assert correct base model tensor shapes * move some params from lora hparams into model hparams and load model params from gguf this equalizes the model definition in finetune and text-from-scratch and removes the need for additional llama api functions to get model parameters * remove now unnecessary llama API functions to get model params that where added by this PR * train-text-from-scratch: automatically allocate model tensors, remove option '--mem-model N' * train-text-from-scratch: automatically allocate opt context * train-text-from-scratch: automatically allocate input tensors * train-text-from-scratch: automatically allocate compute memory * remove unused options and equalize train-text-from-scratch with finetune * initialize opt->loss_after with zero * add export-lora program * remove trailing whitespace * add export-lora build in Makefile * remove unused struct tensor_info from export-lora * add export-lora build dependency to llama because it depends on common, which depends on llama * update finetune README.md * cancel optimization when specified number of epochs is completed * improve handling of export-lora arguments print errors and warnings when files could not be read or created * Fix export-lora.cpp "not enough space in the context's memory pool" (#1) * Fix export-lora.cpp "not enough space in the context's memory pool" Without this patch, export-lora would sometimes error with "not enough space in the context's memory pool (needed 656784, available 656800)". * increase required context size by 5*GGML_MEM_ALIGN instead of plain 16 --------- Co-authored-by: xaedes <xaedes@gmail.com> * improve handling of not yet supported tensor types --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: meatbag-18a <145869052+meatbag-18a@users.noreply.github.com>
2089 lines
74 KiB
C
2089 lines
74 KiB
C
#pragma once
|
|
|
|
//
|
|
// GGML Tensor Library
|
|
//
|
|
// This documentation is still a work in progress.
|
|
// If you wish some specific topics to be covered, feel free to drop a comment:
|
|
//
|
|
// https://github.com/ggerganov/whisper.cpp/issues/40
|
|
//
|
|
// ## Overview
|
|
//
|
|
// This library implements:
|
|
//
|
|
// - a set of tensor operations
|
|
// - automatic differentiation
|
|
// - basic optimization algorithms
|
|
//
|
|
// The aim of this library is to provide a minimalistic approach for various machine learning tasks. This includes,
|
|
// but is not limited to, the following:
|
|
//
|
|
// - linear regression
|
|
// - support vector machines
|
|
// - neural networks
|
|
//
|
|
// The library allows the user to define a certain function using the available tensor operations. This function
|
|
// definition is represented internally via a computation graph. Each tensor operation in the function definition
|
|
// corresponds to a node in the graph. Having the computation graph defined, the user can choose to compute the
|
|
// function's value and/or its gradient with respect to the input variables. Optionally, the function can be optimized
|
|
// using one of the available optimization algorithms.
|
|
//
|
|
// For example, here we define the function: f(x) = a*x^2 + b
|
|
//
|
|
// {
|
|
// struct ggml_init_params params = {
|
|
// .mem_size = 16*1024*1024,
|
|
// .mem_buffer = NULL,
|
|
// };
|
|
//
|
|
// // memory allocation happens here
|
|
// struct ggml_context * ctx = ggml_init(params);
|
|
//
|
|
// struct ggml_tensor * x = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1);
|
|
//
|
|
// ggml_set_param(ctx, x); // x is an input variable
|
|
//
|
|
// struct ggml_tensor * a = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1);
|
|
// struct ggml_tensor * b = ggml_new_tensor_1d(ctx, GGML_TYPE_F32, 1);
|
|
// struct ggml_tensor * x2 = ggml_mul(ctx, x, x);
|
|
// struct ggml_tensor * f = ggml_add(ctx, ggml_mul(ctx, a, x2), b);
|
|
//
|
|
// ...
|
|
// }
|
|
//
|
|
// Notice that the function definition above does not involve any actual computation. The computation is performed only
|
|
// when the user explicitly requests it. For example, to compute the function's value at x = 2.0:
|
|
//
|
|
// {
|
|
// ...
|
|
//
|
|
// struct ggml_cgraph gf = ggml_build_forward(f);
|
|
//
|
|
// // set the input variable and parameter values
|
|
// ggml_set_f32(x, 2.0f);
|
|
// ggml_set_f32(a, 3.0f);
|
|
// ggml_set_f32(b, 4.0f);
|
|
//
|
|
// ggml_graph_compute_with_ctx(ctx, &gf, n_threads);
|
|
//
|
|
// printf("f = %f\n", ggml_get_f32_1d(f, 0));
|
|
//
|
|
// ...
|
|
// }
|
|
//
|
|
// The actual computation is performed in the ggml_graph_compute() function.
|
|
//
|
|
// The ggml_new_tensor_...() functions create new tensors. They are allocated in the memory buffer provided to the
|
|
// ggml_init() function. You have to be careful not to exceed the memory buffer size. Therefore, you have to know
|
|
// in advance how much memory you need for your computation. Alternatively, you can allocate a large enough memory
|
|
// and after defining the computation graph, call the ggml_used_mem() function to find out how much memory was
|
|
// actually needed.
|
|
//
|
|
// The ggml_set_param() function marks a tensor as an input variable. This is used by the automatic
|
|
// differentiation and optimization algorithms.
|
|
//
|
|
// The described approach allows to define the function graph once and then compute its forward or backward graphs
|
|
// multiple times. All computations will use the same memory buffer allocated in the ggml_init() function. This way
|
|
// the user can avoid the memory allocation overhead at runtime.
|
|
//
|
|
// The library supports multi-dimensional tensors - up to 4 dimensions. The FP16 and FP32 data types are first class
|
|
// citizens, but in theory the library can be extended to support FP8 and integer data types.
|
|
//
|
|
// Each tensor operation produces a new tensor. Initially the library was envisioned to support only the use of unary
|
|
// and binary operations. Most of the available operations fall into one of these two categories. With time, it became
|
|
// clear that the library needs to support more complex operations. The way to support these operations is not clear
|
|
// yet, but a few examples are demonstrated in the following operations:
|
|
//
|
|
// - ggml_permute()
|
|
// - ggml_conv_1d_1s()
|
|
// - ggml_conv_1d_2s()
|
|
//
|
|
// For each tensor operator, the library implements a forward and backward computation function. The forward function
|
|
// computes the output tensor value given the input tensor values. The backward function computes the adjoint of the
|
|
// input tensors given the adjoint of the output tensor. For a detailed explanation of what this means, take a
|
|
// calculus class, or watch the following video:
|
|
//
|
|
// What is Automatic Differentiation?
|
|
// https://www.youtube.com/watch?v=wG_nF1awSSY
|
|
//
|
|
//
|
|
// ## Tensor data (struct ggml_tensor)
|
|
//
|
|
// The tensors are stored in memory via the ggml_tensor struct. The structure provides information about the size of
|
|
// the tensor, the data type, and the memory buffer where the tensor data is stored. Additionally, it contains
|
|
// pointers to the "source" tensors - i.e. the tensors that were used to compute the current tensor. For example:
|
|
//
|
|
// {
|
|
// struct ggml_tensor * c = ggml_add(ctx, a, b);
|
|
//
|
|
// assert(c->src[0] == a);
|
|
// assert(c->src[1] == b);
|
|
// }
|
|
//
|
|
// The multi-dimensional tensors are stored in row-major order. The ggml_tensor struct contains fields for the
|
|
// number of elements in each dimension ("ne") as well as the number of bytes ("nb", a.k.a. stride). This allows
|
|
// to store tensors that are not contiguous in memory, which is useful for operations such as transposition and
|
|
// permutation. All tensor operations have to take the stride into account and not assume that the tensor is
|
|
// contiguous in memory.
|
|
//
|
|
// The data of the tensor is accessed via the "data" pointer. For example:
|
|
//
|
|
// {
|
|
// const int nx = 2;
|
|
// const int ny = 3;
|
|
//
|
|
// struct ggml_tensor * a = ggml_new_tensor_2d(ctx, GGML_TYPE_F32, nx, ny);
|
|
//
|
|
// for (int y = 0; y < ny; y++) {
|
|
// for (int x = 0; x < nx; x++) {
|
|
// *(float *) ((char *) a->data + y*a->nb[1] + x*a->nb[0]) = x + y;
|
|
// }
|
|
// }
|
|
//
|
|
// ...
|
|
// }
|
|
//
|
|
// Alternatively, there are helper functions, such as ggml_get_f32_1d() and ggml_set_f32_1d() that can be used.
|
|
//
|
|
// ## The matrix multiplication operator (ggml_mul_mat)
|
|
//
|
|
// TODO
|
|
//
|
|
//
|
|
// ## Multi-threading
|
|
//
|
|
// TODO
|
|
//
|
|
//
|
|
// ## Overview of ggml.c
|
|
//
|
|
// TODO
|
|
//
|
|
//
|
|
// ## SIMD optimizations
|
|
//
|
|
// TODO
|
|
//
|
|
//
|
|
// ## Debugging ggml
|
|
//
|
|
// TODO
|
|
//
|
|
//
|
|
|
|
#ifdef GGML_SHARED
|
|
# if defined(_WIN32) && !defined(__MINGW32__)
|
|
# ifdef GGML_BUILD
|
|
# define GGML_API __declspec(dllexport)
|
|
# else
|
|
# define GGML_API __declspec(dllimport)
|
|
# endif
|
|
# else
|
|
# define GGML_API __attribute__ ((visibility ("default")))
|
|
# endif
|
|
#else
|
|
# define GGML_API
|
|
#endif
|
|
|
|
// TODO: support for clang
|
|
#ifdef __GNUC__
|
|
# define GGML_DEPRECATED(func, hint) func __attribute__((deprecated(hint)))
|
|
#elif defined(_MSC_VER)
|
|
# define GGML_DEPRECATED(func, hint) __declspec(deprecated(hint)) func
|
|
#else
|
|
# define GGML_DEPRECATED(func, hint) func
|
|
#endif
|
|
|
|
#ifndef __GNUC__
|
|
# define GGML_ATTRIBUTE_FORMAT(...)
|
|
#elif defined(__MINGW32__)
|
|
# define GGML_ATTRIBUTE_FORMAT(...) __attribute__((format(gnu_printf, __VA_ARGS__)))
|
|
#else
|
|
# define GGML_ATTRIBUTE_FORMAT(...) __attribute__((format(printf, __VA_ARGS__)))
|
|
#endif
|
|
|
|
#include <stdint.h>
|
|
#include <stddef.h>
|
|
#include <stdbool.h>
|
|
|
|
#define GGML_FILE_MAGIC 0x67676d6c // "ggml"
|
|
#define GGML_FILE_VERSION 1
|
|
|
|
#define GGML_QNT_VERSION 2 // bump this on quantization format changes
|
|
#define GGML_QNT_VERSION_FACTOR 1000 // do not change this
|
|
|
|
#define GGML_MAX_DIMS 4
|
|
#define GGML_MAX_NODES 16384
|
|
#define GGML_MAX_PARAMS 1024
|
|
#define GGML_MAX_CONTEXTS 64
|
|
#define GGML_MAX_SRC 6
|
|
#define GGML_MAX_NAME 64
|
|
#define GGML_MAX_OP_PARAMS 32
|
|
#define GGML_DEFAULT_N_THREADS 4
|
|
|
|
#if UINTPTR_MAX == 0xFFFFFFFF
|
|
#define GGML_MEM_ALIGN 4
|
|
#else
|
|
#define GGML_MEM_ALIGN 16
|
|
#endif
|
|
|
|
#define GGML_EXIT_SUCCESS 0
|
|
#define GGML_EXIT_ABORTED 1
|
|
|
|
#define GGUF_MAGIC 0x46554747 // "GGUF"
|
|
#define GGUF_VERSION 2
|
|
|
|
#define GGUF_DEFAULT_ALIGNMENT 32
|
|
|
|
#define GGML_UNUSED(x) (void)(x)
|
|
|
|
#define GGML_PAD(x, n) (((x) + (n) - 1) & ~((n) - 1))
|
|
|
|
#define GGML_ASSERT(x) \
|
|
do { \
|
|
if (!(x)) { \
|
|
fprintf(stderr, "GGML_ASSERT: %s:%d: %s\n", __FILE__, __LINE__, #x); \
|
|
abort(); \
|
|
} \
|
|
} while (0)
|
|
|
|
// used to copy the number of elements and stride in bytes of tensors into local variables.
|
|
// main purpose is to reduce code duplication and improve readability.
|
|
//
|
|
// example:
|
|
//
|
|
// GGML_TENSOR_LOCALS(int64_t, ne1, src1, ne);
|
|
// GGML_TENSOR_LOCALS(size_t, nb1, src1, nb);
|
|
//
|
|
#define GGML_TENSOR_LOCALS_1(type, prefix, pointer, array) \
|
|
const type prefix##0 = (pointer)->array[0]; \
|
|
GGML_UNUSED(prefix##0);
|
|
#define GGML_TENSOR_LOCALS_2(type, prefix, pointer, array) \
|
|
GGML_TENSOR_LOCALS_1 (type, prefix, pointer, array) \
|
|
const type prefix##1 = (pointer)->array[1]; \
|
|
GGML_UNUSED(prefix##1);
|
|
#define GGML_TENSOR_LOCALS_3(type, prefix, pointer, array) \
|
|
GGML_TENSOR_LOCALS_2 (type, prefix, pointer, array) \
|
|
const type prefix##2 = (pointer)->array[2]; \
|
|
GGML_UNUSED(prefix##2);
|
|
#define GGML_TENSOR_LOCALS(type, prefix, pointer, array) \
|
|
GGML_TENSOR_LOCALS_3 (type, prefix, pointer, array) \
|
|
const type prefix##3 = (pointer)->array[3]; \
|
|
GGML_UNUSED(prefix##3);
|
|
|
|
#ifdef __cplusplus
|
|
extern "C" {
|
|
#endif
|
|
|
|
#if defined(__ARM_NEON) && defined(__CUDACC__)
|
|
typedef half ggml_fp16_t;
|
|
#elif defined(__ARM_NEON)
|
|
typedef __fp16 ggml_fp16_t;
|
|
#else
|
|
typedef uint16_t ggml_fp16_t;
|
|
#endif
|
|
|
|
// convert FP16 <-> FP32
|
|
GGML_API float ggml_fp16_to_fp32(ggml_fp16_t x);
|
|
GGML_API ggml_fp16_t ggml_fp32_to_fp16(float x);
|
|
|
|
GGML_API void ggml_fp16_to_fp32_row(const ggml_fp16_t * x, float * y, int n);
|
|
GGML_API void ggml_fp32_to_fp16_row(const float * x, ggml_fp16_t * y, int n);
|
|
|
|
struct ggml_object;
|
|
struct ggml_context;
|
|
|
|
enum ggml_type {
|
|
GGML_TYPE_F32 = 0,
|
|
GGML_TYPE_F16 = 1,
|
|
GGML_TYPE_Q4_0 = 2,
|
|
GGML_TYPE_Q4_1 = 3,
|
|
// GGML_TYPE_Q4_2 = 4, support has been removed
|
|
// GGML_TYPE_Q4_3 (5) support has been removed
|
|
GGML_TYPE_Q5_0 = 6,
|
|
GGML_TYPE_Q5_1 = 7,
|
|
GGML_TYPE_Q8_0 = 8,
|
|
GGML_TYPE_Q8_1 = 9,
|
|
// k-quantizations
|
|
GGML_TYPE_Q2_K = 10,
|
|
GGML_TYPE_Q3_K = 11,
|
|
GGML_TYPE_Q4_K = 12,
|
|
GGML_TYPE_Q5_K = 13,
|
|
GGML_TYPE_Q6_K = 14,
|
|
GGML_TYPE_Q8_K = 15,
|
|
GGML_TYPE_I8,
|
|
GGML_TYPE_I16,
|
|
GGML_TYPE_I32,
|
|
GGML_TYPE_COUNT,
|
|
};
|
|
|
|
enum ggml_backend {
|
|
GGML_BACKEND_CPU = 0,
|
|
GGML_BACKEND_GPU = 10,
|
|
GGML_BACKEND_GPU_SPLIT = 20,
|
|
};
|
|
|
|
// model file types
|
|
enum ggml_ftype {
|
|
GGML_FTYPE_UNKNOWN = -1,
|
|
GGML_FTYPE_ALL_F32 = 0,
|
|
GGML_FTYPE_MOSTLY_F16 = 1, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q4_0 = 2, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q4_1 = 3, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q4_1_SOME_F16 = 4, // tok_embeddings.weight and output.weight are F16
|
|
GGML_FTYPE_MOSTLY_Q8_0 = 7, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q5_0 = 8, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q5_1 = 9, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q2_K = 10, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q3_K = 11, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q4_K = 12, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q5_K = 13, // except 1d tensors
|
|
GGML_FTYPE_MOSTLY_Q6_K = 14, // except 1d tensors
|
|
};
|
|
|
|
// available tensor operations:
|
|
enum ggml_op {
|
|
GGML_OP_NONE = 0,
|
|
|
|
GGML_OP_DUP,
|
|
GGML_OP_ADD,
|
|
GGML_OP_ADD1,
|
|
GGML_OP_ACC,
|
|
GGML_OP_SUB,
|
|
GGML_OP_MUL,
|
|
GGML_OP_DIV,
|
|
GGML_OP_SQR,
|
|
GGML_OP_SQRT,
|
|
GGML_OP_LOG,
|
|
GGML_OP_SUM,
|
|
GGML_OP_SUM_ROWS,
|
|
GGML_OP_MEAN,
|
|
GGML_OP_ARGMAX,
|
|
GGML_OP_REPEAT,
|
|
GGML_OP_REPEAT_BACK,
|
|
GGML_OP_CONCAT,
|
|
GGML_OP_SILU_BACK,
|
|
GGML_OP_NORM, // normalize
|
|
GGML_OP_RMS_NORM,
|
|
GGML_OP_RMS_NORM_BACK,
|
|
GGML_OP_GROUP_NORM,
|
|
|
|
GGML_OP_MUL_MAT,
|
|
GGML_OP_OUT_PROD,
|
|
|
|
GGML_OP_SCALE,
|
|
GGML_OP_SET,
|
|
GGML_OP_CPY,
|
|
GGML_OP_CONT,
|
|
GGML_OP_RESHAPE,
|
|
GGML_OP_VIEW,
|
|
GGML_OP_PERMUTE,
|
|
GGML_OP_TRANSPOSE,
|
|
GGML_OP_GET_ROWS,
|
|
GGML_OP_GET_ROWS_BACK,
|
|
GGML_OP_DIAG,
|
|
GGML_OP_DIAG_MASK_INF,
|
|
GGML_OP_DIAG_MASK_ZERO,
|
|
GGML_OP_SOFT_MAX,
|
|
GGML_OP_SOFT_MAX_BACK,
|
|
GGML_OP_ROPE,
|
|
GGML_OP_ROPE_BACK,
|
|
GGML_OP_ALIBI,
|
|
GGML_OP_CLAMP,
|
|
GGML_OP_CONV_1D,
|
|
GGML_OP_CONV_2D,
|
|
GGML_OP_CONV_TRANSPOSE_2D,
|
|
GGML_OP_POOL_1D,
|
|
GGML_OP_POOL_2D,
|
|
|
|
GGML_OP_UPSCALE, // nearest interpolate
|
|
|
|
GGML_OP_FLASH_ATTN,
|
|
GGML_OP_FLASH_FF,
|
|
GGML_OP_FLASH_ATTN_BACK,
|
|
GGML_OP_WIN_PART,
|
|
GGML_OP_WIN_UNPART,
|
|
GGML_OP_GET_REL_POS,
|
|
GGML_OP_ADD_REL_POS,
|
|
|
|
GGML_OP_UNARY,
|
|
|
|
GGML_OP_MAP_UNARY,
|
|
GGML_OP_MAP_BINARY,
|
|
|
|
GGML_OP_MAP_CUSTOM1_F32,
|
|
GGML_OP_MAP_CUSTOM2_F32,
|
|
GGML_OP_MAP_CUSTOM3_F32,
|
|
|
|
GGML_OP_MAP_CUSTOM1,
|
|
GGML_OP_MAP_CUSTOM2,
|
|
GGML_OP_MAP_CUSTOM3,
|
|
|
|
GGML_OP_CROSS_ENTROPY_LOSS,
|
|
GGML_OP_CROSS_ENTROPY_LOSS_BACK,
|
|
|
|
GGML_OP_COUNT,
|
|
};
|
|
|
|
enum ggml_unary_op {
|
|
GGML_UNARY_OP_ABS,
|
|
GGML_UNARY_OP_SGN,
|
|
GGML_UNARY_OP_NEG,
|
|
GGML_UNARY_OP_STEP,
|
|
GGML_UNARY_OP_TANH,
|
|
GGML_UNARY_OP_ELU,
|
|
GGML_UNARY_OP_RELU,
|
|
GGML_UNARY_OP_GELU,
|
|
GGML_UNARY_OP_GELU_QUICK,
|
|
GGML_UNARY_OP_SILU,
|
|
};
|
|
|
|
enum ggml_object_type {
|
|
GGML_OBJECT_TENSOR,
|
|
GGML_OBJECT_GRAPH,
|
|
GGML_OBJECT_WORK_BUFFER
|
|
};
|
|
|
|
enum ggml_log_level {
|
|
GGML_LOG_LEVEL_ERROR = 2,
|
|
GGML_LOG_LEVEL_WARN = 3,
|
|
GGML_LOG_LEVEL_INFO = 4
|
|
};
|
|
|
|
// ggml object
|
|
struct ggml_object {
|
|
size_t offs;
|
|
size_t size;
|
|
|
|
struct ggml_object * next;
|
|
|
|
enum ggml_object_type type;
|
|
|
|
char padding[4];
|
|
};
|
|
|
|
static const size_t GGML_OBJECT_SIZE = sizeof(struct ggml_object);
|
|
|
|
// n-dimensional tensor
|
|
struct ggml_tensor {
|
|
enum ggml_type type;
|
|
enum ggml_backend backend;
|
|
|
|
int n_dims;
|
|
int64_t ne[GGML_MAX_DIMS]; // number of elements
|
|
size_t nb[GGML_MAX_DIMS]; // stride in bytes:
|
|
// nb[0] = sizeof(type)
|
|
// nb[1] = nb[0] * ne[0] + padding
|
|
// nb[i] = nb[i-1] * ne[i-1]
|
|
|
|
// compute data
|
|
enum ggml_op op;
|
|
|
|
// op params - allocated as int32_t for alignment
|
|
int32_t op_params[GGML_MAX_OP_PARAMS / sizeof(int32_t)];
|
|
|
|
bool is_param;
|
|
|
|
struct ggml_tensor * grad;
|
|
struct ggml_tensor * src[GGML_MAX_SRC];
|
|
|
|
// performance
|
|
int perf_runs;
|
|
int64_t perf_cycles;
|
|
int64_t perf_time_us;
|
|
|
|
struct ggml_tensor * view_src;
|
|
size_t view_offs;
|
|
|
|
void * data;
|
|
|
|
char name[GGML_MAX_NAME];
|
|
|
|
void * extra; // extra things e.g. for ggml-cuda.cu
|
|
|
|
char padding[4];
|
|
};
|
|
|
|
static const size_t GGML_TENSOR_SIZE = sizeof(struct ggml_tensor);
|
|
|
|
// the compute plan that needs to be prepared for ggml_graph_compute()
|
|
// since https://github.com/ggerganov/ggml/issues/287
|
|
struct ggml_cplan {
|
|
size_t work_size; // size of work buffer, calculated by `ggml_graph_plan()`
|
|
uint8_t * work_data; // work buffer, to be allocated by caller before calling to `ggml_graph_compute()`
|
|
|
|
int n_threads;
|
|
|
|
// the `n_tasks` of nodes, 1:1 mapping to cgraph nodes
|
|
int n_tasks[GGML_MAX_NODES];
|
|
|
|
// abort ggml_graph_compute when true
|
|
bool (*abort_callback)(void * data);
|
|
void * abort_callback_data;
|
|
};
|
|
|
|
// next prime after GGML_MAX_NODES
|
|
// #define GGML_GRAPH_HASHTABLE_SIZE 4099
|
|
// next prime after GGML_MAX_NODES * 2 (nodes + leafs)
|
|
// #define GGML_GRAPH_HASHTABLE_SIZE 8273
|
|
// #define GGML_GRAPH_HASHTABLE_SIZE 16411
|
|
#define GGML_GRAPH_HASHTABLE_SIZE 32771
|
|
|
|
enum ggml_cgraph_eval_order {
|
|
GGML_CGRAPH_EVAL_ORDER_LEFT_TO_RIGHT = 0,
|
|
GGML_CGRAPH_EVAL_ORDER_RIGHT_TO_LEFT,
|
|
GGML_CGRAPH_EVAL_ORDER_COUNT
|
|
};
|
|
|
|
// computation graph
|
|
struct ggml_cgraph {
|
|
int n_nodes;
|
|
int n_leafs;
|
|
|
|
struct ggml_tensor * nodes[GGML_MAX_NODES];
|
|
struct ggml_tensor * grads[GGML_MAX_NODES];
|
|
struct ggml_tensor * leafs[GGML_MAX_NODES];
|
|
|
|
void * visited_hash_table[GGML_GRAPH_HASHTABLE_SIZE];
|
|
|
|
enum ggml_cgraph_eval_order order;
|
|
|
|
// performance
|
|
int perf_runs;
|
|
int64_t perf_cycles;
|
|
int64_t perf_time_us;
|
|
};
|
|
|
|
static const size_t GGML_GRAPH_SIZE = sizeof(struct ggml_cgraph);
|
|
|
|
// scratch buffer
|
|
struct ggml_scratch {
|
|
size_t offs;
|
|
size_t size;
|
|
void * data;
|
|
};
|
|
|
|
struct ggml_init_params {
|
|
// memory pool
|
|
size_t mem_size; // bytes
|
|
void * mem_buffer; // if NULL, memory will be allocated internally
|
|
bool no_alloc; // don't allocate memory for the tensor data
|
|
};
|
|
|
|
|
|
// compute types
|
|
|
|
// NOTE: the INIT or FINALIZE pass is not scheduled unless explicitly enabled.
|
|
// This behavior was changed since https://github.com/ggerganov/llama.cpp/pull/1995.
|
|
enum ggml_task_type {
|
|
GGML_TASK_INIT = 0,
|
|
GGML_TASK_COMPUTE,
|
|
GGML_TASK_FINALIZE,
|
|
};
|
|
|
|
struct ggml_compute_params {
|
|
enum ggml_task_type type;
|
|
|
|
// ith = thread index, nth = number of threads
|
|
int ith, nth;
|
|
|
|
// work buffer for all threads
|
|
size_t wsize;
|
|
void * wdata;
|
|
};
|
|
|
|
// misc
|
|
|
|
GGML_API void ggml_time_init(void); // call this once at the beginning of the program
|
|
GGML_API int64_t ggml_time_ms(void);
|
|
GGML_API int64_t ggml_time_us(void);
|
|
GGML_API int64_t ggml_cycles(void);
|
|
GGML_API int64_t ggml_cycles_per_ms(void);
|
|
|
|
GGML_API void ggml_numa_init(void); // call once for better performance on NUMA systems
|
|
GGML_API bool ggml_is_numa(void); // true if init detected that system has >1 NUMA node
|
|
|
|
GGML_API void ggml_print_object (const struct ggml_object * obj);
|
|
GGML_API void ggml_print_objects(const struct ggml_context * ctx);
|
|
|
|
GGML_API int64_t ggml_nelements (const struct ggml_tensor * tensor);
|
|
GGML_API int64_t ggml_nrows (const struct ggml_tensor * tensor);
|
|
GGML_API size_t ggml_nbytes (const struct ggml_tensor * tensor);
|
|
GGML_API size_t ggml_nbytes_pad (const struct ggml_tensor * tensor); // same as ggml_nbytes() but padded to GGML_MEM_ALIGN
|
|
GGML_API size_t ggml_nbytes_split(const struct ggml_tensor * tensor, int nrows_split);
|
|
|
|
GGML_API int ggml_blck_size (enum ggml_type type);
|
|
GGML_API size_t ggml_type_size (enum ggml_type type); // size in bytes for all elements in a block
|
|
GGML_API float ggml_type_sizef(enum ggml_type type); // ggml_type_size()/ggml_blck_size() as float
|
|
|
|
GGML_API const char * ggml_type_name(enum ggml_type type);
|
|
GGML_API const char * ggml_op_name (enum ggml_op op);
|
|
GGML_API const char * ggml_op_symbol(enum ggml_op op);
|
|
|
|
GGML_API size_t ggml_element_size(const struct ggml_tensor * tensor);
|
|
|
|
GGML_API bool ggml_is_quantized(enum ggml_type type);
|
|
|
|
// TODO: temporary until model loading of ggml examples is refactored
|
|
GGML_API enum ggml_type ggml_ftype_to_ggml_type(enum ggml_ftype ftype);
|
|
|
|
GGML_API bool ggml_is_transposed(const struct ggml_tensor * tensor);
|
|
GGML_API bool ggml_is_contiguous(const struct ggml_tensor * tensor);
|
|
GGML_API bool ggml_is_permuted (const struct ggml_tensor * tensor);
|
|
|
|
GGML_API bool ggml_are_same_shape(const struct ggml_tensor * t0, const struct ggml_tensor * t1);
|
|
|
|
// use this to compute the memory overhead of a tensor
|
|
GGML_API size_t ggml_tensor_overhead(void);
|
|
|
|
// main
|
|
|
|
GGML_API struct ggml_context * ggml_init(struct ggml_init_params params);
|
|
GGML_API void ggml_free(struct ggml_context * ctx);
|
|
|
|
GGML_API size_t ggml_used_mem(const struct ggml_context * ctx);
|
|
|
|
GGML_API size_t ggml_set_scratch (struct ggml_context * ctx, struct ggml_scratch scratch);
|
|
GGML_API bool ggml_get_no_alloc(struct ggml_context * ctx);
|
|
GGML_API void ggml_set_no_alloc(struct ggml_context * ctx, bool no_alloc);
|
|
|
|
GGML_API void * ggml_get_mem_buffer (const struct ggml_context * ctx);
|
|
GGML_API size_t ggml_get_mem_size (const struct ggml_context * ctx);
|
|
GGML_API size_t ggml_get_max_tensor_size(const struct ggml_context * ctx);
|
|
|
|
GGML_API struct ggml_tensor * ggml_new_tensor(
|
|
struct ggml_context * ctx,
|
|
enum ggml_type type,
|
|
int n_dims,
|
|
const int64_t *ne);
|
|
|
|
GGML_API struct ggml_tensor * ggml_new_tensor_1d(
|
|
struct ggml_context * ctx,
|
|
enum ggml_type type,
|
|
int64_t ne0);
|
|
|
|
GGML_API struct ggml_tensor * ggml_new_tensor_2d(
|
|
struct ggml_context * ctx,
|
|
enum ggml_type type,
|
|
int64_t ne0,
|
|
int64_t ne1);
|
|
|
|
GGML_API struct ggml_tensor * ggml_new_tensor_3d(
|
|
struct ggml_context * ctx,
|
|
enum ggml_type type,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2);
|
|
|
|
GGML_API struct ggml_tensor * ggml_new_tensor_4d(
|
|
struct ggml_context * ctx,
|
|
enum ggml_type type,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2,
|
|
int64_t ne3);
|
|
|
|
GGML_API struct ggml_tensor * ggml_new_i32(struct ggml_context * ctx, int32_t value);
|
|
GGML_API struct ggml_tensor * ggml_new_f32(struct ggml_context * ctx, float value);
|
|
|
|
GGML_API struct ggml_tensor * ggml_dup_tensor (struct ggml_context * ctx, const struct ggml_tensor * src);
|
|
GGML_API struct ggml_tensor * ggml_view_tensor(struct ggml_context * ctx, struct ggml_tensor * src);
|
|
|
|
GGML_API struct ggml_tensor * ggml_get_tensor(struct ggml_context * ctx, const char * name);
|
|
|
|
GGML_API struct ggml_tensor * ggml_set_zero(struct ggml_tensor * tensor);
|
|
GGML_API struct ggml_tensor * ggml_set_i32 (struct ggml_tensor * tensor, int32_t value);
|
|
GGML_API struct ggml_tensor * ggml_set_f32 (struct ggml_tensor * tensor, float value);
|
|
|
|
// Converts a flat index into coordinates
|
|
GGML_API void ggml_unravel_index(const struct ggml_tensor * tensor, int64_t i, int64_t * i0, int64_t * i1, int64_t * i2, int64_t * i3);
|
|
|
|
GGML_API int32_t ggml_get_i32_1d(const struct ggml_tensor * tensor, int i);
|
|
GGML_API void ggml_set_i32_1d(const struct ggml_tensor * tensor, int i, int32_t value);
|
|
|
|
GGML_API int32_t ggml_get_i32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3);
|
|
GGML_API void ggml_set_i32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3, int32_t value);
|
|
|
|
GGML_API float ggml_get_f32_1d(const struct ggml_tensor * tensor, int i);
|
|
GGML_API void ggml_set_f32_1d(const struct ggml_tensor * tensor, int i, float value);
|
|
|
|
GGML_API float ggml_get_f32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3);
|
|
GGML_API void ggml_set_f32_nd(const struct ggml_tensor * tensor, int i0, int i1, int i2, int i3, float value);
|
|
|
|
GGML_API void * ggml_get_data (const struct ggml_tensor * tensor);
|
|
GGML_API float * ggml_get_data_f32(const struct ggml_tensor * tensor);
|
|
|
|
GGML_API enum ggml_unary_op ggml_get_unary_op(const struct ggml_tensor * tensor);
|
|
|
|
GGML_API const char * ggml_get_name (const struct ggml_tensor * tensor);
|
|
GGML_API struct ggml_tensor * ggml_set_name ( struct ggml_tensor * tensor, const char * name);
|
|
GGML_ATTRIBUTE_FORMAT(2, 3)
|
|
GGML_API struct ggml_tensor * ggml_format_name( struct ggml_tensor * tensor, const char * fmt, ...);
|
|
|
|
//
|
|
// operations on tensors with backpropagation
|
|
//
|
|
|
|
GGML_API struct ggml_tensor * ggml_dup(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_dup_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_add(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_add_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_add_cast(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
enum ggml_type type);
|
|
|
|
GGML_API struct ggml_tensor * ggml_add1(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_add1_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_acc(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t nb1,
|
|
size_t nb2,
|
|
size_t nb3,
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_acc_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t nb1,
|
|
size_t nb2,
|
|
size_t nb3,
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sub(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sub_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_mul(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_mul_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_div(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_div_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sqr(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sqr_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sqrt(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sqrt_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_log(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_log_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// return scalar
|
|
GGML_API struct ggml_tensor * ggml_sum(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// sums along rows, with input shape [a,b,c,d] return shape [1,b,c,d]
|
|
GGML_API struct ggml_tensor * ggml_sum_rows(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// mean along rows
|
|
GGML_API struct ggml_tensor * ggml_mean(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// argmax along rows
|
|
GGML_API struct ggml_tensor * ggml_argmax(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// if a is the same shape as b, and a is not parameter, return a
|
|
// otherwise, return a new tensor: repeat(a) to fit in b
|
|
GGML_API struct ggml_tensor * ggml_repeat(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// sums repetitions in a into shape of b
|
|
GGML_API struct ggml_tensor * ggml_repeat_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// concat a and b on dim 2
|
|
// used in stable-diffusion
|
|
GGML_API struct ggml_tensor * ggml_concat(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_abs(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_abs_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sgn(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_sgn_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_neg(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_neg_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_step(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_step_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_tanh(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_tanh_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_elu(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_elu_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_relu(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_relu_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// TODO: double-check this computation is correct
|
|
GGML_API struct ggml_tensor * ggml_gelu(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_gelu_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_gelu_quick(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_gelu_quick_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_silu(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_silu_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// a - x
|
|
// b - dy
|
|
GGML_API struct ggml_tensor * ggml_silu_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// normalize along rows
|
|
GGML_API struct ggml_tensor * ggml_norm(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
float eps);
|
|
|
|
GGML_API struct ggml_tensor * ggml_norm_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
float eps);
|
|
|
|
GGML_API struct ggml_tensor * ggml_rms_norm(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
float eps);
|
|
|
|
GGML_API struct ggml_tensor * ggml_rms_norm_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
float eps);
|
|
|
|
// group normalize along ne0*ne1*n_groups
|
|
// used in stable-diffusion
|
|
// TODO: eps is hardcoded to 1e-6 for now
|
|
GGML_API struct ggml_tensor * ggml_group_norm(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_groups);
|
|
|
|
GGML_API struct ggml_tensor * ggml_group_norm_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_groups);
|
|
|
|
// a - x
|
|
// b - dy
|
|
GGML_API struct ggml_tensor * ggml_rms_norm_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
float eps);
|
|
|
|
// A: n columns, m rows
|
|
// B: n columns, p rows (i.e. we transpose it internally)
|
|
// result is m columns, p rows
|
|
GGML_API struct ggml_tensor * ggml_mul_mat(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// A: m columns, n rows,
|
|
// B: p columns, n rows,
|
|
// result is m columns, p rows
|
|
GGML_API struct ggml_tensor * ggml_out_prod(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
//
|
|
// operations on tensors without backpropagation
|
|
//
|
|
|
|
GGML_API struct ggml_tensor * ggml_scale(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_scale_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// b -> view(a,offset,nb1,nb2,3), return modified a
|
|
GGML_API struct ggml_tensor * ggml_set(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t nb1,
|
|
size_t nb2,
|
|
size_t nb3,
|
|
size_t offset);
|
|
|
|
// b -> view(a,offset,nb1,nb2,3), return view(a)
|
|
GGML_API struct ggml_tensor * ggml_set_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t nb1,
|
|
size_t nb2,
|
|
size_t nb3,
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_set_1d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_set_1d_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t offset);
|
|
|
|
// b -> view(a,offset,nb1,nb2,3), return modified a
|
|
GGML_API struct ggml_tensor * ggml_set_2d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t nb1,
|
|
size_t offset);
|
|
|
|
// b -> view(a,offset,nb1,nb2,3), return view(a)
|
|
GGML_API struct ggml_tensor * ggml_set_2d_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
size_t nb1,
|
|
size_t offset);
|
|
|
|
// a -> b, return view(b)
|
|
GGML_API struct ggml_tensor * ggml_cpy(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// a -> b, in-place, return view(b)
|
|
GGML_API struct ggml_tensor * ggml_cpy_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// make contiguous
|
|
GGML_API struct ggml_tensor * ggml_cont(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// make contiguous, in-place
|
|
GGML_API struct ggml_tensor * ggml_cont_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// make contiguous, with new shape
|
|
GGML_API struct ggml_tensor * ggml_cont_1d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0);
|
|
|
|
GGML_API struct ggml_tensor * ggml_cont_2d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1);
|
|
|
|
GGML_API struct ggml_tensor * ggml_cont_3d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2);
|
|
|
|
GGML_API struct ggml_tensor * ggml_cont_4d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2,
|
|
int64_t ne3);
|
|
|
|
// return view(a), b specifies the new shape
|
|
// TODO: when we start computing gradient, make a copy instead of view
|
|
GGML_API struct ggml_tensor * ggml_reshape(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// return view(a)
|
|
// TODO: when we start computing gradient, make a copy instead of view
|
|
GGML_API struct ggml_tensor * ggml_reshape_1d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0);
|
|
|
|
GGML_API struct ggml_tensor * ggml_reshape_2d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1);
|
|
|
|
// return view(a)
|
|
// TODO: when we start computing gradient, make a copy instead of view
|
|
GGML_API struct ggml_tensor * ggml_reshape_3d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2);
|
|
|
|
GGML_API struct ggml_tensor * ggml_reshape_4d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2,
|
|
int64_t ne3);
|
|
|
|
// offset in bytes
|
|
GGML_API struct ggml_tensor * ggml_view_1d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_view_2d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
size_t nb1, // row stride in bytes
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_view_3d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2,
|
|
size_t nb1, // row stride in bytes
|
|
size_t nb2, // slice stride in bytes
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_view_4d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int64_t ne0,
|
|
int64_t ne1,
|
|
int64_t ne2,
|
|
int64_t ne3,
|
|
size_t nb1, // row stride in bytes
|
|
size_t nb2, // slice stride in bytes
|
|
size_t nb3,
|
|
size_t offset);
|
|
|
|
GGML_API struct ggml_tensor * ggml_permute(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int axis0,
|
|
int axis1,
|
|
int axis2,
|
|
int axis3);
|
|
|
|
// alias for ggml_permute(ctx, a, 1, 0, 2, 3)
|
|
GGML_API struct ggml_tensor * ggml_transpose(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_get_rows(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_get_rows_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
struct ggml_tensor * c);
|
|
|
|
GGML_API struct ggml_tensor * ggml_diag(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// set elements above the diagonal to -INF
|
|
GGML_API struct ggml_tensor * ggml_diag_mask_inf(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_past);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_diag_mask_inf_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_past);
|
|
|
|
// set elements above the diagonal to 0
|
|
GGML_API struct ggml_tensor * ggml_diag_mask_zero(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_past);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_diag_mask_zero_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_past);
|
|
|
|
GGML_API struct ggml_tensor * ggml_soft_max(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_soft_max_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a);
|
|
|
|
GGML_API struct ggml_tensor * ggml_soft_max_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_soft_max_back_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// rotary position embedding
|
|
// if mode & 1 == 1, skip n_past elements (DEPRECATED)
|
|
// if mode & 2 == 1, GPT-NeoX style
|
|
// if mode & 4 == 1, ChatGLM style
|
|
//
|
|
// b is an int32 vector with size a->ne[2], it contains the positions
|
|
GGML_API struct ggml_tensor * ggml_rope(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int n_dims,
|
|
int mode,
|
|
int n_ctx);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_rope_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int n_dims,
|
|
int mode,
|
|
int n_ctx);
|
|
|
|
// custom RoPE
|
|
GGML_API struct ggml_tensor * ggml_rope_custom(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int n_dims,
|
|
int mode,
|
|
int n_ctx,
|
|
float freq_base,
|
|
float freq_scale);
|
|
|
|
// in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_rope_custom_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int n_dims,
|
|
int mode,
|
|
int n_ctx,
|
|
float freq_base,
|
|
float freq_scale);
|
|
|
|
// xPos RoPE, in-place, returns view(a)
|
|
GGML_API struct ggml_tensor * ggml_rope_xpos_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int n_dims,
|
|
float base,
|
|
bool down);
|
|
|
|
// rotary position embedding backward, i.e compute dx from dy
|
|
// a - dy
|
|
GGML_API struct ggml_tensor * ggml_rope_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int n_dims,
|
|
int mode,
|
|
int n_ctx,
|
|
float freq_base,
|
|
float freq_scale,
|
|
float xpos_base,
|
|
bool xpos_down);
|
|
|
|
// alibi position embedding
|
|
// in-place, returns view(a)
|
|
struct ggml_tensor * ggml_alibi(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int n_past,
|
|
int n_head,
|
|
float bias_max);
|
|
|
|
// clamp
|
|
// in-place, returns view(a)
|
|
struct ggml_tensor * ggml_clamp(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
float min,
|
|
float max);
|
|
|
|
GGML_API struct ggml_tensor * ggml_conv_1d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int s0, // stride
|
|
int p0, // padding
|
|
int d0); // dilation
|
|
|
|
// conv_1d with padding = half
|
|
// alias for ggml_conv_1d(a, b, s, a->ne[0]/2, d)
|
|
GGML_API struct ggml_tensor* ggml_conv_1d_ph(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int s,
|
|
int d);
|
|
|
|
GGML_API struct ggml_tensor * ggml_conv_2d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int s0,
|
|
int s1,
|
|
int p0,
|
|
int p1,
|
|
int d0,
|
|
int d1);
|
|
|
|
|
|
// kernel size is a->ne[0] x a->ne[1]
|
|
// stride is equal to kernel size
|
|
// padding is zero
|
|
// example:
|
|
// a: 16 16 3 768
|
|
// b: 1024 1024 3 1
|
|
// res: 64 64 768 1
|
|
// used in sam
|
|
GGML_API struct ggml_tensor * ggml_conv_2d_sk_p0(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
// kernel size is a->ne[0] x a->ne[1]
|
|
// stride is 1
|
|
// padding is half
|
|
// example:
|
|
// a: 3 3 256 256
|
|
// b: 64 64 256 1
|
|
// res: 64 64 256 1
|
|
// used in sam
|
|
GGML_API struct ggml_tensor * ggml_conv_2d_s1_ph(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_conv_transpose_2d_p0(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
int stride);
|
|
|
|
enum ggml_op_pool {
|
|
GGML_OP_POOL_MAX,
|
|
GGML_OP_POOL_AVG,
|
|
GGML_OP_POOL_COUNT,
|
|
};
|
|
|
|
GGML_API struct ggml_tensor * ggml_pool_1d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
enum ggml_op_pool op,
|
|
int k0, // kernel size
|
|
int s0, // stride
|
|
int p0); // padding
|
|
|
|
GGML_API struct ggml_tensor * ggml_pool_2d(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
enum ggml_op_pool op,
|
|
int k0,
|
|
int k1,
|
|
int s0,
|
|
int s1,
|
|
int p0,
|
|
int p1);
|
|
|
|
// nearest interpolate
|
|
// used in stable-diffusion
|
|
GGML_API struct ggml_tensor * ggml_upscale(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int scale_factor);
|
|
|
|
GGML_API struct ggml_tensor * ggml_flash_attn(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * q,
|
|
struct ggml_tensor * k,
|
|
struct ggml_tensor * v,
|
|
bool masked);
|
|
|
|
GGML_API struct ggml_tensor * ggml_flash_attn_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * q,
|
|
struct ggml_tensor * k,
|
|
struct ggml_tensor * v,
|
|
struct ggml_tensor * d,
|
|
bool masked);
|
|
|
|
GGML_API struct ggml_tensor * ggml_flash_ff(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b0,
|
|
struct ggml_tensor * b1,
|
|
struct ggml_tensor * c0,
|
|
struct ggml_tensor * c1);
|
|
|
|
// partition into non-overlapping windows with padding if needed
|
|
// example:
|
|
// a: 768 64 64 1
|
|
// w: 14
|
|
// res: 768 14 14 25
|
|
// used in sam
|
|
GGML_API struct ggml_tensor * ggml_win_part(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int w);
|
|
|
|
// reverse of ggml_win_part
|
|
// used in sam
|
|
GGML_API struct ggml_tensor * ggml_win_unpart(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int w0,
|
|
int h0,
|
|
int w);
|
|
|
|
GGML_API struct ggml_tensor * ggml_unary(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
enum ggml_unary_op op);
|
|
|
|
GGML_API struct ggml_tensor * ggml_unary_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
enum ggml_unary_op op);
|
|
|
|
// used in sam
|
|
GGML_API struct ggml_tensor * ggml_get_rel_pos(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
int qh,
|
|
int kh);
|
|
|
|
// used in sam
|
|
|
|
GGML_API struct ggml_tensor * ggml_add_rel_pos(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * pw,
|
|
struct ggml_tensor * ph);
|
|
|
|
GGML_API struct ggml_tensor * ggml_add_rel_pos_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * pw,
|
|
struct ggml_tensor * ph);
|
|
|
|
// custom operators
|
|
|
|
typedef void (*ggml_unary_op_f32_t) (const int, float *, const float *);
|
|
typedef void (*ggml_binary_op_f32_t)(const int, float *, const float *, const float *);
|
|
|
|
typedef void (*ggml_custom1_op_f32_t)(struct ggml_tensor *, const struct ggml_tensor *);
|
|
typedef void (*ggml_custom2_op_f32_t)(struct ggml_tensor *, const struct ggml_tensor *, const struct ggml_tensor *);
|
|
typedef void (*ggml_custom3_op_f32_t)(struct ggml_tensor *, const struct ggml_tensor *, const struct ggml_tensor *, const struct ggml_tensor *);
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_unary_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
ggml_unary_op_f32_t fun),
|
|
"use ggml_map_custom1 instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_unary_inplace_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
ggml_unary_op_f32_t fun),
|
|
"use ggml_map_custom1_inplace instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_binary_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
ggml_binary_op_f32_t fun),
|
|
"use ggml_map_custom2 instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_binary_inplace_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
ggml_binary_op_f32_t fun),
|
|
"use ggml_map_custom2_inplace instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_custom1_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
ggml_custom1_op_f32_t fun),
|
|
"use ggml_map_custom1 instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_custom1_inplace_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
ggml_custom1_op_f32_t fun),
|
|
"use ggml_map_custom1_inplace instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_custom2_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
ggml_custom2_op_f32_t fun),
|
|
"use ggml_map_custom2 instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_custom2_inplace_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
ggml_custom2_op_f32_t fun),
|
|
"use ggml_map_custom2_inplace instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_custom3_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
struct ggml_tensor * c,
|
|
ggml_custom3_op_f32_t fun),
|
|
"use ggml_map_custom3 instead");
|
|
|
|
GGML_DEPRECATED(GGML_API struct ggml_tensor * ggml_map_custom3_inplace_f32(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
struct ggml_tensor * c,
|
|
ggml_custom3_op_f32_t fun),
|
|
"use ggml_map_custom3_inplace instead");
|
|
|
|
// custom operators v2
|
|
|
|
typedef void (*ggml_custom1_op_t)(struct ggml_tensor * dst , const struct ggml_tensor * a, int ith, int nth, void * userdata);
|
|
typedef void (*ggml_custom2_op_t)(struct ggml_tensor * dst , const struct ggml_tensor * a, const struct ggml_tensor * b, int ith, int nth, void * userdata);
|
|
typedef void (*ggml_custom3_op_t)(struct ggml_tensor * dst , const struct ggml_tensor * a, const struct ggml_tensor * b, const struct ggml_tensor * c, int ith, int nth, void * userdata);
|
|
|
|
#define GGML_N_TASKS_MAX -1
|
|
|
|
GGML_API struct ggml_tensor * ggml_map_custom1(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
ggml_custom1_op_t fun,
|
|
int n_tasks,
|
|
void * userdata);
|
|
|
|
GGML_API struct ggml_tensor * ggml_map_custom1_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
ggml_custom1_op_t fun,
|
|
int n_tasks,
|
|
void * userdata);
|
|
|
|
GGML_API struct ggml_tensor * ggml_map_custom2(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
ggml_custom2_op_t fun,
|
|
int n_tasks,
|
|
void * userdata);
|
|
|
|
GGML_API struct ggml_tensor * ggml_map_custom2_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
ggml_custom2_op_t fun,
|
|
int n_tasks,
|
|
void * userdata);
|
|
|
|
GGML_API struct ggml_tensor * ggml_map_custom3(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
struct ggml_tensor * c,
|
|
ggml_custom3_op_t fun,
|
|
int n_tasks,
|
|
void * userdata);
|
|
|
|
GGML_API struct ggml_tensor * ggml_map_custom3_inplace(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
struct ggml_tensor * c,
|
|
ggml_custom3_op_t fun,
|
|
int n_tasks,
|
|
void * userdata);
|
|
|
|
// loss function
|
|
|
|
GGML_API struct ggml_tensor * ggml_cross_entropy_loss(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b);
|
|
|
|
GGML_API struct ggml_tensor * ggml_cross_entropy_loss_back(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * a,
|
|
struct ggml_tensor * b,
|
|
struct ggml_tensor * c);
|
|
|
|
//
|
|
// automatic differentiation
|
|
//
|
|
|
|
GGML_API void ggml_set_param(
|
|
struct ggml_context * ctx,
|
|
struct ggml_tensor * tensor);
|
|
|
|
|
|
GGML_API void ggml_build_forward_expand (struct ggml_cgraph * cgraph, struct ggml_tensor * tensor);
|
|
GGML_API void ggml_build_backward_expand(struct ggml_context * ctx, struct ggml_cgraph * gf, struct ggml_cgraph * gb, bool keep);
|
|
|
|
GGML_API struct ggml_cgraph ggml_build_forward (struct ggml_tensor * tensor);
|
|
GGML_API struct ggml_cgraph ggml_build_backward(struct ggml_context * ctx, struct ggml_cgraph * gf, bool keep);
|
|
|
|
// graph allocation in a context
|
|
GGML_API struct ggml_cgraph * ggml_new_graph (struct ggml_context * ctx);
|
|
GGML_API struct ggml_cgraph * ggml_build_forward_ctx(struct ggml_context * ctx, struct ggml_tensor * tensor);
|
|
GGML_API size_t ggml_graph_overhead(void);
|
|
|
|
// ggml_graph_plan() has to be called before ggml_graph_compute()
|
|
// when plan.work_size > 0, caller must allocate memory for plan.work_data
|
|
GGML_API struct ggml_cplan ggml_graph_plan (struct ggml_cgraph * cgraph, int n_threads /*= GGML_DEFAULT_N_THREADS*/);
|
|
GGML_API int ggml_graph_compute(struct ggml_cgraph * cgraph, struct ggml_cplan * cplan);
|
|
GGML_API void ggml_graph_reset (struct ggml_cgraph * cgraph);
|
|
|
|
// same as ggml_graph_compute() but the work data is allocated as a part of the context
|
|
// note: the drawback of this API is that you must have ensured that the context has enough memory for the work data
|
|
GGML_API void ggml_graph_compute_with_ctx(struct ggml_context * ctx, struct ggml_cgraph * cgraph, int n_threads);
|
|
|
|
GGML_API struct ggml_tensor * ggml_graph_get_tensor(struct ggml_cgraph * cgraph, const char * name);
|
|
|
|
GGML_API void ggml_graph_export(const struct ggml_cgraph * cgraph, const char * fname);
|
|
GGML_API struct ggml_cgraph ggml_graph_import(const char * fname, struct ggml_context ** ctx_data, struct ggml_context ** ctx_eval);
|
|
|
|
// print info and performance information for the graph
|
|
GGML_API void ggml_graph_print(const struct ggml_cgraph * cgraph);
|
|
|
|
// dump the graph into a file using the dot format
|
|
GGML_API void ggml_graph_dump_dot(const struct ggml_cgraph * gb, const struct ggml_cgraph * gf, const char * filename);
|
|
|
|
// build gradient checkpointing backward graph gb for gf using provided checkpoints
|
|
// gb_tmp will contain original backward graph with rewritten backward process nodes,
|
|
// but without the second forward pass nodes.
|
|
GGML_API void ggml_build_backward_gradient_checkpointing(
|
|
struct ggml_context * ctx,
|
|
struct ggml_cgraph * gf,
|
|
struct ggml_cgraph * gb,
|
|
struct ggml_cgraph * gb_tmp,
|
|
struct ggml_tensor * * checkpoints,
|
|
int n_checkpoints);
|
|
//
|
|
// optimization
|
|
//
|
|
|
|
// optimization methods
|
|
enum ggml_opt_type {
|
|
GGML_OPT_ADAM,
|
|
GGML_OPT_LBFGS,
|
|
};
|
|
|
|
// linesearch methods
|
|
enum ggml_linesearch {
|
|
GGML_LINESEARCH_DEFAULT = 1,
|
|
|
|
GGML_LINESEARCH_BACKTRACKING_ARMIJO = 0,
|
|
GGML_LINESEARCH_BACKTRACKING_WOLFE = 1,
|
|
GGML_LINESEARCH_BACKTRACKING_STRONG_WOLFE = 2,
|
|
};
|
|
|
|
// optimization return values
|
|
enum ggml_opt_result {
|
|
GGML_OPT_OK = 0,
|
|
GGML_OPT_DID_NOT_CONVERGE,
|
|
GGML_OPT_NO_CONTEXT,
|
|
GGML_OPT_INVALID_WOLFE,
|
|
GGML_OPT_FAIL,
|
|
|
|
GGML_LINESEARCH_FAIL = -128,
|
|
GGML_LINESEARCH_MINIMUM_STEP,
|
|
GGML_LINESEARCH_MAXIMUM_STEP,
|
|
GGML_LINESEARCH_MAXIMUM_ITERATIONS,
|
|
GGML_LINESEARCH_INVALID_PARAMETERS,
|
|
};
|
|
|
|
typedef void (*ggml_opt_callback)(void * data, int accum_step, float * sched, bool * cancel);
|
|
typedef void (*ggml_log_callback)(enum ggml_log_level level, const char * text, void * user_data);
|
|
|
|
// optimization parameters
|
|
//
|
|
// see ggml.c (ggml_opt_default_params) for default values
|
|
//
|
|
struct ggml_opt_params {
|
|
enum ggml_opt_type type;
|
|
|
|
int n_threads;
|
|
|
|
// delta-based convergence test
|
|
//
|
|
// if past == 0 - disabled
|
|
// if past > 0:
|
|
// stop if |f(x) - f(x_past)| < delta * max(1, |f(x)|)
|
|
//
|
|
int past;
|
|
float delta;
|
|
|
|
// maximum number of iterations without improvement
|
|
//
|
|
// if 0 - disabled
|
|
// if > 0:
|
|
// assume convergence if no cost improvement in this number of iterations
|
|
//
|
|
int max_no_improvement;
|
|
|
|
bool print_forward_graph;
|
|
bool print_backward_graph;
|
|
|
|
int n_gradient_accumulation;
|
|
|
|
// ADAM parameters
|
|
struct {
|
|
int n_iter;
|
|
|
|
float sched; // schedule multiplier (fixed, decay or warmup)
|
|
float decay; // weight decay for AdamW, use 0.0f to disable
|
|
int decay_min_ndim; // minimum number of tensor dimension to apply weight decay
|
|
float alpha; // learning rate
|
|
float beta1;
|
|
float beta2;
|
|
float eps; // epsilon for numerical stability
|
|
float eps_f; // epsilon for convergence test
|
|
float eps_g; // epsilon for convergence test
|
|
float gclip; // gradient clipping
|
|
} adam;
|
|
|
|
// LBFGS parameters
|
|
struct {
|
|
int m; // number of corrections to approximate the inv. Hessian
|
|
int n_iter;
|
|
int max_linesearch;
|
|
|
|
float eps; // convergence tolerance
|
|
float ftol; // line search tolerance
|
|
float wolfe;
|
|
float min_step;
|
|
float max_step;
|
|
|
|
enum ggml_linesearch linesearch;
|
|
} lbfgs;
|
|
};
|
|
|
|
struct ggml_opt_context {
|
|
struct ggml_context * ctx;
|
|
struct ggml_opt_params params;
|
|
|
|
int iter;
|
|
int64_t nx; // number of parameter elements
|
|
|
|
bool just_initialized;
|
|
|
|
float loss_before;
|
|
float loss_after;
|
|
|
|
struct {
|
|
struct ggml_tensor * g; // current gradient
|
|
struct ggml_tensor * m; // first moment
|
|
struct ggml_tensor * v; // second moment
|
|
struct ggml_tensor * pf; // past function values
|
|
float fx_best;
|
|
float fx_prev;
|
|
int n_no_improvement;
|
|
} adam;
|
|
|
|
struct {
|
|
struct ggml_tensor * x; // current parameters
|
|
struct ggml_tensor * xp; // previous parameters
|
|
struct ggml_tensor * g; // current gradient
|
|
struct ggml_tensor * gp; // previous gradient
|
|
struct ggml_tensor * d; // search direction
|
|
struct ggml_tensor * pf; // past function values
|
|
struct ggml_tensor * lmal; // the L-BFGS memory alpha
|
|
struct ggml_tensor * lmys; // the L-BFGS memory ys
|
|
struct ggml_tensor * lms; // the L-BFGS memory s
|
|
struct ggml_tensor * lmy; // the L-BFGS memory y
|
|
float fx_best;
|
|
float step;
|
|
int j;
|
|
int k;
|
|
int end;
|
|
int n_no_improvement;
|
|
} lbfgs;
|
|
};
|
|
|
|
GGML_API struct ggml_opt_params ggml_opt_default_params(enum ggml_opt_type type);
|
|
|
|
// optimize the function defined by the tensor f
|
|
GGML_API enum ggml_opt_result ggml_opt(
|
|
struct ggml_context * ctx,
|
|
struct ggml_opt_params params,
|
|
struct ggml_tensor * f);
|
|
|
|
// initialize optimizer context
|
|
GGML_API void ggml_opt_init(
|
|
struct ggml_context * ctx,
|
|
struct ggml_opt_context * opt,
|
|
struct ggml_opt_params params,
|
|
int64_t nx);
|
|
|
|
// continue optimizing the function defined by the tensor f
|
|
GGML_API enum ggml_opt_result ggml_opt_resume(
|
|
struct ggml_context * ctx,
|
|
struct ggml_opt_context * opt,
|
|
struct ggml_tensor * f);
|
|
|
|
// continue optimizing the function defined by the tensor f
|
|
GGML_API enum ggml_opt_result ggml_opt_resume_g(
|
|
struct ggml_context * ctx,
|
|
struct ggml_opt_context * opt,
|
|
struct ggml_tensor * f,
|
|
struct ggml_cgraph * gf,
|
|
struct ggml_cgraph * gb,
|
|
ggml_opt_callback callback,
|
|
void * callback_data);
|
|
|
|
//
|
|
// quantization
|
|
//
|
|
|
|
GGML_API size_t ggml_quantize_q4_0(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
GGML_API size_t ggml_quantize_q4_1(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
GGML_API size_t ggml_quantize_q5_0(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
GGML_API size_t ggml_quantize_q5_1(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
GGML_API size_t ggml_quantize_q8_0(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
|
|
GGML_API size_t ggml_quantize_chunk(enum ggml_type type, const float * src, void * dst, int start, int n, int64_t * hist);
|
|
|
|
//
|
|
// gguf
|
|
//
|
|
|
|
enum gguf_type {
|
|
GGUF_TYPE_UINT8 = 0,
|
|
GGUF_TYPE_INT8 = 1,
|
|
GGUF_TYPE_UINT16 = 2,
|
|
GGUF_TYPE_INT16 = 3,
|
|
GGUF_TYPE_UINT32 = 4,
|
|
GGUF_TYPE_INT32 = 5,
|
|
GGUF_TYPE_FLOAT32 = 6,
|
|
GGUF_TYPE_BOOL = 7,
|
|
GGUF_TYPE_STRING = 8,
|
|
GGUF_TYPE_ARRAY = 9,
|
|
GGUF_TYPE_UINT64 = 10,
|
|
GGUF_TYPE_INT64 = 11,
|
|
GGUF_TYPE_FLOAT64 = 12,
|
|
GGUF_TYPE_COUNT, // marks the end of the enum
|
|
};
|
|
|
|
struct gguf_context;
|
|
|
|
struct gguf_init_params {
|
|
bool no_alloc;
|
|
|
|
// if not NULL, create a ggml_context and allocate the tensor data in it
|
|
struct ggml_context ** ctx;
|
|
};
|
|
|
|
GGML_API struct gguf_context * gguf_init_empty(void);
|
|
GGML_API struct gguf_context * gguf_init_from_file(const char * fname, struct gguf_init_params params);
|
|
//GGML_API struct gguf_context * gguf_init_from_buffer(..);
|
|
|
|
GGML_API void gguf_free(struct gguf_context * ctx);
|
|
|
|
GGML_API const char * gguf_type_name(enum gguf_type type);
|
|
|
|
GGML_API int gguf_get_version (const struct gguf_context * ctx);
|
|
GGML_API size_t gguf_get_alignment (const struct gguf_context * ctx);
|
|
GGML_API size_t gguf_get_data_offset(const struct gguf_context * ctx);
|
|
GGML_API void * gguf_get_data (const struct gguf_context * ctx);
|
|
|
|
GGML_API int gguf_get_n_kv(const struct gguf_context * ctx);
|
|
GGML_API int gguf_find_key(const struct gguf_context * ctx, const char * key);
|
|
GGML_API const char * gguf_get_key (const struct gguf_context * ctx, int key_id);
|
|
|
|
GGML_API enum gguf_type gguf_get_kv_type (const struct gguf_context * ctx, int key_id);
|
|
GGML_API enum gguf_type gguf_get_arr_type(const struct gguf_context * ctx, int key_id);
|
|
|
|
// will abort if the wrong type is used for the key
|
|
GGML_API uint8_t gguf_get_val_u8 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API int8_t gguf_get_val_i8 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API uint16_t gguf_get_val_u16 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API int16_t gguf_get_val_i16 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API uint32_t gguf_get_val_u32 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API int32_t gguf_get_val_i32 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API float gguf_get_val_f32 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API uint64_t gguf_get_val_u64 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API int64_t gguf_get_val_i64 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API double gguf_get_val_f64 (const struct gguf_context * ctx, int key_id);
|
|
GGML_API bool gguf_get_val_bool(const struct gguf_context * ctx, int key_id);
|
|
GGML_API const char * gguf_get_val_str (const struct gguf_context * ctx, int key_id);
|
|
GGML_API int gguf_get_arr_n (const struct gguf_context * ctx, int key_id);
|
|
GGML_API const void * gguf_get_arr_data(const struct gguf_context * ctx, int key_id);
|
|
GGML_API const char * gguf_get_arr_str (const struct gguf_context * ctx, int key_id, int i);
|
|
|
|
GGML_API int gguf_get_n_tensors (const struct gguf_context * ctx);
|
|
GGML_API int gguf_find_tensor (const struct gguf_context * ctx, const char * name);
|
|
GGML_API size_t gguf_get_tensor_offset(const struct gguf_context * ctx, int i);
|
|
GGML_API char * gguf_get_tensor_name (const struct gguf_context * ctx, int i);
|
|
|
|
// overrides existing values or adds a new one
|
|
GGML_API void gguf_set_val_u8 (struct gguf_context * ctx, const char * key, uint8_t val);
|
|
GGML_API void gguf_set_val_i8 (struct gguf_context * ctx, const char * key, int8_t val);
|
|
GGML_API void gguf_set_val_u16 (struct gguf_context * ctx, const char * key, uint16_t val);
|
|
GGML_API void gguf_set_val_i16 (struct gguf_context * ctx, const char * key, int16_t val);
|
|
GGML_API void gguf_set_val_u32 (struct gguf_context * ctx, const char * key, uint32_t val);
|
|
GGML_API void gguf_set_val_i32 (struct gguf_context * ctx, const char * key, int32_t val);
|
|
GGML_API void gguf_set_val_f32 (struct gguf_context * ctx, const char * key, float val);
|
|
GGML_API void gguf_set_val_u64 (struct gguf_context * ctx, const char * key, uint64_t val);
|
|
GGML_API void gguf_set_val_i64 (struct gguf_context * ctx, const char * key, int64_t val);
|
|
GGML_API void gguf_set_val_f64 (struct gguf_context * ctx, const char * key, double val);
|
|
GGML_API void gguf_set_val_bool(struct gguf_context * ctx, const char * key, bool val);
|
|
GGML_API void gguf_set_val_str (struct gguf_context * ctx, const char * key, const char * val);
|
|
GGML_API void gguf_set_arr_data(struct gguf_context * ctx, const char * key, enum gguf_type type, const void * data, int n);
|
|
GGML_API void gguf_set_arr_str (struct gguf_context * ctx, const char * key, const char ** data, int n);
|
|
|
|
// set or add KV pairs from another context
|
|
GGML_API void gguf_set_kv(struct gguf_context * ctx, struct gguf_context * src);
|
|
|
|
// manage tensor info
|
|
GGML_API void gguf_add_tensor(struct gguf_context * ctx, const struct ggml_tensor * tensor);
|
|
GGML_API void gguf_set_tensor_type(struct gguf_context * ctx, const char * name, enum ggml_type type);
|
|
GGML_API void gguf_set_tensor_data(struct gguf_context * ctx, const char * name, const void * data, size_t size);
|
|
|
|
// writing gguf files can be done in 2 ways:
|
|
//
|
|
// - write the entire gguf_context to a binary file in a single pass:
|
|
//
|
|
// gguf_write_to_file(ctx, fname);
|
|
//
|
|
// - first prepare a file with a placeholder for the meta data, write the tensor data, then write the meta data:
|
|
//
|
|
// FILE * f = fopen(fname, "wb");
|
|
// fseek(f, gguf_get_meta_size(ctx), SEEK_SET);
|
|
// fwrite(f, ...);
|
|
// void * data = gguf_meta_get_meta_data(ctx);
|
|
// fseek(f, 0, SEEK_SET);
|
|
// fwrite(f, data, gguf_get_meta_size(ctx));
|
|
// free(data);
|
|
// fclose(f);
|
|
//
|
|
|
|
// write the entire context to a binary file
|
|
GGML_API void gguf_write_to_file(const struct gguf_context * ctx, const char * fname, bool only_meta);
|
|
|
|
// get the size in bytes of the meta data (header, kv pairs, tensor info) including padding
|
|
GGML_API size_t gguf_get_meta_size(const struct gguf_context * ctx);
|
|
GGML_API void gguf_get_meta_data(const struct gguf_context * ctx, void * data);
|
|
|
|
//
|
|
// system info
|
|
//
|
|
|
|
GGML_API int ggml_cpu_has_avx (void);
|
|
GGML_API int ggml_cpu_has_avx2 (void);
|
|
GGML_API int ggml_cpu_has_avx512 (void);
|
|
GGML_API int ggml_cpu_has_avx512_vbmi(void);
|
|
GGML_API int ggml_cpu_has_avx512_vnni(void);
|
|
GGML_API int ggml_cpu_has_fma (void);
|
|
GGML_API int ggml_cpu_has_neon (void);
|
|
GGML_API int ggml_cpu_has_arm_fma (void);
|
|
GGML_API int ggml_cpu_has_metal (void);
|
|
GGML_API int ggml_cpu_has_f16c (void);
|
|
GGML_API int ggml_cpu_has_fp16_va (void);
|
|
GGML_API int ggml_cpu_has_wasm_simd (void);
|
|
GGML_API int ggml_cpu_has_blas (void);
|
|
GGML_API int ggml_cpu_has_cublas (void);
|
|
GGML_API int ggml_cpu_has_clblast (void);
|
|
GGML_API int ggml_cpu_has_gpublas (void);
|
|
GGML_API int ggml_cpu_has_sse3 (void);
|
|
GGML_API int ggml_cpu_has_ssse3 (void);
|
|
GGML_API int ggml_cpu_has_vsx (void);
|
|
|
|
//
|
|
// Internal types and functions exposed for tests and benchmarks
|
|
//
|
|
|
|
#ifdef __cplusplus
|
|
// restrict not standard in C++
|
|
#define GGML_RESTRICT
|
|
#else
|
|
#define GGML_RESTRICT restrict
|
|
#endif
|
|
typedef void (*ggml_to_float_t) (const void * GGML_RESTRICT x, float * GGML_RESTRICT y, int k);
|
|
typedef void (*ggml_from_float_t)(const float * GGML_RESTRICT x, void * GGML_RESTRICT y, int k);
|
|
typedef void (*ggml_vec_dot_t) (const int n, float * GGML_RESTRICT s, const void * GGML_RESTRICT x, const void * GGML_RESTRICT y);
|
|
|
|
typedef struct {
|
|
const char * type_name;
|
|
int blck_size;
|
|
size_t type_size;
|
|
bool is_quantized;
|
|
ggml_to_float_t to_float;
|
|
ggml_from_float_t from_float;
|
|
ggml_from_float_t from_float_reference;
|
|
ggml_vec_dot_t vec_dot;
|
|
enum ggml_type vec_dot_type;
|
|
} ggml_type_traits_t;
|
|
|
|
ggml_type_traits_t ggml_internal_get_type_traits(enum ggml_type type);
|
|
|
|
#ifdef __cplusplus
|
|
}
|
|
#endif
|