Georgi Gerganov
2fcdf869cd
batched-bench : add mmq CLI arg
2023-10-11 19:42:33 +03:00
Georgi Gerganov
daeb834da9
batched-bench : pass custom set of PP, TG and PL
2023-10-11 19:36:31 +03:00
Georgi Gerganov
c062ffd18c
batched-bench : init warm-up batch
2023-10-11 19:24:59 +03:00
Georgi Gerganov
76e17f8d93
Merge branch 'master' into batched-bench
2023-10-11 19:18:35 +03:00
Georgi Gerganov
026bb1b1cd
batched-bench : add readme + n_kv_max is now configurable
2023-10-11 19:10:35 +03:00
Zane Shannon
24ba3d829e
examples : add batched.swift + improve CI for swift ( #3562 )
2023-10-11 06:14:05 -05:00
Galunid
9f6ede19f3
Add MPT model to supported models in README.md ( #3574 )
2023-10-10 19:02:49 -04:00
goerch
233fc1c69f
Minor improvements in GPT2 tokenizer ( #3567 )
...
* Fixing minor bugs in bpe_gpt2_preprocess
* Don't add bos token in test
2023-10-10 18:59:52 +02:00
Xingchen Song(宋星辰)
c5b49360d0
readme : add bloom ( #3570 )
2023-10-10 19:28:50 +03:00
Xingchen Song(宋星辰)
02d2875def
llm : add bloom models ( #3553 )
...
* feat: Support bloom models
* fix(bloom): fix model size
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-10 17:48:21 +03:00
Jhen-Jie Hong
0aa6595ae0
swift : improvements and fixes ( #3564 )
...
* swift : use macOS 12 as minimum requirement
* swift : add missing ggml-backend.c source
* swift : add -O3 -DNDEBUG unsafe flags
2023-10-10 14:31:13 +03:00
Jan Ploski
f5f9121de1
llm : add MPT support ( #3417 )
...
* CUDA: added support for ggml_clamp (see also: https://github.com/ggerganov/ggml/issues/545 )
* mpt : added an implementation based (mostly) on falcon integration, modified with deltas from ggml/examples/mpt
* mpt : protect against "clip_qkv": null in mpt-7b
* mpt : quick fix to avoid "Strange model" warning when quantizing MPT models
* mpt : addendum to changeset:84e30e8 - leave parameter clamp_kqv out from metadata rather than use 0.0 to indicate "no clamping" (more compliant with the current GGUF spec?)
* mpt : standardized all tensor names to follow GGUF spec
* mpt : addendum to changeset:1be89c40 - use "req" parameter of GGUF_GET_KEY macro instead of duplicate code
* mpt : fixed comment s/gptneox/mpt/
* mpt : remove tabs, trailing whitespace
* mpt : removed ne01 + n_past == ne00 assertion from alibi (cuda/f32) and rope_shift from build_mpt
* mpt : updated convert-mpt-hf-to-gguf.py to reflect changes made to convert-gptneox-hf-to-gguf.py in pr:3252
* comment out n_past instead of marking it unused
* mpt : removed hardcoded +178 from convert script in favor of utilizing hparams["vocab_size"]
* mpt : remove unused tokenizer_json in convert script
* ggml : remove obsolete n_past assert in ggml_alibi
* llama : print clam_kqv and max_alibi_bias hparams
---------
Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-10 10:50:23 +03:00
vvhg1
11ea5c7d96
infill. : fix tokenization ( #3508 )
...
* infill tokens correction
* serverinfill tokens correction
* removing any leading whitespace from infill suffix and removing leeading space token from suffix when params.escape
* removing any leading whitespace from infill suffix and removing leeading space token from suffix when params.escape
* only rm when params.escape, rm space if possible which is added back or rm added space token
* only rm when params.escape, rm space if possible which is added back or rm added space token
* Revert "only rm when params.escape, rm space if possible which is added back or rm added space token"
This reverts commit 63ba0b621f
2023-10-10 10:31:21 +03:00
slaren
95bd60a0a6
ggml-alloc : fix assert in debug builds ( #3555 )
2023-10-09 15:44:58 +03:00
Georgi Gerganov
fcca0a7004
refact : fix convert script + zero out KV cache to avoid nans ( #3523 )
...
* refact : fix convert script + zero out KV cache to avoid nans
* ggml : silu(-inf) should never happen
* metal : assert various kernel requirements
2023-10-09 14:32:17 +03:00
Georgi Gerganov
dcc09d2596
metal : do not use mul_mm kernels when ne00 < 64 ( #3542 )
2023-10-09 14:28:27 +03:00
Georgi Gerganov
db3abcc114
sync : ggml (ggml-backend) ( #3548 )
...
* sync : ggml (ggml-backend)
ggml-ci
* zig : add ggml-backend to the build
2023-10-08 20:19:14 +03:00
Matheus C. França
eee42c670e
ci : add Zig CI/CD and fix build ( #2996 )
...
* zig CI/CD and fix build
Signed-off-by: Matheus Catarino França <matheus-catarino@hotmail.com>
* fix build_compiler
* ci : remove trailing whitespace
---------
Signed-off-by: Matheus Catarino França <matheus-catarino@hotmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-08 16:59:20 +03:00
Georgi Gerganov
7438728d51
batched : minor fix table
2023-10-08 16:35:54 +03:00
Georgi Gerganov
bf06d654de
batched : add bench tool
2023-10-08 15:57:16 +03:00
Ryder Wishart
8e6716a102
api_like_OAI.py : compat with Microsoft Guidance ( #2746 )
...
Check for None in addition to empty string check in all request params
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-08 13:55:58 +03:00
arcrank
9c38d181d4
api_like_OAI.py : simplify function ( #2796 )
...
Simplify function
2023-10-08 13:52:57 +03:00
Johannes Rudolph
a1202a31ed
k-quants : fix comments about block sizing ( #3499 )
2023-10-08 13:21:19 +03:00
Georgi Gerganov
94e502dfb7
ci : enable on obj-c changes + fix metal build ( #3540 )
2023-10-08 11:24:50 +03:00
Luo Tian
7d8b24932f
zig : fix build by introducing train.cpp ( #3539 )
2023-10-08 11:24:01 +03:00
Georgi Gerganov
b0ec5218c3
metal : support MTLGPUFamily < Apple7, formatting, style ( #3524 )
...
* metal : improve decoding speed for batches of 2-16
* metal : rename kernels mul_mat_ to mul_mv_
* metal : indentations
* minor
* metal : print more GPU info + disable mul_mm for MTLGPUFamiliy < Apple7
2023-10-08 10:01:53 +03:00
Kerfuffle
63d3b06a43
llama : fix missing break in Persimmon arch case statements ( #3535 )
2023-10-08 08:22:17 +03:00
Kerfuffle
a16e89cec8
Fix trying to strip newline from empty prompt and cfg prompt file content ( #3534 )
2023-10-07 15:31:41 -06:00
M. Yusuf Sarıgöz
4d03833211
gguf.py : fix CI for publishing GGUF package ( #3532 )
...
* Fix CI for publishing GGUF package
* Bump version
* fix
* bump version
* bump version
* bump version
2023-10-07 22:14:10 +03:00
Tom C
c47066d833
py : change version of numpy requirement to 1.24.4 ( #3515 )
...
Co-authored-by: Lyjia <me@lyjia.us>
2023-10-07 12:56:15 +03:00
cebtenzzre
f1782c68de
quantize : fail fast on write errors ( #3521 )
2023-10-07 11:41:52 +03:00
Jhen-Jie Hong
c26765a0a1
metal : support default.metallib load & reuse code for swift package ( #3522 )
...
* metal : support load default.metallib & reuse code for swift package
* metal : use SWIFT_PACKAGE def instead of define GGML_SWIFT
2023-10-07 11:40:27 +03:00
Phillip Kravtsov
0e797c2fc5
llm : support Adept Persimmon 8B ( #3410 )
...
* Produces garbage output
* wip: correct tensors up to RoPE
* correct tensors thru RoPE
* Correct outputs through masked & softmax'd KQ
* fp32 works
* Rename adept->persimmon
* Produces correct outputs
* clean up convert scripts
* remove printing logic from ggml.c
* remove prints from llama.cpp & fix merge
* trivial cleanups
* Add offload funcs
* update conversion script to directly take adept artifacts rather than .saftensors file
* Fix norm eps bug
* Support sqr and concat on metal, persimmon-8b-q4 runs correctly
* Small changes from review
* Formatting changes
* Minor changes to conversion script
* Remove old script
* Fix editorconfig formatting
* Fix build
* add overlooked offload code ggml-ci
2023-10-07 10:12:43 +03:00
goerch
3a716b4dae
Fix for #3454 ( #3455 )
...
Fix: `sentencepiece` tokenizers with added tokens failed with an incorrect assertion
2023-10-07 06:57:01 +02:00
BarfingLemurs
1faaae8c2b
readme : update models, cuda + ppl instructions ( #3510 )
2023-10-06 22:13:36 +03:00
Mihai
cb13d73a72
server : docs fix default values and add n_probs ( #3506 )
2023-10-06 21:39:33 +03:00
Kerfuffle
9ca79d5cbb
kv cache slot search improvements ( #3493 )
...
* kv cache slot search improvements
* Use n_ctx in kv find slot for consistency
* Ensure kv cache head points to a valid slot in llama_decode internal
* Add some comments to prevent dumb people (like me) from getting confused.
2023-10-06 10:10:13 -06:00
Georgi Gerganov
0c731ca403
prompts : fix editorconfig checks after #3416
2023-10-06 16:36:32 +03:00
pudepiedj
a8777ad84e
parallel : add option to load external prompt file ( #3416 )
...
* Enable external file and add datestamp
* Add name of external file at end
* Upload ToK2024
* Delete ToK2024.txt
* Experiments with jeopardy
* Move ParallelQuestions to /proimpts and rename
* Interim commit
* Interim commit
* Final revision
* Remove trailing whitespace
* remove cmake_all.sh
* Remove cmake_all.sh
* Changed .gitignore
* Improved reporting and new question files.
* Corrected typo
* More LLM questions
* Update LLM-questions.txt
* Yet more LLM-questions
* Remove jeopardy results file
* Reinstate original jeopardy.sh
* Update examples/parallel/parallel.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-06 16:16:38 +03:00
Jhen-Jie Hong
97af49fa39
server : reuse llama_sample_token common util ( #3494 )
...
* server : reuse llama_sample_token common function
* common : use n_probs for temperature sampling
2023-10-06 15:44:24 +03:00
l3utterfly
16820a5a0d
llama : correct hparams comparison ( #3446 )
...
* fixed floating point comparison issues
* updated implementation for hparam comparison to handle inf and NaN
* fixed code review comments
* minor simplification
* rename is_float_eq -> is_float_close
---------
Co-authored-by: Cebtenzzre <cebtenzzre@gmail.com>
2023-10-06 13:47:59 +03:00
Jhen-Jie Hong
04b2f4386e
ci : fix xcodebuild destinations ( #3491 )
...
* ci : fix xcodebuild destinations
* ci : add .swift to paths
2023-10-06 13:36:43 +03:00
cebtenzzre
48edda30ee
convert : update Falcon script for new HF config ( #3448 )
...
Also adds Falcon-180B support.
Closes #3049
Co-authored-by: jb <jonathan.t.barnard@gmail.com>
2023-10-05 15:00:34 -04:00
Kenvix ⭐
45eba9369f
build : use std::make_tuple() for compatibility with older GCC versions ( #3488 )
2023-10-05 20:16:39 +03:00
staviq
acec9eaaa9
common : process escape sequences in reverse prompts ( #3461 )
2023-10-05 19:17:29 +03:00
shibe2
e2583cbc29
CLBlast: Fix handling of on-device tensor data
...
Fix uploading tensor data to device, including 3D, 4D, and non-contiguous tensors.
Use correct offsets into data that is already in VRAM.
Correct handling of OpenCL events when multiple commands are queued.
2023-10-05 18:25:23 +04:00
Jhen-Jie Hong
e8b8d32e86
server : fix incorrect num_tokens_predicted ( #3480 )
2023-10-05 17:02:55 +03:00
Jhen-Jie Hong
8f3a642ec1
swift : disable ACCELERATE_NEW_LAPACK ( #3481 )
2023-10-05 17:00:07 +03:00
Jhen-Jie Hong
0745384449
ci : add swift build via xcodebuild ( #3482 )
2023-10-05 16:56:21 +03:00
Kerfuffle
019ba1dcd0
convert : fix Baichuan2 models by using vocab size in config.json ( #3299 )
...
Use local GGUF package when possible in Baichuan converter
2023-10-04 17:20:28 +03:00
