llama : add comment about thread-safety [no ci] (#9449)

This commit is contained in:
Georgi Gerganov 2024-09-28 15:13:21 +03:00
parent 6102037bbb
commit 739842703e
No known key found for this signature in database
GPG Key ID: 449E073F9DC10735

View File

@ -911,6 +911,8 @@ extern "C" {
// //
// Tokenization // Tokenization
// //
// The API is thread-safe.
//
/// @details Convert the provided text into tokens. /// @details Convert the provided text into tokens.
/// @param tokens The tokens pointer must be large enough to hold the resulting tokens. /// @param tokens The tokens pointer must be large enough to hold the resulting tokens.