root/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggerganov/llama.cpp.git synced 2025-01-13 12:10:18 +00:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

agent: simplify syntax (default tools to local w/ default port)

Browse Source
This commit is contained in:
ochafik 2024-10-28 23:54:01 +00:00
parent b51c71c734
commit 74d71a673e
4 changed files with 16 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
examples/agent/README.md
View File

@ -7,11 +7,6 @@
```bash
make -j LLAMA_CURL=1 llama-server
# Mistral NeMo
./llama-server --jinja -fa --verbose \
-hfr bartowski/Mistral-Nemo-Instruct-2407-GGUF -hff Mistral-Nemo-Instruct-2407-Q8_0.gguf \
--chat-template "$( python scripts/get_hf_chat_template.py mistralai/Mistral-Nemo-Instruct-2407 )"
# Nous Hermes 2 Pro Llama 3 8B
./llama-server --jinja -fa --verbose \
-hfr NousResearch/Hermes-2-Pro-Llama-3-8B-GGUF -hff Hermes-2-Pro-Llama-3-8B-Q8_0.gguf \
@ -39,6 +34,11 @@
./llama-server --jinja -fa --verbose \
-hfr lmstudio-community/Llama-3.2-1B-Instruct-GGUF -hff Llama-3.2-1B-Instruct-Q4_K_M.gguf \
--chat-template "$( python scripts/get_hf_chat_template.py meta-llama/Llama-3.2-3B-Instruct )"
# Mistral NeMo
./llama-server --jinja -fa --verbose \
-hfr bartowski/Mistral-Nemo-Instruct-2407-GGUF -hff Mistral-Nemo-Instruct-2407-Q8_0.gguf \
--chat-template "$( python scripts/get_hf_chat_template.py mistralai/Mistral-Nemo-Instruct-2407 )"
```
- Run the tools in [examples/agent/tools](./examples/agent/tools) inside a docker container for *some* level of isolation (+ sneaky logging of outgoing http and https traffic: you wanna watch over those agents' shoulders for the time being 🧐). Check http://localhost:8088/docs to see the tools exposed.
@ -54,8 +54,7 @@
- Run the agent with some goal
```bash
uv run examples/agent/run.py --tools http://localhost:8088 \
"What is the sum of 2535 squared and 32222000403?"
uv run examples/agent/run.py "What is the sum of 2535 squared and 32222000403?"
```
<details><summary>See output w/ Hermes-3-Llama-3.1-8B</summary>
@ -70,8 +69,7 @@
</details>
```bash
uv run examples/agent/run.py --tools http://localhost:8088 \
"What is the best BBQ joint in Laguna Beach?"
uv run examples/agent/run.py "What is the best BBQ joint in Laguna Beach?"
```
<details><summary>See output w/ Hermes-3-Llama-3.1-8B</summary>
@ -86,8 +84,7 @@
</details>
```bash
uv run examples/agent/run.py --tools http://localhost:8088 \
"Search for, fetch and summarize the homepage of llama.cpp"
uv run examples/agent/run.py "Search for, fetch and summarize the homepage of llama.cpp"
```
<details><summary>See output w/ Hermes-3-Llama-3.1-8B</summary>
@ -109,9 +106,7 @@
export OPENAI_API_KEY=... # for --provider=openai https://platform.openai.com/api-keys
export TOGETHER_API_KEY=... # for --provider=together https://api.together.ai/settings/api-keys
export GROQ_API_KEY=... # for --provider=groq https://console.groq.com/keys
uv run examples/agent/run.py --tools http://localhost:8088 \
"Search for, fetch and summarize the homepage of llama.cpp" \
--provider=openai
uv run examples/agent/run.py "Search for, fetch and summarize the homepage of llama.cpp" --provider=openai
```
## TODO

3
examples/agent/run.py
View File

@ -71,6 +71,9 @@ async def main(
endpoint: Optional[str] = None,
api_key: Optional[str] = None,
):
if not tools:
tools = ["http://localhost:8088"]
provider_info = _PROVIDERS[provider]
if endpoint is None:
endpoint = provider_info['endpoint']

2
examples/agent/tools/python.py
View File

@ -15,7 +15,7 @@ def _strip_ansi_codes(text):
def python(code: str) -> str:
'''
Execute Python code in a siloed environment using IPython and returns the output.
Execute Python code in a siloed environment using IPython and return the output.
Parameters:
code (str): The Python code to execute.

11
examples/agent/tools/search.py
View File

@ -9,17 +9,12 @@ import requests
def _extract_values(keys, obj):
values = {}
for k in keys:
v = obj.get(k)
if v is not None:
values[k] = v
return values
return dict((k, v) for k in keys if (v := obj.get(k)) is not None)
# Let's keep this tool aligned w/ llama_stack.providers.impls.meta_reference.agents.tools.builtin.BraveSearch
# (see https://github.com/meta-llama/llama-stack/blob/main/llama_stack/providers/impls/meta_reference/agents/tools/builtin.py)
_result_keys_by_type = {
_brave_search_result_keys_by_type = {
'web': ('type', 'title', 'url', 'description', 'date', 'extra_snippets'),
'videos': ('type', 'title', 'url', 'description', 'date'),
'news': ('type', 'title', 'url', 'description'),
@ -54,7 +49,7 @@ async def brave_search(*, query: str) -> List[Dict]:
# print("SEARCH RESPONSE: " + json.dumps(search_response, indent=2))
for m in search_response['mixed']['main']:
result_type = m['type']
keys = _result_keys_by_type.get(result_type)
keys = _brave_search_result_keys_by_type.get(result_type)
if keys is None:
logging.warning(f'[brave_search] Unknown result type: %s', result_type)
continue