docs(coding-agent): update llama.cpp setup for llama.app (#10179)

update the llama.cpp docs to use the llama.app installer and the new
`llama serve` command

commit by @hanouticelina

disclaimer: my julien-cto agent helped me write this

---------

Co-authored-by: Celina Hanouti <hanouticelina@gmail.com>
Co-authored-by: Mario Zechner <contact@badlogicgames.com>
This commit is contained in:
Julien Chaumond 2026-10-27 16:21:37 +01:00 • committed by GitHub
parent 479d6e5e29
commit 59ca40cf58
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -1,31 +1,42 @@
# Local Models with llama.cpp
Pi supports the [llama.cpp](https://github.com/ggml-org/llama.cpp) router server. The router discovers multiple GGUF models and loads or unloads them on demand.
Pi supports the [llama.cpp](https://llama.app) router server. The router discovers multiple GGUF models and loads or unloads them on demand.
Use a current llama.cpp build with router support. Follow the [build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) or install a [prebuilt release](https://github.com/ggml-org/llama.cpp/releases) for your platform.
## Install llama.cpp
Install llama.cpp with the [official installer](https://llama.app/docs/installation):
```bash
curl -LsSf https://llama.app/install.sh | sh
```
Verify the installation:
```bash
llama cli --version
```
See [llama.app](https://llama.app) for other installation methods, including package managers and building from source.
## Start the router
Start `llama-server` without `--model` or `-m`. Passing a model starts single-model mode instead of router mode.
Start `llama serve` without `--model`, `-m`, or `-hf`. Passing a model starts single-model mode instead of router mode. See the [llama.cpp server guide](https://llama.app/docs/serve#serving-multiple-models-router-mode) for router configuration.
```bash
llama-server \
--models-dir ~/models \
llama serve \
--no-models-autoload \
--jinja \
--host 127.0.0.1 \
--port 8080 \
-ngl 999 \
-ngl all \
-c 32768
```
Important options:
- `--models-dir ~/models` discovers local GGUF files.
- `--no-models-autoload` keeps loading explicit through `/llama`.
- `--jinja` enables compatible chat templates and tool calling.
- `-ngl 999` offloads as many layers as possible to the GPU.
- `-c 32768` sets the context window for each loaded model. Omit it to use the model's native context, which may require substantially more memory.
- `-ngl all` offloads all layers to the GPU. Omit it to use automatic GPU offloading.
- `-c 32768` sets the context size in tokens. Use `-c 0` for the model's full context window, which may require substantially more memory. Context is shared across parallel server slots.
A single-file model can sit directly in the model directory. Put multimodal and multi-shard models in separate subdirectories:
@ -41,7 +52,7 @@ A single-file model can sit directly in the model directory. Put multimodal and
└── large-model-Q4_K_M-00003-of-00003.gguf
```
Restart the router after manually adding files. For per-model context sizes and other options, use [llama.cpp model presets](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#model-presets). For settings tuned to a specific NVIDIA GPU and model, see [Run Pi with NVIDIA RTX and llama.cpp](nvidia-rtx-llama-cpp.md).
Restart the router after manually adding files. The router also supports cached models and preset files via `--models-preset`, as described in the [server guide](https://llama.app/docs/serve#serving-multiple-models-router-mode).
## Configure Pi
@ -63,7 +74,7 @@ export LLAMA_API_KEY=optional-secret
pi
```
If the server uses an API key, start `llama-server` with the matching `--api-key` value. Keep `--host 127.0.0.1` for local-only access.
If the server uses an API key, start `llama serve` with the matching `--api-key` value. Keep `--host 127.0.0.1` for local-only access.
## Manage models