From 59ca40cf5890f3d72056d3c1ed644db7339bb930 Mon Sep 17 00:00:00 2001 From: Julien Chaumond Date: Tue, 27 Oct 2026 16:21:37 +0100 Subject: [PATCH] docs(coding-agent): update llama.cpp setup for llama.app (#10179) update the llama.cpp docs to use the llama.app installer and the new `llama serve` command commit by @hanouticelina disclaimer: my julien-cto agent helped me write this --------- Co-authored-by: Celina Hanouti Co-authored-by: Mario Zechner --- packages/coding-agent/docs/llama-cpp.md | 35 ++++++++++++++++--------- 1 file changed, 23 insertions(+), 12 deletions(-) diff --git a/packages/coding-agent/docs/llama-cpp.md b/packages/coding-agent/docs/llama-cpp.md index d4c41ab2e..6684bb4df 100644 --- a/packages/coding-agent/docs/llama-cpp.md +++ b/packages/coding-agent/docs/llama-cpp.md @@ -1,31 +1,42 @@ # Local Models with llama.cpp -Pi supports the [llama.cpp](https://github.com/ggml-org/llama.cpp) router server. The router discovers multiple GGUF models and loads or unloads them on demand. +Pi supports the [llama.cpp](https://llama.app) router server. The router discovers multiple GGUF models and loads or unloads them on demand. -Use a current llama.cpp build with router support. Follow the [build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md) or install a [prebuilt release](https://github.com/ggml-org/llama.cpp/releases) for your platform. +## Install llama.cpp + +Install llama.cpp with the [official installer](https://llama.app/docs/installation): + +```bash +curl -LsSf https://llama.app/install.sh | sh +``` + +Verify the installation: + +```bash +llama cli --version +``` + +See [llama.app](https://llama.app) for other installation methods, including package managers and building from source. ## Start the router -Start `llama-server` without `--model` or `-m`. Passing a model starts single-model mode instead of router mode. +Start `llama serve` without `--model`, `-m`, or `-hf`. Passing a model starts single-model mode instead of router mode. See the [llama.cpp server guide](https://llama.app/docs/serve#serving-multiple-models-router-mode) for router configuration. ```bash -llama-server \ - --models-dir ~/models \ +llama serve \ --no-models-autoload \ - --jinja \ --host 127.0.0.1 \ --port 8080 \ - -ngl 999 \ + -ngl all \ -c 32768 ``` Important options: -- `--models-dir ~/models` discovers local GGUF files. - `--no-models-autoload` keeps loading explicit through `/llama`. - `--jinja` enables compatible chat templates and tool calling. -- `-ngl 999` offloads as many layers as possible to the GPU. -- `-c 32768` sets the context window for each loaded model. Omit it to use the model's native context, which may require substantially more memory. +- `-ngl all` offloads all layers to the GPU. Omit it to use automatic GPU offloading. +- `-c 32768` sets the context size in tokens. Use `-c 0` for the model's full context window, which may require substantially more memory. Context is shared across parallel server slots. A single-file model can sit directly in the model directory. Put multimodal and multi-shard models in separate subdirectories: @@ -41,7 +52,7 @@ A single-file model can sit directly in the model directory. Put multimodal and └── large-model-Q4_K_M-00003-of-00003.gguf ``` -Restart the router after manually adding files. For per-model context sizes and other options, use [llama.cpp model presets](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#model-presets). For settings tuned to a specific NVIDIA GPU and model, see [Run Pi with NVIDIA RTX and llama.cpp](nvidia-rtx-llama-cpp.md). +Restart the router after manually adding files. The router also supports cached models and preset files via `--models-preset`, as described in the [server guide](https://llama.app/docs/serve#serving-multiple-models-router-mode). ## Configure Pi @@ -63,7 +74,7 @@ export LLAMA_API_KEY=optional-secret pi ``` -If the server uses an API key, start `llama-server` with the matching `--api-key` value. Keep `--host 127.0.0.1` for local-only access. +If the server uses an API key, start `llama serve` with the matching `--api-key` value. Keep `--host 127.0.0.1` for local-only access. ## Manage models