diff --git a/packages/coding-agent/docs/docs.json b/packages/coding-agent/docs/docs.json index 4382848de..c9df247a3 100644 --- a/packages/coding-agent/docs/docs.json +++ b/packages/coding-agent/docs/docs.json @@ -47,6 +47,10 @@ "title": "Run Local Models", "path": "llama-cpp.md" }, + { + "title": "Run Local Models on NVIDIA RTX", + "path": "nvidia-rtx-llama-cpp.md" + }, { "title": "Configure Your Terminal", "path": "terminal-setup.md" diff --git a/packages/coding-agent/docs/llama-cpp.md b/packages/coding-agent/docs/llama-cpp.md index ea51675e0..d4c41ab2e 100644 --- a/packages/coding-agent/docs/llama-cpp.md +++ b/packages/coding-agent/docs/llama-cpp.md @@ -41,7 +41,7 @@ A single-file model can sit directly in the model directory. Put multimodal and └── large-model-Q4_K_M-00003-of-00003.gguf ``` -Restart the router after manually adding files. For per-model context sizes and other options, use [llama.cpp model presets](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#model-presets). +Restart the router after manually adding files. For per-model context sizes and other options, use [llama.cpp model presets](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#model-presets). For settings tuned to a specific NVIDIA GPU and model, see [Run Pi with NVIDIA RTX and llama.cpp](nvidia-rtx-llama-cpp.md). ## Configure Pi diff --git a/packages/coding-agent/docs/nvidia-rtx-llama-cpp.md b/packages/coding-agent/docs/nvidia-rtx-llama-cpp.md new file mode 100644 index 000000000..6a6163034 --- /dev/null +++ b/packages/coding-agent/docs/nvidia-rtx-llama-cpp.md @@ -0,0 +1,97 @@ +# Run Pi with NVIDIA RTX and llama.cpp + +[NVIDIA Local AI Inference Recipes](https://nvidia.github.io/local-ai-inference-recipes/index.html) provides llama.cpp settings tuned for specific NVIDIA hardware and models. Each recipe gives a single-model `llama-server` command. Pi works with the llama.cpp router, which can serve and manage several models, so this guide converts the recipe command into a router preset. + +The example below uses Windows PowerShell. See [Run Pi on Windows](windows.md) for native Windows and WSL setup, and [Local Models with llama.cpp](llama-cpp.md) for Pi's complete router and model-management reference. + +## Choose a recipe + +Open the [recipe catalog](https://nvidia.github.io/local-ai-inference-recipes/index.html), select your hardware, operating system, and model, then choose **llama.cpp**. Check the prerequisites and memory requirements. Keep the recipe's launch command for the conversion below. + +## Install llama.cpp + +Follow the recipe's setup instructions to install llama.cpp. Use a current build with router and model-preset support. + +## Create a model preset + +A preset lists the settings for one or more models. Each section name becomes the model ID shown in Pi. Save the preset next to `llama-server`, or pass its full path when starting the router. + +Convert each option from the recipe command into one preset line: + +- Remove the leading dashes: `--ctx-size 65536` becomes `ctx-size = 65536`. +- Write options without a value as `true`: `--jinja` becomes `jinja = true`. +- Remove quotes and PowerShell line-continuation backticks around values. +- Leave out `--host` and `--port`. They belong on the router command line. +- Add `load-on-startup = true` so the router loads the model when it starts. +- Replace short model-file options with their long preset keys: + +| Recipe option | Preset key | +|---|---| +| `-hf` | `hf-repo` | +| `-hff` | `hf-file` | +| `-hfrd` | `hf-repo-draft` | +| `--spec-draft-model` | `spec-draft-model` | + +For example, this recipe command: + +```powershell +llama-server.exe -hf "unsloth/Qwen3.8-27B-GGUF" ` + -hff "Qwen3.8-27B-UD-Q4_K_M.gguf" ` + -hfrd "z-lab/Qwen3.8-27B-DFlash2-GGUF" ` + --spec-draft-model "Qwen3.8-27B-DFlash2-Q4_K_M.gguf" ` + --no-mmproj ` + --host 127.0.0.1 ` + --port 9931 ` + --batch-size 4096 ` + --ubatch-size 1024 ` + --ctx-size 65536 ` + --parallel 1 ` + -ngl 999 ` + --jinja ` + --chat-template-kwargs '{"preserve_thinking":true}' ` + --spec-type draft-dflash ` + --spec-draft-n-max 7 ` + --spec-draft-backend-sampling +``` + +becomes `qwen3.8-27b.ini`: + +```ini +version = 1 + +[qwen3.8-27b] +load-on-startup = true +hf-repo = unsloth/Qwen3.8-27B-GGUF +hf-file = Qwen3.8-27B-UD-Q4_K_M.gguf +hf-repo-draft = z-lab/Qwen3.8-27B-DFlash2-GGUF +spec-draft-model = Qwen3.8-27B-DFlash2-Q4_K_M.gguf +no-mmproj = true +batch-size = 4096 +ubatch-size = 1024 +ctx-size = 65536 +parallel = 1 +ngl = 999 +jinja = true +chat-template-kwargs = {"preserve_thinking":true} +spec-type = draft-dflash +spec-draft-n-max = 7 +spec-draft-backend-sampling = true +``` + +## Start the router + +Open PowerShell in the llama.cpp folder and run: + +```powershell +.\llama-server.exe --models-preset qwen3.8-27b.ini --host 127.0.0.1 --port 8080 +``` + +If the recipe requires a Hugging Face token, set `$env:HF_TOKEN` in the same PowerShell window before starting the router. On its first start, the router downloads the model files. + +Use the [router health checks](llama-cpp.md#troubleshooting) to confirm that the server is ready. In Windows PowerShell, use `curl.exe` rather than `curl`, which PowerShell maps to a different command. The model is ready when `/models` lists `qwen3.8-27b` with the status `loaded`. + +## Connect Pi + +Follow [Configure Pi](llama-cpp.md#configure-pi) and connect to `http://127.0.0.1:8080`. The preset section name is the model ID, so select `qwen3.8-27b` in `/model` for this example. + +NVIDIA recipe memory settings assume that only the selected model uses the GPU. If `/llama` shows other loaded models, unload them before using the recipe model.