mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
c7372af420
Ports the coding agent's tool ranking fixes for #11331 (#11440, #11625) to durable, which keeps its own copy of the ranker in `src/search.ts`. Problem: durable's `rankDocuments()` was still the coding agent's ranker from before both fixes. `thread_read` with `limit: 1` returned `queue_read`, and a tool name among other words (`peekaboo app`) lost to tools that only mention the words in their descriptions. Change: - `src/search.ts`: same scoring as the coding agent's `Bm25Ranker`, plus coverage first: a tool matching more of the query's terms (in any field) ranks above one matching fewer, whatever the scores. Without it, name weighting regressed `screenshot window` (1 to 2, `window` beat `image`) and `describe app` (1 to 2, `app` beat `see`). The name is a separate field (3x IDF per term, above any number of description mentions), a tool whose whole local name is in the query gets a bonus worth one more match of every query term, a query that is exactly a tool's name ranks it first, and namespace text is a separate 0.5x field that is not length-normalized. - `tool_search`: the search text no longer repeats the name; the ranker reads it from `name`. - `searchTools()`: same, and the namespace text now includes the namespace instructions, as in the coding agent. - `test/search.test.ts`: the coding agent's ranker cases. Remaining difference from the coding agent: durable's `tool_search` has no namespaces (they are a code mode option), and `searchTools()` ranks on the tool sample from the start data (description plus TypeScript declaration) rather than description plus schema text. `searchTools()` results are journaled, so replaying a script started before this change returns the old results; no `CODEMODE_MODULE_VERSION` bump. ## Before/after `searchTools()` ranking on a made-up set of 16 MCP tools from 5 servers. Each tool is searched with its description; each server is a code mode namespace (prefix `mcp__<server>__`) with the description and instructions below. Rank of the wanted tool among all matches, with the top result when it is another tool. | Server | Description | Instructions | Tools: description | |---|---|---|---| | `github` | GitHub repositories, issues, and pull requests | Use owner/repo for repositories. Issues and pull requests share numbers. | `list_issues`: List issues in a repository, filtered by state, labels, or assignee.<br>`get_issue`: Get an issue with its comments by number.<br>`create_issue`: Open a new issue in a repository.<br>`search_code`: Search code across repositories.<br>`create_pull_request`: Open a pull request from a branch. | | `linear` | Linear issue tracker | Search issues before creating new ones. Issues belong to teams. | `search_issues`: Search Linear issues by text.<br>`create_issue`: Create a Linear issue in a team. | | `peekaboo` | macOS automation | Peekaboo controls macOS apps. Use app names or bundle IDs. Peekaboo can capture any app window. | `app`: Launch, quit, relaunch, focus, hide, or switch applications. Targets an application by name, bundle identifier, or process ID, and can wait until it is ready.<br>`see`: Capture the app window and describe the UI elements of the app.<br>`window`: Move, resize, or close a window of an app.<br>`menu`: Click a menu item in an app.<br>`image`: Take a screenshot of the screen, an app, or a window. | | `app` | Thread and queue state of the app | Threads hold messages; the queue holds pending messages of a thread. | `queue_read`: Read a queued message of this thread.<br>`thread_read`: Read durable state and a paginated timeline from any thread. The default view returns user messages, assistant messages and plans. Continue with afterPosition until hasMore is false. | | `docs` | Project documentation | Search the docs before answering questions about the API. | `search`: Search the documentation.<br>`read_page`: Read a documentation page by path. | Before, a tool's search text was its name, the name with `_` as spaces, its description, and the namespace name and description (no instructions). After, the name and the namespace (with instructions) are separate fields. | Query | Wanted | Before: rank | After: rank | Change | |---|---|---|---|---| | `thread_read` | `app__thread_read` | 2 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `mcp__app__thread_read` | `app__thread_read` | 2 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `peekaboo app` | `peekaboo__app` | 5 (top: `peekaboo__see`) | 1 | $\color{green}{\textsf{better}}$ | | `app` | `peekaboo__app` | 4 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `read thread messages` | `app__thread_read` | 2 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `open the peekaboo app tool` | `peekaboo__app` | 5 (top: `peekaboo__see`) | 1 | $\color{green}{\textsf{better}}$ | | `launch app` | `peekaboo__app` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `create linear issue` | `linear__create_issue` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `search documentation` | `docs__search` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `read docs` | `docs__read_page` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `pull request` | `github__create_pull_request` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `get issue comments` | `github__get_issue` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `screenshot window` | `peekaboo__image` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `describe app` | `peekaboo__see` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `create issue` | `github__create_issue` | 2 (top: `linear__create_issue`) | 2 (top: `linear__create_issue`) | $\color{gray}{\textsf{same}}$ | | `search github issues` | `github__list_issues` | 5 (top: `github__search_code`) | 5 (top: `github__search_code`) | $\color{gray}{\textsf{same}}$ | No regressions on this set. `search github issues` stays at 5: `list_issues` does not contain "search", so it ties on coverage (2 of 3 terms) with `search_code` and `linear__search_issues`, and their scores win. |
||
|---|---|---|
| .. | ||
| examples | ||
| fixtures | ||
| chat-support.ts | ||
| codemode-bench.ts | ||
| codemode-checkpoints.test.ts | ||
| codemode-describe.test.ts | ||
| codemode-imports.test.ts | ||
| codemode-models.test.ts | ||
| codemode.test.ts | ||
| conversation-transcript.test.ts | ||
| conversation-view-bench.ts | ||
| document-append-bench.ts | ||
| env-line-scan.test.ts | ||
| env-node-conformance.test.ts | ||
| env-node-spill.test.ts | ||
| env-node.test.ts | ||
| env-truncate.test.ts | ||
| harness-after-run.test.ts | ||
| harness-blocked.test.ts | ||
| harness-callers.test.ts | ||
| harness-cancellation-barrier.test.ts | ||
| harness-compaction.test.ts | ||
| harness-context-range.test.ts | ||
| harness-context.test.ts | ||
| harness-continue.test.ts | ||
| harness-conversations.test.ts | ||
| harness-deferred-progress.test.ts | ||
| harness-failed-run-inbox.test.ts | ||
| harness-generation-recovery.test.ts | ||
| harness-generation.test.ts | ||
| harness-group-commit.test.ts | ||
| harness-hook-reads.test.ts | ||
| harness-host-inputs.test.ts | ||
| harness-inbox.test.ts | ||
| harness-input-entries.test.ts | ||
| harness-inspect.test.ts | ||
| harness-json.test.ts | ||
| harness-lifecycle.test.ts | ||
| harness-live-deltas.test.ts | ||
| harness-nested-tools-restart.test.ts | ||
| harness-nested-tools.test.ts | ||
| harness-observers.test.ts | ||
| harness-ownership.test.ts | ||
| harness-partial-changes.test.ts | ||
| harness-prompt.test.ts | ||
| harness-read-only.test.ts | ||
| harness-registry-reader.test.ts | ||
| harness-registry.test.ts | ||
| harness-reports.test.ts | ||
| harness-restart-failure.test.ts | ||
| harness-run-failure.test.ts | ||
| harness-scheduler-indexes.test.ts | ||
| harness-scheduler-retention.test.ts | ||
| harness-sleep.test.ts | ||
| harness-storage-failure.test.ts | ||
| harness-structured-output.test.ts | ||
| harness-structured.test.ts | ||
| harness-submissions.test.ts | ||
| harness-submit-in-commit.test.ts | ||
| harness-support.ts | ||
| harness-task-graph.test.ts | ||
| harness-tasks-recovery.test.ts | ||
| harness-tasks.test.ts | ||
| harness-telemetry-options.test.ts | ||
| harness-telemetry.test.ts | ||
| harness-tool-hooks.test.ts | ||
| harness-tools-recovery.test.ts | ||
| harness-tools.test.ts | ||
| harness-turn-limit.test.ts | ||
| harness-view-run.test.ts | ||
| harness-view.test.ts | ||
| heap-retained.ts | ||
| images-wasm-module.test.ts | ||
| indexeddb-storage.test.ts | ||
| jsonl-storage.test.ts | ||
| live-support.ts | ||
| memory-storage.test.ts | ||
| nested-images-bench.ts | ||
| nested-tools-bench.ts | ||
| provider-session-cache-e2e.test.ts | ||
| scheduler-fanout-bench.ts | ||
| scheduler-memory-bench.ts | ||
| search.test.ts | ||
| session-adoption.test.ts | ||
| session-cached-storage.test.ts | ||
| session-checkpoints-migrations.test.ts | ||
| session-conversation-tails.test.ts | ||
| session-deferred-commits.test.ts | ||
| session-definitions.test.ts | ||
| session-document-cache-bench.ts | ||
| session-document-retention.test.ts | ||
| session-documents.test.ts | ||
| session-forks.test.ts | ||
| session-group-commit.test.ts | ||
| session-observers.test.ts | ||
| session-support.ts | ||
| session-tables.test.ts | ||
| session-watches.test.ts | ||
| setup-document-retention.ts | ||
| spec-usage.test.ts | ||
| sqlite-cloudflare.test.ts | ||
| sqlite-facade.test.ts | ||
| sqlite-migrations.test.ts | ||
| sqlite-storage.test.ts | ||
| storage-memory.ts | ||
| storage-runtime-boundary.test.ts | ||
| storage.bench.ts | ||
| store-storage-commit.test.ts | ||
| store-storage-node.ts | ||
| store-storage-support.ts | ||
| store-storage.test.ts | ||
| system-order-cache-e2e.test.ts | ||
| task-support.ts | ||
| tool-output-bench.ts | ||
| tool-search.test.ts | ||
| tools-images.test.ts | ||
| tools-read-differential.test.ts | ||
| tools.test.ts | ||
| types.test.ts | ||