mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
fix(durable): rank tool search like the coding agent (#11631)
Ports the coding agent's tool ranking fixes for #11331 (#11440, #11625) to durable, which keeps its own copy of the ranker in `src/search.ts`. Problem: durable's `rankDocuments()` was still the coding agent's ranker from before both fixes. `thread_read` with `limit: 1` returned `queue_read`, and a tool name among other words (`peekaboo app`) lost to tools that only mention the words in their descriptions. Change: - `src/search.ts`: same scoring as the coding agent's `Bm25Ranker`, plus coverage first: a tool matching more of the query's terms (in any field) ranks above one matching fewer, whatever the scores. Without it, name weighting regressed `screenshot window` (1 to 2, `window` beat `image`) and `describe app` (1 to 2, `app` beat `see`). The name is a separate field (3x IDF per term, above any number of description mentions), a tool whose whole local name is in the query gets a bonus worth one more match of every query term, a query that is exactly a tool's name ranks it first, and namespace text is a separate 0.5x field that is not length-normalized. - `tool_search`: the search text no longer repeats the name; the ranker reads it from `name`. - `searchTools()`: same, and the namespace text now includes the namespace instructions, as in the coding agent. - `test/search.test.ts`: the coding agent's ranker cases. Remaining difference from the coding agent: durable's `tool_search` has no namespaces (they are a code mode option), and `searchTools()` ranks on the tool sample from the start data (description plus TypeScript declaration) rather than description plus schema text. `searchTools()` results are journaled, so replaying a script started before this change returns the old results; no `CODEMODE_MODULE_VERSION` bump. ## Before/after `searchTools()` ranking on a made-up set of 16 MCP tools from 5 servers. Each tool is searched with its description; each server is a code mode namespace (prefix `mcp__<server>__`) with the description and instructions below. Rank of the wanted tool among all matches, with the top result when it is another tool. | Server | Description | Instructions | Tools: description | |---|---|---|---| | `github` | GitHub repositories, issues, and pull requests | Use owner/repo for repositories. Issues and pull requests share numbers. | `list_issues`: List issues in a repository, filtered by state, labels, or assignee.<br>`get_issue`: Get an issue with its comments by number.<br>`create_issue`: Open a new issue in a repository.<br>`search_code`: Search code across repositories.<br>`create_pull_request`: Open a pull request from a branch. | | `linear` | Linear issue tracker | Search issues before creating new ones. Issues belong to teams. | `search_issues`: Search Linear issues by text.<br>`create_issue`: Create a Linear issue in a team. | | `peekaboo` | macOS automation | Peekaboo controls macOS apps. Use app names or bundle IDs. Peekaboo can capture any app window. | `app`: Launch, quit, relaunch, focus, hide, or switch applications. Targets an application by name, bundle identifier, or process ID, and can wait until it is ready.<br>`see`: Capture the app window and describe the UI elements of the app.<br>`window`: Move, resize, or close a window of an app.<br>`menu`: Click a menu item in an app.<br>`image`: Take a screenshot of the screen, an app, or a window. | | `app` | Thread and queue state of the app | Threads hold messages; the queue holds pending messages of a thread. | `queue_read`: Read a queued message of this thread.<br>`thread_read`: Read durable state and a paginated timeline from any thread. The default view returns user messages, assistant messages and plans. Continue with afterPosition until hasMore is false. | | `docs` | Project documentation | Search the docs before answering questions about the API. | `search`: Search the documentation.<br>`read_page`: Read a documentation page by path. | Before, a tool's search text was its name, the name with `_` as spaces, its description, and the namespace name and description (no instructions). After, the name and the namespace (with instructions) are separate fields. | Query | Wanted | Before: rank | After: rank | Change | |---|---|---|---|---| | `thread_read` | `app__thread_read` | 2 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `mcp__app__thread_read` | `app__thread_read` | 2 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `peekaboo app` | `peekaboo__app` | 5 (top: `peekaboo__see`) | 1 | $\color{green}{\textsf{better}}$ | | `app` | `peekaboo__app` | 4 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `read thread messages` | `app__thread_read` | 2 (top: `app__queue_read`) | 1 | $\color{green}{\textsf{better}}$ | | `open the peekaboo app tool` | `peekaboo__app` | 5 (top: `peekaboo__see`) | 1 | $\color{green}{\textsf{better}}$ | | `launch app` | `peekaboo__app` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `create linear issue` | `linear__create_issue` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `search documentation` | `docs__search` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `read docs` | `docs__read_page` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `pull request` | `github__create_pull_request` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `get issue comments` | `github__get_issue` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `screenshot window` | `peekaboo__image` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `describe app` | `peekaboo__see` | 1 | 1 | $\color{gray}{\textsf{same}}$ | | `create issue` | `github__create_issue` | 2 (top: `linear__create_issue`) | 2 (top: `linear__create_issue`) | $\color{gray}{\textsf{same}}$ | | `search github issues` | `github__list_issues` | 5 (top: `github__search_code`) | 5 (top: `github__search_code`) | $\color{gray}{\textsf{same}}$ | No regressions on this set. `search github issues` stays at 5: `list_issues` does not contain "search", so it ties on coverage (2 of 3 terms) with `search_code` and `linear__search_issues`, and their scores win.
This commit is contained in:
parent
e033f255f0
commit
c7372af420
6 changed files with 251 additions and 55 deletions
|
|
@ -3808,8 +3808,14 @@ declares them, built from the resolved tools. Loaded tools follow the other
|
|||
tools in load order, so a load appends declarations to the transcript and the
|
||||
cached prefix holds. `@earendil-works/pi-durable/tool-search` provides the
|
||||
coding agent's `tool_search` (`createToolSearchExtension()`): it ranks
|
||||
`Agent.deferred` with BM25 over the tools' names, descriptions, and parameter
|
||||
names and descriptions, and loads the matches. The `codemode` section does not
|
||||
`Agent.deferred` with the coding agent's tool ranking and loads the matches.
|
||||
The ranking is BM25 over the tools' descriptions and parameter names and
|
||||
descriptions, with the name as a separate field: a tool matching more of the
|
||||
query's terms ranks above one matching fewer, a query term in the name
|
||||
outweighs any number of mentions in the description, a tool whose whole name
|
||||
after its last `__` is in the query gets a large bonus, and a query that is
|
||||
exactly a tool's name ranks it first. Code mode's `searchTools()` ranks the
|
||||
same way, with each tool's namespace text as a further, low-weight field. The `codemode` section does not
|
||||
list deferred tools; scripts find them with `searchTools()`.
|
||||
|
||||
Tools reach files and processes only through `api.env`, never through an
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue