diff --git a/packages/durable/CHANGELOG.md b/packages/durable/CHANGELOG.md index e35117126..f67553b7e 100644 --- a/packages/durable/CHANGELOG.md +++ b/packages/durable/CHANGELOG.md @@ -16,7 +16,7 @@ - Input submissions take `entry: { kind, data }` (`InputEntry`): the input is placed as an entry of that kind carrying that data, with its content as the user message, instead of `pi.user`, and behaves as any other input; the kind must not start with `pi.`. Queued inputs carry it in their `pi.inbox` item ([#10407](https://github.com/earendil-works/pi/issues/10407)). - `Session.closed` settles once the Session has closed, Storage included: `{ reason: "closed" }` after `close()`, or `{ reason: "failed", error }` after a storage error. `WatchEnd` gains `{ reason: "session_failed", error }`, and the README has an Errors section. - `ToolExecutionApi.executeTool(name, args, context, { key?, progress? })`: a tool calls another tool as a nested call. The nested call is its own `pi.tool` task owned by the caller, runs validation, the `ToolTask` hooks (whose `ToolHookCall` carries `parent`), output limits, and its replay policy, and returns a `NestedToolExecutionResult` (its `taskId`, `structuredOutput`, `details`, `diagnostics`, `usage`, `isError`, `durationMs`, but not the model's `output`) to the caller instead of the transcript. Each nested call is a member of `NestedCallDoc`, a task-scoped document family of the caller keyed by the call's key, holding its `taskId` and, once it ends, its `result` as returned, so it is gone from storage once the caller settles; its task's receipt stays small. Keys, by default the call's position, let a replay-safe caller's rerun reattach to its nested calls; a caller that is not replay-safe has its unfinished nested calls abandoned after a restart. Nested calls show in `pi.live.nestedTools`, with the arguments they run with and a `summary` (error, duration, usage) once done (`progress: false` keeps their running output out), and as tool events with `parentToolCallId` and `parentTaskId`; every tool event also carries its call's `taskId`. A caller aborts the nested calls it left running before it settles. -- Structured tool output: a tool declares `structuredOutputSchema` and returns `structuredOutput`, which the Harness validates and gives to programs that call the tool; a tool without a schema gives them its bounded output, one text item as a string, one image as itself, and otherwise the content list, for errors too. A nested call with invalid structured output gets an error result; a model-issued one only loses it, and the host gets a report. `api.retainedOutput()` reads the bounded output the model will see. `bash` and `powershell` return `{ output, truncated, fullOutputPath?, exitCode }`. +- Structured tool output: a tool declares `structuredOutputSchema` and returns `structuredOutput`, which the Harness validates and gives to programs that call the tool; a tool without a schema gives them its bounded output, one text item as a string, one image as itself, and otherwise the content list, for errors too, and so does a schema tool's error result without `structuredOutput`, so programs get the text of a returned error; a thrown error's message is in its `tool_error` diagnostic. A nested call with invalid structured output gets an error result; a model-issued one only loses it, and the host gets a report. `api.retainedOutput()` reads the bounded output the model will see. `bash` and `powershell` return `{ output, truncated, fullOutputPath?, exitCode }`. - `ToolRegistration.callers` (`model`, `tools`, or both, the default) and `pi.agent` `modelTools`: who may call a tool, and which of its tools a conversation offers the model; `Agent.callable` lists the tools nested calls resolve among. `addTools` also extends `modelTools`. - `TaskOptions.abandonOnRestart`: a task its creator awaits only in memory is aborted, with its owned work, when a later Harness starts scheduling, before any of it runs again. Its mark carries `abortReason: "restart"`, which cascades; such a task waits for a missing definition instead of becoming `orphaned`. Nested calls and `api.createTask()` children of tools that are not replay-safe get it by default. No task below an owner with cancellation intent starts a run phase any more, even before the cascade marks it. - `TaskRuntime.abortOwned(id, context)`: abort a task this task owns and wait until it is terminal. diff --git a/packages/durable/README.md b/packages/durable/README.md index ee5ed92ce..d8f969268 100644 --- a/packages/durable/README.md +++ b/packages/durable/README.md @@ -179,7 +179,7 @@ registry.install(defineExtension({ name: "count", tools: [count] })); Each call runs as its own durable task. Its intent is committed before `execute()` runs. If the process dies mid-call, the tool reruns on reopen only when it is declared `replay: "safe"`; otherwise the model gets an `interrupted` error result with the output committed so far. Throwing from `execute()` gives the model an error result. A result can also return `usage`, which is added to the conversation's [usage](#usage-and-cost). It can also return `control: { terminate: true }`: when every result of the round asks for it, the run ends without another model request. -A result has three channels: `output` for the model, `structuredOutput` for programs that call the tool (see [Calling tools from tools](#calling-tools-from-tools)), and `details` for UIs. A tool that declares `structuredOutputSchema` returns a matching `structuredOutput`, which the Harness validates. Programs calling a tool without one get its bounded output: one text item as a string, one image as its `ImageContent`, nothing as `""`, and anything else as the content list, for errors too. The transcript stores only `output` and `details`: +A result has three channels: `output` for the model, `structuredOutput` for programs that call the tool (see [Calling tools from tools](#calling-tools-from-tools)), and `details` for UIs. A tool that declares `structuredOutputSchema` returns a matching `structuredOutput`, which the Harness validates. Programs calling a tool without one get its bounded output: one text item as a string, one image as its `ImageContent`, nothing as `""`, and anything else as the content list, for errors too. They get the same for a schema tool's error result without `structuredOutput`, so they still see the text of a returned error; a thrown error's message is in its `tool_error` diagnostic. The transcript stores only `output` and `details`: ```typescript const lines = defineTool({ @@ -195,7 +195,7 @@ const lines = defineTool({ }); ``` -A missing, undeclared, or invalid `structuredOutput` turns a nested call's result into an error result with an `invalid_structured_output` diagnostic; for a call the model made, which never sees it, it is dropped and reported through `onReport`. An error result may omit it; error results the Harness writes itself (blocked, invalid, interrupted, aborted) carry none, and their `diagnostics` say what went wrong. An `afterTool` hook that redacts a schema tool's `output` must redact its `structuredOutput` too. The built-in `bash` returns `{ output, truncated, fullOutputPath?, exitCode }`, where `output` is the retained tail the model sees, read with `api.retainedOutput()`, and answers a nonzero exit with an error result that still carries it. +A missing, undeclared, or invalid `structuredOutput` turns a nested call's result into an error result with an `invalid_structured_output` diagnostic; for a call the model made, which never sees it, it is dropped and reported through `onReport`. An error result may omit it, and programs then get its bounded output; error results the Harness writes itself (blocked, invalid, interrupted, aborted) carry none, and their `diagnostics` say what went wrong. An `afterTool` hook that redacts a schema tool's `output` must redact its `structuredOutput` too. The built-in `bash` returns `{ output, truncated, fullOutputPath?, exitCode }`, where `output` is the retained tail the model sees, read with `api.retainedOutput()`, and answers a nonzero exit with an error result that still carries it. A later extension's tool with the same name replaces an earlier one where both are selected, and `wrapTool()` decorates whichever tool won: diff --git a/packages/durable/docs/spec.md b/packages/durable/docs/spec.md index b4233d336..d3fc415ee 100644 --- a/packages/durable/docs/spec.md +++ b/packages/durable/docs/spec.md @@ -2856,7 +2856,7 @@ interface ToolExecutionApi extends Docum type NestedToolExecutionResult = { /** The nested call's tool task. */ readonly taskId: TaskId; - /** A schema tool's validated `structuredOutput`, else its bounded output, collapsed (below). */ + /** A schema tool's validated `structuredOutput`; else, and for an error result without one, its bounded output, collapsed (below). */ readonly structuredOutput?: JsonValue; readonly isError: boolean; readonly details?: JsonValue; @@ -2984,14 +2984,19 @@ on an error result, where it may omit it; the Harness validates it after the programs get its bounded output, the content the model sees without the rendered diagnostics, for a success and for an error alike: one text item as its string, one image as its `ImageContent`, no items as `""`, and anything -else as the content list. So a -hook that redacts a schema-less tool's output redacts what programs get. An +else as the content list. A schema tool's error result without +`structuredOutput` gives programs its bounded output the same way, so they get +the text of a returned error, for example `"permission denied: /etc"`. A +thrown error's message is in its `tool_error` diagnostic, not in the output, +so programs read it there, as they do for any tool. So a +hook that redacts a tool's output redacts what programs get of a schema-less +tool and of an error result without structured output. An error result the Harness writes itself (unavailable, invalid arguments, blocked, interrupted, aborted, abandoned, or a scheduler fault) has no -`structuredOutput`; its diagnostics say what went wrong, as they do for a -schema tool's error result without one. A nested call whose result breaks this gets an error result -with an `invalid_structured_output` diagnostic, without the broken value: a -schema tool's then has no `structuredOutput`, a schema-less tool's its output; +`structuredOutput`; its diagnostics say what went wrong. A thrown error +becomes a `tool_error` diagnostic with its message. A nested call whose result breaks this gets an error result +with an `invalid_structured_output` diagnostic, without the broken value, and +with its bounded output as `structuredOutput`; a model-issued call, whose `structuredOutput` the model never sees, only loses it, and the break is reported through `HarnessOptions.onReport`. `structuredOutput` has no size bound; keeping it small is the tool's job, as `bash` keeps its retained tail and the diff --git a/packages/durable/src/harness/tool.ts b/packages/durable/src/harness/tool.ts index bb2036711..be40c8ad4 100644 --- a/packages/durable/src/harness/tool.ts +++ b/packages/durable/src/harness/tool.ts @@ -528,8 +528,11 @@ async function finalResult( runtime.report(new Error(broken)); } } - // Without a schema, programs get the output itself; the model reads only `output`, so model-issued calls skip it. - if (input.kind === "nested" && tool.structuredOutputSchema === undefined) { + // Without a schema, or for an error result without structured output, programs get the output itself, so they get + // the error's text; the model reads only `output`, so model-issued calls skip it. + const schemaValue = + tool.structuredOutputSchema !== undefined && !(final.isError === true && final.structuredOutput === undefined); + if (input.kind === "nested" && !schemaValue) { final = { ...final, structuredOutput: outputValue(bounded.content) as JsonValue }; } return { ...final, output: bounded.content, diagnostics: [...(final.diagnostics ?? []), ...harness] }; diff --git a/packages/durable/src/harness/types.ts b/packages/durable/src/harness/types.ts index 60bb8966c..6548de07c 100644 --- a/packages/durable/src/harness/types.ts +++ b/packages/durable/src/harness/types.ts @@ -177,10 +177,10 @@ export type NestedToolExecutionResult = { /** The nested call's tool task. */ readonly taskId: TaskId; /** - * The value for programs. A tool with `structuredOutputSchema`: its validated `structuredOutput`, absent on an error - * result that has none. A tool without: its bounded output, success or error, as a string for one text item, an - * `ImageContent` for one image, `""` for none, and the content list otherwise. Absent on an error result the Harness - * wrote itself (unavailable, invalid, blocked, interrupted, aborted); `diagnostics` say what went wrong. + * The value for programs. A tool with `structuredOutputSchema`: its validated `structuredOutput`. A tool without, and + * an error result without `structuredOutput`: its bounded output, as a string for one text item, an `ImageContent` + * for one image, `""` for none, and the content list otherwise. Absent on an error result the Harness wrote itself + * (unavailable, invalid, blocked, interrupted, aborted); `diagnostics` say what went wrong. */ readonly structuredOutput?: JsonValue; readonly isError: boolean; diff --git a/packages/durable/test/harness-structured-output.test.ts b/packages/durable/test/harness-structured-output.test.ts index a12535995..97dde4216 100644 --- a/packages/durable/test/harness-structured-output.test.ts +++ b/packages/durable/test/harness-structured-output.test.ts @@ -208,9 +208,8 @@ describe("structured output", () => { addTool(setup.registry, returning(name, result, schema ? { structuredOutputSchema: Count } : {})); const received = await nested(setup, name); expect(received.isError).toBe(true); - // The broken value is dropped; a schema-less tool's error result still carries its output, here none. - if (schema) expect(received).not.toHaveProperty("structuredOutput"); - else expect(received.structuredOutput).toBe(""); + // The broken value is dropped; the error result carries the bounded output instead, here none. + expect(received.structuredOutput).toBe(""); expect(codes(received)).toEqual(["invalid_structured_output"]); expect(received.diagnostics[0]?.message).toContain(message); @@ -237,7 +236,7 @@ describe("structured output", () => { expect(received).not.toHaveProperty("structuredOutput"); }); - it("lets an error result of a schema tool omit structured output, and keeps none of its output", async () => { + it("gives callers the output of a schema tool's error result without structured output", async () => { const setup = chatSetup(); addTool( setup.registry, @@ -247,9 +246,38 @@ describe("structured output", () => { { structuredOutputSchema: Count }, ), ); - const received = await nested(setup, "fails"); - expect(received).toMatchObject({ isError: true, diagnostics: [] }); - expect(received).not.toHaveProperty("structuredOutput"); + // Its text reaches the caller: what a code mode script's rejection carries. + expect(await nested(setup, "fails")).toMatchObject({ isError: true, structuredOutput: "nope", diagnostics: [] }); + + // A thrown error's message is in its tool_error diagnostic, not in the output. + const thrown = chatSetup(); + addTool( + thrown.registry, + defineTool({ + name: "throws", + description: "throws", + parameters: Type.Object({}), + structuredOutputSchema: Count, + execute: async () => { + throw new Error("permission denied"); + }, + }), + ); + expect(await nested(thrown, "throws")).toMatchObject({ + isError: true, + structuredOutput: "", + diagnostics: [{ code: "tool_error", message: "permission denied" }], + }); + + // An error result that has structured output keeps it. + const kept = chatSetup(); + const withValue = { + output: [{ type: "text" as const, text: "nope" }], + structuredOutput: { count: 0 }, + isError: true, + }; + addTool(kept.registry, returning("fails", withValue, { structuredOutputSchema: Count })); + expect(await nested(kept, "fails")).toMatchObject({ isError: true, structuredOutput: { count: 0 } }); }); it("keeps structured output when afterTool replaces the output, and validates one afterTool replaces", async () => {