In pi-durable, a Code Mode script that called the built-in `read` on an
image got `""` back, so `image(await tools.read({ path: "pixel.png" }))`
threw a TypeError. `read` declared no `structuredOutputSchema`. Code
Mode resolves a schema-less tool with only its text, and drops images on
purpose. `read`'s output for an image is a single image block with no
text, so the script got an empty string. The coding agent fixed the same
problem for #10251.
With this change, `read` declares a `structuredOutputSchema`: a string,
or an image block `{ type: "image", data, mimeType }`. It returns the
matching `structuredOutput`:
- for a text file, its text, so scripts get the same value as before;
- for an image, its `ImageContent`, which `image()` accepts.
The "Read image file" note stays an info diagnostic, as before. The
README section on structured output and the changelog are updated.
**Testing**
- New test in `test/codemode.test.ts`. A script reads a real PNG and a
text file through `tools.read()`, checks the block's type, MIME type,
and data, and passes the block to `image()`. It checks that the script
succeeds, the image is saved and shown, and the text read still returns
its text. The test fails without the change: the script errors because
`image()` gets `""`.
- `test/tools-read-differential.test.ts`: the reference implementation
now includes the new `structuredOutput` field.
- `npm run check`.
- `test/codemode.test.ts`, `codemode-describe`, `tools`, `tools-images`,
`tools-read-differential`, `harness-tools`, `harness-hook-reads`,
`harness-read-only`, and `spec-usage` all pass.
Fixes#11015
Requested by @badlogic
Co-authored-by: triagent <triagent@localhost>
OutputBuffer, Progress, boundOutput, sanitizeOutput, characterEnd and the
output types move from Durable's internal harness/output.ts to the new
Chord subpath, with OutputSkip (the shape of Durable's ShellOutputSkip) and
utf8ByteLength. Durable imports them from Chord; its truncate.ts re-exports
Chord's utf8ByteLength instead of keeping a second copy.
read returns an image file as one image block (programs get an ImageContent) plus an info diagnostic,
instead of an unsupported_image error. Limits are the conversation model's inputLimits.images.resize,
by default 2000x2000 pixels and 4.5 MB of base64.
- Without a processor: PNG, JPEG, GIF, and WebP within the limits pass through undecoded; the byte limit
is checked before reading the file, dimensions from the header. Larger images and BMP are error results.
- With createCodingTools({ images }): the processor orients by EXIF, converts BMP to PNG, and shrinks to
fit. ImageProcessor.prepare is async.
- @earendil-works/pi-durable/images: createPhotonImages(wasm) on @silvia-odwyer/photon 0.3.3 (web build);
/images/node and /images/cloudflare load the wasm. The root and ./tools entries never reach it (enforced
in check-entry-graphs).
- An image must be read whole: one that changes while read is read again, unlike a growing text file.
- A model without image input gets a diagnostic that it sees a placeholder.
Nested tool calls: ToolExecutionApi.executeTool() runs another tool as its own pi.tool task owned by the caller,
keyed for replay-safe reattachment. Results for programs live in the caller's NestedResultDoc until it settles;
pi.live.nestedTools shows each call with the arguments it runs with and a summary. TaskOptions.abandonOnRestart
aborts work its creator awaited only in memory before it runs again after a restart.
Structured output: structuredOutputSchema and structuredOutput, validated; a tool without a schema gives programs its
bounded output (one text as a string, one image as itself, else the list). ToolExecutionResult.content is now output.
callers and modelTools decide who may call a tool.
Storage errors: any error a Storage method throws fails the Session, which closes itself; later calls and waits reject
with SessionFailed, watches end with session_failed, and Session.closed settles. Nothing retries in the Harness.
StorageRejected is removed; StorageRequestError marks invalid reads that fail only their call. Listener and onReport
errors are contained and reported.
Also updates scripts/issues/triage.ts and the vacation experiment for the content -> output rename.
- BinaryReader.scanLines(): one pass inside the environment locates and
measures a span of lines, decoded exactly like the whole file
- read detects images from the header and PNG chunk headers, scans once,
and decodes only the head it shows; differential test against the
whole-file algorithm
- StreamDecoder: Node's streaming TextDecoder dropped a U+FEFF after a chunk
boundary in shell output and line readers
- fix the harness test to pick its tool result by call id
