createNodePhotonImages() now runs Photon in a worker thread, so preparing a large photo (a second or two) no longer blocks the event loop: in the test a 6-megapixel image blocked the loop over 200 ms inline and under 100 ms with the worker. The worker keeps the process alive only while it prepares an image, exits after 30 s without images to free Photon's WebAssembly memory, and a worker that cannot start falls back to the calling thread. Options: { worker: false, idleMs }.
The Session kept every document it loaded or created in memory for the life of the process: every conversation and
session document ever touched, built-in and third-party. Now a cached document that no watch or document state
observes leaves the cache once it was not read or written for `documentRetentionMs` (HarnessSettings and
createSession options; default ten minutes, 0 drops it right after use, Infinity keeps documents; anything that is not
a number of milliseconds means the default), and loads from Storage again on its next access. An observer pins the
incarnation it observes and releases it when that incarnation retires.
Eviction runs on the mutation line, after every commit outcome (published, no-write, rejected, failed settlement) and
from one unreferenced timer for the next expiry, so a transaction never loses a tracker it holds. Where timers cannot
be unreferenced only commits evict. Cache bookkeeping is contained: a throwing clock or setting never changes an
operation's outcome. A hot document moves in the eviction order at most once per granule (a tenth of the retention,
up to 1 s), keeping the fan-out bench at HEAD's speed.
`npm test` runs the durable suite a second time with every unattached document evicted right after use.
5,000 conversations kept 26,000 documents and about 3.8 KB each; with a 50 ms retention, after waiting, none
(bench:document-cache).
Session, Harness, and runtime reads such as snapshot(), documentState(), watchDoc(), and getTask() may queue on the mutation line a commit callback holds; on a cache miss they never resolve. Documented on Session.commit, TaskRuntime.commit, and in the spec's pitfalls.
NestedCallsDoc (one index per caller, a key per nested call, copied on every admission) and NestedResultDoc are
replaced by NestedCallDoc (`pi.tool.nested-call`): a task-scoped family of the caller keyed by the call's key,
holding { taskId, result? }. Admission creates the member; the nested call's terminal commit adds its result;
executeTool() reads it by key. No commit rewrites another call's document.
TaskRuntime.ownedTasks(context) returns copies of the live tasks a task owns directly, whichever invocation
created them, from the scheduler's index. A tool call settling aborts the live nested calls among them, which
the removed index used to list. A task memo marks a tool that made a nested call in any attempt, so settling a
leaf still skips scanning pi.live.
8000 parallel nested calls: 9.2 s -> 2.7 s, linear (bench:scheduler-fanout).
read returns an image file as one image block (programs get an ImageContent) plus an info diagnostic,
instead of an unsupported_image error. Limits are the conversation model's inputLimits.images.resize,
by default 2000x2000 pixels and 4.5 MB of base64.
- Without a processor: PNG, JPEG, GIF, and WebP within the limits pass through undecoded; the byte limit
is checked before reading the file, dimensions from the header. Larger images and BMP are error results.
- With createCodingTools({ images }): the processor orients by EXIF, converts BMP to PNG, and shrinks to
fit. ImageProcessor.prepare is async.
- @earendil-works/pi-durable/images: createPhotonImages(wasm) on @silvia-odwyer/photon 0.3.3 (web build);
/images/node and /images/cloudflare load the wasm. The root and ./tools entries never reach it (enforced
in check-entry-graphs).
- An image must be read whole: one that changes while read is read again, unlike a growing text file.
- A model without image input gets a diagnostic that it sees a placeholder.
Nested tool calls: ToolExecutionApi.executeTool() runs another tool as its own pi.tool task owned by the caller,
keyed for replay-safe reattachment. Results for programs live in the caller's NestedResultDoc until it settles;
pi.live.nestedTools shows each call with the arguments it runs with and a summary. TaskOptions.abandonOnRestart
aborts work its creator awaited only in memory before it runs again after a restart.
Structured output: structuredOutputSchema and structuredOutput, validated; a tool without a schema gives programs its
bounded output (one text as a string, one image as itself, else the list). ToolExecutionResult.content is now output.
callers and modelTools decide who may call a tool.
Storage errors: any error a Storage method throws fails the Session, which closes itself; later calls and waits reject
with SessionFailed, watches end with session_failed, and Session.closed settles. Nothing retries in the Harness.
StorageRejected is removed; StorageRequestError marks invalid reads that fail only their call. Listener and onReport
errors are contained and reported.
Also updates scripts/issues/triage.ts and the vacation experiment for the content -> output rename.
runtime.context() keeps each conversation's last read range and derived
view. A later read by any of the conversation's tasks with the same head
marker scans and derives only the entries committed since, so a
generation no longer rereads and re-derives the whole transcript for
each model request. Busy conversations keep the range; idle ones for
settings.contextRetentionMs (default ten minutes), checked on task
changes and by one unreferenced timer where the runtime supports it.
Kept entries are frozen; each read returns its own view arrays.
Add openDurableObjectSqliteStorage(ctx.storage) in
@earendil-works/pi-durable/storage/sqlite/cloudflare: SQLite storage on
a SQLite-backed Cloudflare Durable Object, without per-row copies.
A run's input is committed before the system prompt is rendered, so requests
started with the user message. Anthropic then sent the tool list at the top of
every request and each tool change invalidated the whole prompt cache. Context
derivation now moves a system message preceded only by user messages to the
front. A live e2e test gates the cache reuse across a tool change.
closes#10542
pi-ai's AssistantMessageEventStream sets AssistantMessage.durationMs with a
monotonic clock for every API implementation. The agent loop and the durable
tool task set ToolResultMessage.durationMs around execute(). Durable task
records carry startedAt/endedAt stamped by the Session. The coding agent's bash
renderer shows Took from the recorded duration, also after a reload.
closes#10549
- required watch(targets, onChange): missing targets, recursive targets
with excludes, paths/overflow/error changes, native or polling mode
- NodeExecutionEnv: fs.watch events trigger rescans; changes are snapshot
differences plus event paths; watchers for new directories before rescans;
polling on network and FUSE file systems; directory budget
- conformance cases in native and polling mode
- BinaryReader.scanLines(): one pass inside the environment locates and
measures a span of lines, decoded exactly like the whole file
- read detects images from the header and PNG chunk headers, scans once,
and decodes only the head it shows; differential test against the
whole-file algorithm
- StreamDecoder: Node's streaming TextDecoder dropped a U+FEFF after a chunk
boundary in shell output and line readers
- fix the harness test to pick its tool result by call id
- ShellExecOptions.window and ShellOutputInfo.skipped: environments may omit
output outside the retained tail and report exact counts
- ToolExecutionApi.outputWindow (tail limits and progress pace) and
output(chunk, skipped); the bash tool passes both through
- OutputBuffer counts skips; property tests against the unskipped stream
- fix: progress snapshots no longer change which tail is kept
settings.progress.partialIntervalMs and outputIntervalMs replace the hardcoded
100 ms throttles for generation partials and running tool output.
refs #10357
- FileSystem.openBinaryReader(): positional reads of one opened regular file, noFollow
- FileSystem.openDirReader(): paged listings, vanished entries skipped
- Shell.exec() accepts an argv array (no shell); onOutput reports the stream
- Env conformance suite in /testing, run for NodeExecutionEnv
- Work-branch workflow running env tests on Linux, macOS and Windows
Assign each conversation its own UUIDv7 and forward it to model requests for prompt-cache affinity. Lazily backfill legacy conversations and cover real Codex cache reuse.
Closes#10424
Rename docs/pico-v5.md to docs/spec.md and link the design documents on GitHub
instead of shipping docs/. Add a default condition to every export so require()
of the ESM package works. Reset the changelog to the initial release.
Close ends states and watches at its seal, joins non-cooperative code, and a
cancelled close cancels only its wait. Conversation.abort() starts scheduling.
New section 9.5 specifies Harness.taskGraph() and watchTaskGraph(). The Chord
guide uses real imports and documents shutdown order.
Extensions installed by name, the rewindable pi.agent document with configure(),
Harness-wide live settings, env built per use, typed tool args with defineTool(),
HarnessOptions.conversationCreated, Tx submission methods, FileSystem.id for
edit/write serialization, and tool resolution with the tools filter applied.
The subagent tool no longer offers wait, is not rerun after a crash (a repeated stop could stop newer work), reports each answer once even when several messages end in it, checks registry names as own properties, and its scenes cannot hang polling.
Generation offers active registered tools through positional system entries and
hands a tool-calling answer to parallel or sequential pi.tool tasks and a
pi.post-tools task that applies addTools/terminate and continues the run. Tool
tasks validate, run beforeTool/afterTool, record intent with the replay policy,
publish bounded, sanitized, adaptively throttled progress in pi.live.tools, and
write results with a rendered diagnostics block. Hook dispatch with conversation
and subtree scopes, prepareArguments, HarnessOptions.env, and read/bash/edit/write
tools in pi-durable/tools. The env shell now streams raw output and spills past
thresholds; its bounded capture and publisher are removed.
Submissions with request-ID deduplication and handles, the pi.live run
control and presentation document, the built-in pi.generation task
(system prompt preparation, streamed durable partials, retries, deferred
polling, abort), typed entry tokens, stream options and retry policy,
TaskRuntime reads, Tx.settleSubmission, registry conversation setups,
Storage.scanSubmissions, Harness.inspect, and byTaskId attribution.
Waiting calls enable scheduling. The scratch tour is split into
test/examples.
pi.live turn control and generation presentation, built-in entry kinds,
pi.generation checkpoint and phases, typed entry tokens, built-in tasks in
the registry, TaskRuntime reads, stream options and retry policy config,
Harness hook for scheduler-written outcomes, and createConversation without
input.
Adds defineTask, registry-resolved phase handlers, and a scheduler that
decides every task transition in one callback on the Session line:
reservation with migration, dependencies, precedence rules in a step
before each phase, handover to replaced definitions, direct-task abort
(mark, signal, join, fresh abort invocation), orphaning of blocked tasks,
open reconciliation, and close that joins invocations without writing
outcomes. runtime.commit() callbacks return the typed next state; Tx no
longer exposes setTask. Adds Harness.resume, getTask, waitForTask,
abortTask, and task-aware waitForIdle; RegistryReader.subscribe wakes
the scheduler.
Implements Pico5 Package 13: Harness.open over a Session with an application-owned
registry (tools, tool wrappers, hooks, tasks, system prompt sections; batched
publication, stable keyed order), lazy root creation with init, atomic
create/fork with init, conversation-bound commits, fork-aware entries, model
context derivation, and the built-in ConversationConfig document.
Fixes cached documents skipping migration when accessed with another definition
version. Rewrites the Pico5 spec and handoff to the registry/system-prompt
design and removes the superseded live-registries proposal.
Storage reports replayed deltas after the selected base; Session tracks the
count across adopted writes so checkpoint predicates can bound replay.
Also adds internal reserved-root creation and conversation-bound commit
defaults for the upcoming Harness.
Replace the alternate tracker implementations with the optimized immutable overlay, validate draft placements as strict JSON, and document the ownership contract. Consolidate tracker tests and benchmarks around the canonical implementation.
Name the Astra-immutable-optimized Chord Delta tracker as the Packages 6-7
prerequisite, include the minimal generic Tx table surface in that
milestone, and revoke drafts at callback settlement. Drop the Chord
prerequisite from Package 11 because the existing replicated-state source
contract already matches. Specify that tool stream() never spills; tools
spill through their execution environment.
Port and extend the agent environment tests for NodeExecutionEnv, output
capture, adaptive publishing, and truncation, including exact-byte
truncation, flushing, line-reader cancellation, spill backpressure, and
Buffer-free UTF-8 counting.
Fix truncation reporting when bytes and lines are both exceeded, when only
bytes are exceeded at the line cap, and when small limits trim multi-byte
edge characters. Report the full size of oversized newline-terminated tail
lines. Resolve durable test imports from source so tests run unbuilt.
Set Pico tool output defaults to 50 KiB and 2,000 lines.
