Commit graph

136 commits

Author SHA1 Message Date
Armin Ronacher
748cd5ac57 feat(coding-agent): run code mode on dquickts instead of quickjs-wasi
Code mode scripts now run on @earendil-works/dquickts 0.2.0 (Bellard's
QuickJS compiled to WebAssembly) instead of quickjs-wasi (QuickJS-NG):
about 1.3 to 1.9 times faster, with a larger wasm (about 850 KB).

- Deep recursion fails with InternalError: stack overflow instead of
  RangeError; the inline and remote sandboxes allow about 950 calls.
- Deterministic runs: the fingerprint names the engine, and Math.random()
  gives other values for a seed.
- Tests: no tool runs after an interrupted async function, and a loop of
  interrupted async functions ends with a timeout.
2026-10-31 15:45:59 +01:00
Mario Zechner
6eb99ad65b feat(chord): Chord v2 slice 8, registries and host.root
Registries: shared tokens (defineRegistry), one provider per host (host.serve(R, { compose, accepts })),
contributions from facets and host-lifetime objects, plain values or followed Chord watchables, a table
sorted by plugin, lazy compose in the batch that changed the table, failures left out and reported with
the last good value kept; self.registry as a declared dependency, host.registry outside objects, older
readers mapped by accepts[old].value; keyedEntries and orderedEntries. Native sources: self.writable,
registry values and ref.state. self.retryOn wakes failed objects on registry values. host.root gives code
outside objects a call scope sealed at settle, abort and host.close.

Durable: registryReader(value, context); a Session close ends synchronous observers failed
("session closed"), watches still end ended.
2026-10-28 21:34:56 +00:00
Mario Zechner
14b4c56e57 feat(chord,durable): Chord v2 slice 6b, Durable and jobs on the store
StoreStorage (Durable's storage on Chord v2's store, @earendil-works/pi-durable/storage/store) passes
Durable's storage conformance suite and runs Durable's harness tests (npm run test:store). The store
gets owner names and claims, one id allocator, Durable's Storage semantics for refused steps, history
none/latest/rewindable with reads at a point, archive revisions and applyBatch. self.follow follows
Durable's synchronous observers as Chord sources. Persisted jobs: records and state in the owner's
store, restarts as Durable's tasks, abort stored, deferred state. Chord's ./next export returns for
the adapter surface.
2026-10-28 19:18:30 +00:00
Mario Zechner
018fdcf43e fix(durable): pi.live version 3 gives a stored run without turns a count of 0
A database an earlier unreleased build wrote at pi.live version 2 has a run
without run.turns; a run in flight across the upgrade then counted NaN turns
and failed its next tool round with "Value contains a non-finite number and
is not strict JSON". Version 3's migrate adds turns: 0 wherever it is
missing.
2026-10-28 14:28:41 +00:00
Mario Zechner
3ac64a074d docs(durable): examples for turn limits and continue, and admission inside a commit 2026-10-28 14:03:36 +00:00
Mario Zechner
7e15f96019 feat(durable): run-end tasks with the afterRun hook
GenerationHooks.afterRun asks, after a run answered, for durable tasks
(runEndTask(task, input)). The answer commits and the run's inputs settle
done first; the generation then creates the tasks as its children in a
finish phase and waits for them in drain, keeping pi.live.run (run.afterRun)
and the conversation busy, so idle waits include them, Stop aborts them and
a crash resumes them. Queued input is placed when they are terminal.
pi.live.run.first and runFailure.first name where a run's history starts,
also for a continue after a failure.

closes #11141
2026-10-28 14:03:25 +00:00
Mario Zechner
65fba65b2e feat(durable): entry reads for hooks and tasks
HookApi.entry() and entries(), and TaskRuntime.entries(): a hook or task
reads the conversation's raw history, which compaction never changes, so a
range bounded by entry IDs reads the same before and after a compaction and
in a rerun.
2026-10-28 14:03:00 +00:00
Mario Zechner
cc9c523107 feat(durable): admit submissions inside the caller's commit with submit(tx, draft)
Conversation.submit(tx, draft) and ConversationHandle.submit(tx, draft) run
the same admission inside the caller's commit and return the submission ID,
so a host's own checks and writes and the admission are stored together or
not at all. Admission must come before the commit's first table write, once
per commit, and throws ReadAfterWrite otherwise on every path. A first
argument that is not a draft is taken as a transaction; one of another
Session, or a wrapped one, is refused. The handle form also rejects once its
invocation is aborted. The pi.durable.submit span covers the admission only.

closes #11149
2026-10-28 13:09:00 +00:00
Mario Zechner
6a28251136 feat(durable): reject input on idle conversations with whenIdle
An input with whenIdle: "reject" is refused with the new ConversationIdle
when the conversation has no run, before any write, so it never starts a
run. With whenBusy: "steer" it steers the current run or is refused.
Request-ID deduplication runs first.
2026-10-28 13:09:00 +00:00
Mario Zechner
f6a9db1d83 feat(durable): per-conversation turn limit
closes #11043
2026-10-28 11:45:59 +00:00
Mario Zechner
fb524dd2ea feat(durable): continue submissions 2026-10-28 11:45:59 +00:00
Mario Zechner
1c6928b309 feat(durable): calls to tools not offered run as tool tasks 2026-10-28 11:45:59 +00:00
Mario Zechner
87eff524f9 feat(durable): afterTool runs on every tool task result 2026-10-28 11:45:59 +00:00
Mario Zechner
bb65e3e497 fix(durable): import toCodemodeIdentifier from pi-codemode/declarations
The Code Mode extension imported it from /portable, which loads the WASM
QuickJS runtime the extension never uses.

closes #11254
2026-10-27 21:31:47 +00:00
Mario Zechner
5f312dac4c feat(chord,durable): failures are watch calls; one vocabulary for watches, observers and reports
Chord's shared watch: listeners take a tagged call, { kind: "value", value, ops, retired } or
{ kind: "failed", error }; a recoverable failure keeps the watch open and the next value arrives as
[["r", value]]; WatchEnd is tagged by kind, and an aborted context ends a watch "aborted".

Durable: observers take the watch's shapes (advance(call, ctx), end(end)); onReport(report) with
report.kind and report.error, session_failed is storage_failed; subscribeCommits is observeCommits;
SessionEnd and ReportFrom are tagged by kind; FileChange is tagged and file watchers stop with
stop(ctx); CompactionFailure and CompactionStatus name their trigger (pi.live version 2 with a
migrate); the task_outdated report uses the scheduler's reasons; TaskInspection's ready is pending;
DocumentCommitChange is tagged. Stored formats are unchanged.
2026-10-27 20:38:08 +00:00
Mario Zechner
b3a29f1111 fix(durable): end restart loops visibly; a host death charges a generation's attempt
A generation whose request or poll fetch was in flight at a host death counts
one failed attempt of the retry policy (backoff, the partial kept, the same
deferred handle polled again); without retries left the run ends with
runFailure interrupted. restartLimit ends every other task visibly instead of
parking it (tool error result, compactionFailure, runFailure restart_limit,
faulted extension tasks) and reports task_interrupted; retryTask, task_parked
and the restart_limit blocked reason are removed.

closes #11139
2026-10-27 16:57:59 +00:00
Mario Zechner
f3255de765 feat(durable): windowed conversation view, ConversationTranscript and runFailure; remove watchEvents
ConversationView.entries is a window over the fork-aware history with before;
conversation.watch(ctx, { from, min, max }) and observeConversation take the
window. ConversationTranscript merges windows into a client's transcript with
gaps, loadEarlier, fillGap and resume. pi.live.runFailure says why a run ended
without an answer. watchEvents and its event types are removed; the changelog
has the migration guide.

closes #11327
2026-10-27 16:57:24 +00:00
Mario Zechner
8025c414d1 feat(chord): SqliteExecutor.runCounted resolves with the rows a statement changed
The Node driver takes the count from node:sqlite; the Durable Object driver reads
SELECT changes() after the statement, since the cursor's rowsWritten also counts
index writes (verified in workerd). run is unchanged and costs nothing more.
2026-10-27 08:54:53 +00:00
Mario Zechner
8b0caefc22 feat(durable,chord): publish commits at adoption, address-following watches, synchronous observers
Durable publishes a stored commit inside Storage.commit's adopt: the
telemetry events, its own observers (document watches, conversation
views, task graph, scheduler, waits), and a dispatch of host
subscribeCommits listeners one microtask later, before every wait and
abort the commit triggers. A commit the storage rejects after adoption is
published before the Session fails; a throwing telemetry adapter is
reported as host_error "telemetry" instead of rejecting a durable commit.

Chord's watch listeners take one change object, { value, ops, retired }
({ value } for native watches); retirement is a flag, not an end.
Durable's document watches follow their address, and a task document's
watch ends after its retirement call. New synchronous observers for
adapters: observeDoc, observeConversation, observeTaskGraph; the watches
are built on them. The env file watcher's change type is renamed
FileChange.
2026-10-26 20:44:34 +00:00
Armin Ronacher
89c921d8b5 fix: harden handling of untrusted URLs, terminal text, and streamed data
- ai: OAuth logins only accept http(s) URLs from servers (shared
  trustedHttpUrl, now also checked by Radius); pi-messages rejects
  contentIndex values that are not the next or an existing block; the
  CLI auth file is written with mode 0600
- ai: OpenCode DeepSeek V4.1 Flash (renamed from V4 Flash upstream) no
  longer sends prompt_cache_retention
- tui: setTitle() replaces control characters; export
  replaceControlCharacters()
- coding-agent: the login dialog only opens web URLs and never prints
  control characters from sign-in, device-code, or info URLs
- durable: partial commits no longer merge a new __proto__ member into
  Object.prototype
2026-10-26 20:46:31 +01:00
Mario Zechner
d5b3de22d0 fix(durable): record the requested thinking level on assistant messages 2026-10-25 08:05:39 +00:00
Mario Zechner
38ff99177f fix(durable): continue runs after tool handoffs 2026-10-24 22:48:11 +02:00
Mario Zechner
6c2fd73b0c feat(durable): deferred storage of throttled commits
A commit may be marked deferrable ({ persistence: { kind: "deferred", withinMs? } }) on Session, Conversation,
TaskRuntime and a tool's api.commit. When every write changes an existing conversation document with history
"latest" and fork "initial", it is adopted in memory and shown to watches at once, and stored with the next commit
that reads or writes the document, before the document leaves memory, at close, or within withinMs (default
settings.progress.persistWithinMs, 1000 ms). The generation partial throttle and the tool output throttle use it.

subscribeCommits() gets these commits only once stored, with exactly what was stored; internal observers get
memory-only publications. Spec §1/§13 changed, new §4.1. 32 streams over SQLite: storage transactions 562/s -> 99/s,
disk 14.3 -> 4.9 MB/s, CPU 29% -> 22% of one core. A crash loses up to about 1 s of streamed content.
2026-10-24 17:49:30 +00:00
Mario Zechner
d8d99950ac feat(chord,durable): PendingOps folds pending ops for slow consumers; Watch delivers one call per busy period
- chord/delta: PendingOps (push, size, take -> PendingTake): raw pending ops, r drops everything
  before it, a leaf index merges appends and primitive sets per path (cleared by any other op),
  past 10,000 ops only "replace with the current value"; one batch is kept by reference and
  received ops are never mutated
- chord: Watch queues frames in PendingOps while its listener is busy or not started and delivers
  one call with the newest value, the folded ops (or [["r", newest]] after an overflow) and the
  newest frame's context; replaces the 100-frame collapse. NativeWatch is unchanged and takes an
  optional replace()
- durable: watchEvents keeps every event batch in order (NativeWatch with a snapshot replace);
  document, view and task-graph watches fold; spec, README, example 04 and tests updated
- tests: fold fuzzers (tracker, appendText, splices/reorders, root arrays, diffRevisions) with
  codec round trips, adversarial cases, cap, ownership, cost and GC guards
2026-10-24 09:37:12 +00:00
Mario Zechner
1e42501bb3 fix(durable): keep pi.live cached in the partial-commit cost test
The suite's evicting run (PI_DURABLE_DOCUMENT_RETENTION_MS=0) evicts the unwatched pi.live after every
commit, so each partial commit replayed the document's delta chain and the per-commit cost grew with the
stream. Pin the document retention in that test so it measures appends, as the default retention does.
2026-10-24 08:35:20 +00:00
Mario Zechner
da412b0392 feat(chord,durable): appendText for cheap string appends; Durable partial commits append instead of reassigning
Follow-up to #9265.

- chord: Delta appendText(draft, key, text) / StringKeys<T> record an append op without the
  O(length) prefix compare that a plain `draft[key] += text` costs in prepare()
- durable: partial commits append text and thinking deltas with appendText (assignJson plan,
  checked by length), and apply tool-call argument changes from their own StreamingJsonParser
  of the raw toolcall_delta text instead of reassigning the arguments snapshot
- durable "." entry-graph budget 73 -> 76 for the parser it now imports
2026-10-24 08:21:41 +00:00
Mario Zechner
f715a15c10 fix(ai,chord,durable,coding-agent): parse streamed tool-call arguments incrementally (fixes #9265)
- ai: StreamingJsonParser / ToolCallArgumentsStream read each argument delta once (tolerant,
  immutable snapshots, linear cost); every provider, pi-messages and the faux provider use it
  and keep stream state (partialJson, partialArgs, index, customInput, streamIndex, Bedrock
  reasoning bytes) off the content blocks; grammar tool input appends per delta
- agent: streamProxy uses the same parser
- chord: Delta append ops own their text (ownedSlice), so ops and replicas no longer keep the
  whole growing string alive
- durable, ai, mcp: bounded error excerpts and summaries own their characters
- coding-agent: tool calls render streamed arguments once per frame, not once per delta
2026-10-24 08:00:07 +00:00
Mario Zechner
42cb822f1c feat(chord): shared watch-handle implementation (Watch, NativeWatch); Durable uses it 2026-10-23 07:05:23 +00:00
Armin Ronacher
be14d9732f feat(durable): open a Harness read-only
HarnessOptions.readOnly opens a viewer of a session another process drives, such as a synced copy on a device that only shows it. Open leaves running tasks as committed, commits with writes reject with ReadOnlySession before they reach Storage, and resume() and every call that would run tasks reject with it.
2026-10-22 14:25:58 +02:00
Mario Zechner
b9fb2b854b docs(durable): renumber examples 2026-10-22 10:56:50 +00:00
Mario Zechner
92ab536024 feat: Durable storage adopt callback inside the commit, watch types in Chord, explicit watch retirement
Breaking changes:
- Durable Storage.commit(writes, adopt, context): storages call adopt(seq) once,
  synchronously, when the batch is durable and before their next commit
- Chord traceIn(options, fn, context): context last
- WatchHandle and WatchEnd move to Chord (Durable re-exports them); end reasons
  ended (was session_closed) and failed (was session_failed); retirement is an
  explicit source event, and an ended watch drops its references into the Session
- Removed Session.documentState(), Conversation.viewState(), Harness.taskGraph()
  and DocumentState; use watchDoc(), watch(), watchTaskGraph()
2026-10-22 10:52:59 +00:00
Armin Ronacher
cae9fd28a2 Revert "feat(durable): replicate storages as a commit log"
This reverts commit 4c25119f54. The work continues on the replication branch.
2026-10-21 14:39:26 +02:00
Armin Ronacher
4c25119f54 feat(durable): replicate storages as a commit log
Every built-in storage implements ReplayableStorage: head(), replay() at an explicit sequence, and export() of the current state as a log that replays to the same reads. SQLite schema version 3 adds sequence indexes so export streams. @earendil-works/pi-durable/sync adds ReplicatingStorage, which appends each committed batch to an outbox with fork copies resolved, compactRecords(), and versioned JSON log lines.
2026-10-21 14:26:19 +02:00
Mario Zechner
04a93bad8f fix(ai,agent,coding-agent,durable): client identity follow-ups
- ClientAttribution is { name, url, categories? } with required name and url, so
  OpenRouter always gets an HTTP-Referer. identity.ts maps the fields to each
  provider's headers; NVIDIA gets the name with its first letter upper-cased.
  attribution: null turns it off. PI_ATTRIBUTION is pi's, including the
  cli-agent category; agent-core, durable, and the coding agent use it.
- Categories are a string array, joined for OpenRouter.
- withClientIdentity() is a typed wrapper instead of a Proxy and keeps a
  MutableModels mutable. Durable no longer caches wrappers in a WeakMap.
- The Codex originator stays pi; login agentName no longer follows attribution.
- Models and the coding agent's ModelRuntime add x-opencode-session for custom
  providers pointing at OpenCode, replacing the coding agent's transformHeaders.
2026-10-21 11:18:16 +00:00
Armin Ronacher
adc5fa9f80 feat(ai,agent,coding-agent): move client identity and provider attribution into pi-ai
pi-ai gets a ClientIdentity request option and createModels() default that choose the User-Agent (detailed with version, short, or custom) and provider attribution headers for OpenRouter, NVIDIA NIM, and OpenCode. Attribution is off by default in pi-ai and on by default in pi-agent-core and pi-durable (HarnessOptions.identity). pi-coding-agent sets its identity from the telemetry setting instead of adding headers itself.
2026-10-21 11:17:02 +00:00
Mario Zechner
16f5f0af4d feat(durable): IndexedDbStorage, a browser storage backed by IndexedDB (@earendil-works/pi-durable/storage/indexeddb) 2026-10-21 03:02:55 +02:00
Petr Houška
61060ddf8c fix(coding-agent): resolve Windows shells from installation directories
Use installation environment variables for PowerShell discovery and legacy WSL launcher detection.
2026-10-19 21:55:12 +02:00
Mario Zechner
5c1736f82d feat: checkpointWhen is required on Durable document definitions
Breaking change: defineDoc and defineDocFamily now require an explicit checkpointWhen predicate; no default is supplied. Set policies on built-in documents, examples, tests, and triage state, and document the required API.
2026-10-19 18:17:42 +00:00
Mario Zechner
2ec55d808d feat: Durable and pi-ai telemetry; Context.telemetry is always a span
- pi-ai: AI_TELEMETRY_SCHEMA; pi.ai.request and pi.ai.poll spans around Models requests via a lazyStream observer
  (no provider changes), first_chunk and block events (never per delta), usage/cost end attributes, provider_error
  status with a bounded message, aborted stays ok; duration, first-chunk, token and cost metrics; option renamed
  telemetryContext -> telemetry (a parent span)
- pi-telemetry: TelemetryContext carries readonly metrics and propagation capabilities on every span
- chord/context: Context.telemetry is always a TelemetrySpan (NOOP default); withTelemetry takes spans
- durable: DURABLE_TELEMETRY_SCHEMA; pi.durable.submit (inputs start their own trace linked to the caller) and
  pi.durable.phase spans for every task phase; stored carriers on submissions and tasks (TaskOptions.trace, links);
  entry, submission, checkpoint, replay and report events after commit; resumed tasks stay in their trace; phase
  duration, active task and report metrics; failed phases carry fixed categories with bounded messages
2026-10-19 06:26:02 +00:00
Mario Zechner
48909d987a feat: telemetry propagation, span links and metrics; telemetry member in chord/context
- pi-telemetry: TelemetryPropagation (inject/extract, opaque carriers), span links (SpanOptions.links, addLink),
  TelemetryMetrics (counter, up-down counter, histogram), typed metric schemas (createTypedMetrics), in-memory
  measurement log, conformance tests for propagation and metrics
- chord/context: fixed Context.telemetry member (NOOP default), withTelemetry, traceIn
- durable: HarnessOptions.telemetry option type (no instrumentation yet)
2026-10-18 20:36:08 +00:00
Mario Zechner
91b33cab3b feat(durable): ReportInfo says what happened, with typed reasons
onReport info is tagged by type (session_failed, task_faulted, task_parked,
migration_failed, task_outdated, task_error, extension_error, host_error,
listener_error). Task reports carry a ReportTask; restart-limit and outdated
definition reasons are typed fields instead of error.cause strings; host_error
names the failing HarnessOptions callback.
2026-10-18 01:58:07 +00:00
Mario Zechner
5ba1cbaf3d fix(durable): close keeps admitted aborts, reports name the extension that ran, review nits
- Close reads the records it writes back on the line, after admitted commits, so a queued abort mark survives.
- A task's report names the extension that installed the definition it ran, captured at reservation.
- Terminal and completing records drop the restart count; retryTask() ignores them.
- wake() counts a reconcile pass only when it has work, so a run's end sends no extra active wake.
- A failed agent resolution no longer breaks the tool result path that names a tool's extension.
- Spec §10.1 list, changelog sections, test cleanup of killed storages.
2026-10-18 01:00:42 +00:00
Mario Zechner
440a5fcdb8 fix(durable): review fixes for the crash-loop guard, reports, and blocked tasks
- A close writes the tasks it stopped back as pending, so only a crash counts a restart.
- An abort mark starts the count over; an abort handler cut off restartLimit times ends the task faulted.
- Only a changed state resets the count; raised and lowered limits reach the scheduler and the graph.
- wake() stays active while a reconcile pass is scheduled or underway; failed migrations recompute it.
- Reports name the extension whose code failed (hooks, wrappers, sections, tools, a task's installer),
  runtime.report() takes a source, and the parked report carries blocked: restart_limit.
- The task graph keeps only what blocked reasons read, and shows abandoned tasks waiting for a definition.
2026-10-18 00:53:14 +00:00
Mario Zechner
a30c2a81df feat(durable): crash-loop guard, report sources, inbox after failed runs, blocked tasks in the task graph
- Open counts restarts per task; at settings.restartLimit (default 5) a task is parked as blocked
  (restart_limit) until Harness.retryTask().
- onReport(error, info) with ReportInfo: session, task (faulted), extension, listener, host, timer, wake.
  Task faults are reported.
- settings.afterFailedRun: keep, continue, or withdraw the inputs queued behind a failed run.
- TaskGraphNode.blocked follows registry changes and failed migrations.
- README: fallback alarm before Harness.open in the Durable Object example.
2026-10-18 00:22:10 +00:00
Mario Zechner
e17e2e1055 fix(durable): release the second slow call when the first finished, not after 200 polls
The code mode ordering test released call 1 from an eventually() poll with a turn limit; on a loaded CI
runner the sandbox worker took more turns and the test hung until its timeout.
2026-10-17 23:54:43 +00:00
Mario Zechner
ec1be23eb9 fix(durable): review fixes for sleeping tasks and host timers
Due sleeps are picked up at every scheduling pass, so a pass that drops the timer cannot strand them.
Code mode drops a global's result after the script finished instead of reading an ended invocation's clock.
A foreground timer the host declines falls back to setTimeout; the default timer never fires early; Progress
rechecks its clock; an undelivered onWake is offered again. Spec, README, and tests follow.
2026-10-17 23:38:18 +00:00
Mario Zechner
f75b3240ab feat(durable): durable sleeping state replaces runtime.sleep
Tasks end a phase asleep with runtime.sleepUntil() or { status: "sleeping", until, checkpoint }:
no invocation is held, the deadline survives restarts, and the task resumes at its checkpoint once due.
Generation retry/poll and compaction retry sleep this way. Harness.wake() and HarnessOptions.onWake let
a host such as a Durable Object wake the process itself. HarnessOptions.setTimer arms every Harness
timer, and chord Progress paces on an injectable clock. SQLite migrates to schema version 2 to admit
the sleeping status.
2026-10-17 22:29:26 +00:00
Mario Zechner
c268154ca1 feat(chord): move Durable's SQLite drivers into chord/sqlite 2026-10-17 16:48:29 +00:00
Mario Zechner
2331785c33 fix(durable): run the Codex prompt-cache e2e test on gpt-6.1-sol
Codex no longer serves gpt-5.5 to ChatGPT accounts ("The 'gpt-5.5' model is not
supported when using Codex with a ChatGPT account"), so the test failed in its
first turn as unanswered.
2026-10-16 19:42:02 +00:00
Mario Zechner
7ec1f648e9 feat(durable): accept functions for HarnessOptions.models and settings
Like env, models and settings may be a function returning the current value.
Generation, compaction, hooks and tools read it at each use (api.models at each
access), so a host can replace its model configuration without reopening the
Harness. A plain object still works.
2026-10-16 19:30:35 +00:00