p2 rewritten by the Chord v2 authoring rules (decision "App authoring rules confirmed
by the p2 audit"), in four parts merged together:
- core: session lifecycle (relocate, clone, fork, archive) and search rebuild are
persisted jobs on a new server store; Sessions v6 with client-chosen ids and visible
operations; Retry answers with the worker's real outcome; workers push branch,
subagent and status state to the server; roots take the method's ctx.
- services: models, envs, ports, login, mcp, logs, sharing and voice follow external
state (self.follow, followNative) instead of copy loops; methods answer after their
state commits; presence from host.references.
- chat: chat mode, artifacts and prompt resources follow Durable and native state;
shell commands are Chord jobs (shellId is the requestId); status pushed before
send and stop answer; race-free draft claim.
- ui: optimistic updates on web (useOptimistic) and terminal (TuiScope.optimistic,
extended in Chord with a reducer and a pending flag), pending states for long work,
visible rollback, load waterfalls removed; merged with the one composer and
immediate first send (ccff9876e) and the folder + button (5a31a6d5f).
docs/optimistic-ui.md describes the pattern.
The pure swap is gone: a reload restarts every live object of a changed plugin
(documents and tables kept). Calls running at the commit and every call they cause
(the ctx call chain; an old flag on call from trusted links) run on the open old
attempt, which seals once its work drained and its successor's recorded requests are
ready, bounded by reloadDrainTimeout (5 s); then the successor's factory runs. New work
waits at the gate; same-host consumers see the draining object live; a reload cycle
fails at once with unavailable/reload_cycle. Example 64, design section 21 rewritten.
Adds experimental session sync (`@earendil-works/pi-durable/sync`).
Devices upload and pull a session's commits through a remote, reading
them from the Chord store's outbox and applying them with `applyBatch`.
If two devices continue the same session, the one that uploads second
forks to a new session.
I took out the extra sequence indexes from the earlier version of this
branch, since the Chord outbox already persists every commit, so this no
longer touches storage.
---------
Co-authored-by: Armin Ronacher <armin.ronacher@active-4.com>
Before a provider sends or forwards a call's result or error, it sends the origin's
waiting state, past the window if it is full, so a resolved call's changes are
applied on the page. TuiScope.optimistic shows a value while calls are pending; the
guide shows useOptimistic over a Chord reference; example 63.
buildWorkerBase bundles with esbuild code splitting into one private directory: the entry
and its static chunks load at each fork, a lazily imported module and the packages it
imports only when imported. @earendil-works/chord/workers/child exports the worker's side
(workerMain, processLock) without esbuild.
createWorkers bundles the worker entry once (esbuild, Node builtins and installed
packages external) and forks every worker from that snapshot, so an edit of the
app's source while the server runs reaches workers only through a restart. The kit
watches the bundle's inputs as base code. p2 creates its session workers with
SessionWorkers.create and keeps the worker base's inputs as a dev restart baseline.
Commits queued behind a running commit step share the next step's backend
transaction: each keeps its own step, seq, outbox commit and adoption in
queue order, the metadata is written once. A refusal rolls the group back
and its admitted commits run again alone; series, exports, replays and empty
commits end a group; backend failures while closing are latched; any throw
after COMMIT stops the store with memory at the durable group.
An end-to-end reload suite (a kit app, a worker, a real Chromium page; one test per row of design's reload table).
The kit gains timeouts and base (base bundles' inputs and worker entries mark base). Fixes: a call crossing a failed
or a removal gets the cause, not not_ready; a call sent with an old method index is decoded by the names its consumer
had until the new bound is acked; a page retries a failed script fetch; applyChanges runs one at a time.
A hold unbinding with loading (a restart, a parked swap, a live failure retrying) keeps its calls in flight; each
gets one answer from what it waits on (the instance it runs on, or the gate). Calls sent by the old method names
before the consumer learns of the loading wait at the gate too. failed and removal still reject.
The server drops a page's own chord.* attributes and writes chord.host and the principal hook's result itself; tests
for the hook returning nothing, the NOOP adapter and calls over a link.
The four telemetry gaps (rows on commits, chord.host, microsecond times, a read-only store) and: pages record and
send span batches over their link to the server's ingest object (checked, capped, rate-limited, stamped browser and
with the link's principal); workers send theirs too, so the server is the store's only writer; principalAttributes
lets an app choose what a span records about its principal. The tutorial shows a real page's click traced to the
worker, with the trace viewer.
Chord's telemetry schema; spans for calls, methods, relays, holds, activations, natives, loads, reloads, resolves,
revocations, routines, jobs and HTTP requests; trace carriers on hold and call (bounded, printable ASCII) continued by
relays and workers; events and host-wide metrics; the app kit passes an adapter to its hosts. renderTrace escapes
control characters. Example 62 prints a trace across three hosts.
A call arriving while an attempt activates waits for its outcome (bounded by readyTimeout) instead of failing
not_ready while its client sees live. Fates start in the reverse of the order objects went live, so a pure
dependent of a swapping provider swaps instead of parking.
defineMessages with typed t, t.message/t.render for messages as data, the Translations registry with fallbacks and
conflicts, a locale preference per principal with the device as default and a live switch, useMessages and
scope.messages, formatTime/formatNumber. The tutorial gains a chapter on languages.
self.query(table, q, limit) runs before the object goes live and again in the commit step of every commit that
touched the table, publishing only the diff; a reload swaps the object. The tutorial and 59-chat use it.
The heartbeat no longer counts time the process did not run (a laptop's sleep, a blocked event loop) as silence: after such a pause each open link pings at once and closes only if its peer stays silent for close - ping.
A running worker's lost link is now a stop instead of a failure: worker_link_lost is reported and the held route starts a new worker. A second lost link within a minute of the last recovery still fails the address.
A worker killed by a signal left its lock's socket file, and every later start of its address failed closed on it, retry too. The workers component now removes a lock socket file nobody accepts on before each start and after a child's exit by a signal, in the address's queue. processLock also treats ENOTSOCK as stale instead of retrying until its context ended.
- get, hold and registry on a host that has not loaded throw not_ready at once.
- A registry with contributions and no provider is reported (registry_undeclared) at publication.
- A request waiting unbound is reported once after 5 s (unbound_long; one clock per host), and keeps waiting.
- An invalid plugin name makes load and reload throw before any change.
- A transaction cancelled by the end of its context is reported (transaction_cancelled) besides rejecting.
- Docs: who a reference acts as, no per-link call limit, await what you start in a root or a handler.
src/next is src (the main export), src/next-node is src/node with short subpaths (/http, /workers, /plugins,
/plugins/runtime, /app, /surfaces, /web, /react, /tui, /tui/launcher); the foundations get /json and /watch;
Chord v1 moves to src/old (/old, /old/node) until p2 runs on v2; tests and docs move likewise (test/, docs/,
test/old, docs/old). Durable and env import the foundations from their subpaths.
- Web: base bundles built from the app's entry at start (content-hashed, served
forever), plugin scripts that register themselves and one combined file,
WebScripts/TuiScripts showing the current set, startPage with the connection
loop, in-place plugin reloads and a reload on a new base bundle.
- React bindings: HostProvider, useHold, useWatch, useRegistry, useRefState.
- Terminal: startTerminal, a launcher that signs in with a token, a token file or
device pairing and always runs the app's terminal code in a sandboxed child
process (pipes, empty environment, --permission granting nothing, no SQLite,
a builtin allowlist and a frozen process inside); output filter, confirmed
copy/open/attach actions; startTerminalRuntime and TuiScope inside.
- Themes: the Themes registry, ThemeChoice per principal, CSS variables on the
page, terminal colours by role; CSS modules in plugin builds.
- App kit: startServer gains web and tui; startApp renamed startServer,
exportAddress/importAddress renamed exportData/importData and export no longer
stops the worker; ephemeral sessions end with their client's last link.
- Examples 56-web, 57-tui, 58-themes, 59-chat.
chord/next/app: startApp and startWorker with the fixed app layout, worker kinds,
AppInfo and WorkerInfo; sessions (hashed tokens, rolling cookies, bearer tokens),
the one-time owner link, the owner token file, device pairing; Files per host
with ordered chunked uploads; export and import per address; plugin roots,
installs and applyChanges; optional users, invite and ephemeral login facets.
HTTP: deny by default (public routes and sockets), remoteAddress,
readSignedCookie. Examples 53-55.
Plugins import the app's versioned API (createPlugins api option: unlisted
versions, other kinds' modules, mixed versions and direct Chord imports refused),
each other by relative path within a root or by package name across
repositories (npm or git pins), with per-kind files checked by name.
chord.plugins lists a package's plugins. Required roots, single-file plugins,
installPlugin from git or npm, testPlugin with stub facets.
plugin-authoring.md for authors and agents. Examples 49-52.
A relayed hold released (or its principal refused or revoked) while the route resolved made #resolve throw before adding the connection, so the spawned host's link was never read and its own upstream references failed unavailable/ready_timeout after readyTimeout. The connection is now added right after resolve; the checks only fail the relay.
Schema helpers are a thin layer over pi-schema's builders with the same JSON,
so fingerprints and stored data are unchanged (proven by schema-switch tests and
a store written by the TypeBox code). Checks through pi-schema; error messages
are pi-schema's; format is an annotation. v1 contracts take any JSON Schema.
map(bytes()) works.
chord/next/http: a Web fetch handler built from the registry Http (routes, wraps,
sockets, hosts, relays, CSP sources) and Authenticators, responses as descriptions,
a security baseline on by default, Chord's own Node listener as a host-lifetime
object, nodeRequest for apps mounting it, links and plugin sockets.
chord/next/workers: a forked worker per address, started on demand, stopped when
idle, never restarted by itself, plugins built by the server and shipped.
Plugins: kinds with platform and host packages, createPluginRuntime, scripts(kind),
rollout results per host, imports of other plugins by name. Kernel: RouteHandle
(retry, suspend), no route backoff, anonymous host.root, origins only from relays.
ws is an optional peer dependency.
Plugin discovery, esbuild builds (optional peer dependency), vm evaluation with
pluginFile and mapStack, dependency installs, a watcher with dirty marks and
rollout across hosts; host.load and host.reload per plugin with the closure rule;
examples 35-39 as folders with real plugin directories.
The owner's audit decisions (decisions.md, "Simplification audit"): cut the reload-only
installing state (staging is synchronous) and the static registry cycle edges (the
runtime guard stays); retry propagation by a visited set and a hop budget (retry.hops)
instead of operation ids and through lists; a pure swap sends holds a fresh bound; one
rule after a live failure (every member followed until the admitted commits published);
parked_long for any long park, naming the first blocker in declaration order with its
state; one seal token per attempt. Keys follow the old instance during a restart's drain
like members. src/next: about 255 code lines fewer.
`sanitizeOutput()` in `@earendil-works/chord/output` deleted the ESC
byte (0x1b) along with the other control characters, but it left the
rest of each escape sequence in place. Colored tool output such as
`\x1b[31mERROR\x1b[0m` came back from `OutputBuffer.snapshot()` as
`[31mERROR[0m`. Because the ESC byte was already gone, later ANSI
stripping could no longer recognize the leftover fragments.
`sanitizeOutput()` now removes whole 7-bit escape sequences before it
removes single control characters. It handles:
- CSI sequences (colors, cursor and erase commands)
- string sequences (OSC, DCS, APC, PM, SOS) terminated by BEL or `ESC \`
- short `ESC x` forms such as `ESC ( B` or `ESC 7`
`snapshot()` sanitizes the concatenated retained text, so a sequence
split across pushed chunks is also removed.
**Testing**
- Added a regression test in `packages/chord/test/output.test.ts`. It
covers SGR colors, cursor and erase CSI, charset and short escapes, OSC
8 hyperlinks and an OSC title, plus an `OutputBuffer` fed a color
sequence split across three chunks. The test fails without the change
and passes with it.
- `test/output.test.ts` and `test/output-skip.test.ts` pass.
- Biome is clean. The only `tsc` errors in `npm run check` are in
`packages/ai` tests and are unrelated to this change.
- The durable package test results are the same with and without the
change: 71 test files fail on both, and every individual test that runs
passes.
- Reproduction from the issue: before the change both lines printed
`"[31mERROR[0m\n"`; after it, both print `"ERROR\n"`.
Fixes#11604
Requested by @badlogic
Co-authored-by: triagent <triagent@localhost>
A Session no longer holds its line while a commit is stored. Commits that
arrive while one storage call is in flight go out together in the next call,
each with its own seq and outcome; a job runs until it touches a key an
unpublished commit wrote, then waits for that commit's publication. Outcomes
are delivered in line order; publication is in line order, one commit per
macrotask. The scheduler's decisions run as regions that record the keys they
read and rerun after a wait, with side effects only after a clean check.
Storage.commit(batches, receive, context) replaces commit(writes, adopt,
context): prefix semantics (Durable's own checks refuse a batch and keep the
ones before it; any other failure rejects the call), receipts per stored
batch, optional publishing() scope. Memory, SQLite, JSONL, IndexedDB and
StoreStorage implement it natively; StoreStorage through the new
ChordStore.commitSeries (one store transaction per call), replacing
commitInBatch.
A normal run makes no storage reads: bounded record cache, conversation
tails, task-document index, kept context ranges. A tool slot's running status
and a generation's attempt are stored with the next commit (12 instead of 14
commits per run).
One Harness with 128 conversations over storage with a 4 ms round trip: about
5 -> 440 agent runs per second. Design: packages/durable/docs/group-commit.md.
host.reload(facets, ctx): staging into a staged table with a recording self (all or
nothing), one commit (types, registries, contributions; per-name wakes; installing
holds), and each object's fate: pure swap in place, native restart after its drain,
retry on the new code, or its end by its type's removal, with a report. Old calls
finish on old handles (R15 for documents and tables), routines hand over per object,
jobs keep their versions (accepts, blocked { version }, persisted runs per R3),
registries swap providers and contributions in one batch, holds are kept,
re-subscribed or unbound with failed { waiting } (model §5), accepts maps older
consumers' state, arguments, results and keyed members.
References are the only indirection (owner, 2026-10-29): a consumer's watches and
derives attach through the reference, which passes them straight to the object
instance's own document or derive, moves them to a new instance (one frame) and keeps
its last values while unbound; MemberSource, keyed-members and the per-version
hand-over are removed. Older consumers read through the instance's shared mapping
per version. Examples 33-34, reload benchmark.
A reference is a request that outlives its provider: a missing provider or a version that does not fit leaves
it unbound, waiting for the name (failed { waiting }, a new FailureCause kind), local and remote alike. An
attempt parks until its dependencies are live, with no timer and no backoff, woken on every state change of a
dependency; its readyTimeout runs only for its own steps. Requests wait only for the host's publication (the
startup deadlock is gone); load resolves at the first outcomes of its host-lifetime objects. Getters wait through
waiting until their readyTimeout. Registry readers are waiting dependencies, woken at every usable value; a last
good value is dropped when a contribution it was built from leaves. Unbound holds count for keep-alive and are
revoked; retry passes on to failed blockers once per operation (operation id and visited list on the wire); the
dependency deadline is gone and parked_long reports a long wait on a remote attempt.
The contract field placement is hostAddress, required on every remote() (type and declaration error); the
fallback for remote types without one is gone. invalid/placement is invalid/host_address. host/placement.ts
(Placement) is host/destination.ts (Destination); here is decided by address alone, and a host without an
address refuses to serve a remote type. A hold always carries its address on the wire (the binary codec's
ADDRESS flag is gone, the text codec requires it). Tests and examples declare their host addresses.
Registries: shared tokens (defineRegistry), one provider per host (host.serve(R, { compose, accepts })),
contributions from facets and host-lifetime objects, plain values or followed Chord watchables, a table
sorted by plugin, lazy compose in the batch that changed the table, failures left out and reported with
the last good value kept; self.registry as a declared dependency, host.registry outside objects, older
readers mapped by accepts[old].value; keyedEntries and orderedEntries. Native sources: self.writable,
registry values and ref.state. self.retryOn wakes failed objects on registry values. host.root gives code
outside objects a call scope sealed at settle, abort and host.close.
Durable: registryReader(value, context); a Session close ends synchronous observers failed
("session closed"), watches still end ended.
A failed activation releases its references after its natives closed; a dependency going live while
the dependent activates retries it at once; a call on a local object runs with the host's principal
and incarnation; retiring an unloaded document checks its stored version and fingerprint, and a cut
retirement no longer fails its successor; ref.retry() re-runs a refused hold's start on the same id
(the provider keeps refused holds), and ends released/closed on a closed host; a relayed hold waits
while the parent link is down; a link drop while a dependency is still loading is no failure.
StoreStorage (Durable's storage on Chord v2's store, @earendil-works/pi-durable/storage/store) passes
Durable's storage conformance suite and runs Durable's harness tests (npm run test:store). The store
gets owner names and claims, one id allocator, Durable's Storage semantics for refused steps, history
none/latest/rewindable with reads at a point, archive revisions and applyBatch. self.follow follows
Durable's synchronous observers as Chord sources. Persisted jobs: records and state in the owner's
store, restarts as Durable's tasks, abort stored, deferred state. Chord's ./next export returns for
the adapter surface.
The next object runtime moves from the private repository into pi: src/next (contracts, hosts and
objects, documents, derives, links and relays, routines and jobs, the store) with its tests and
examples in test/next, src/next-node (Node compression), and its design in docs/next (design.md, the
decision log decisions.md, the slice plan slices.md, slice specs, books and benchmarks). AGENTS.md
gets its code style. Not exported: the API changes with every slice until it has an entry point.
Chord and Durable change a document one transaction at a time, so a tracker registered a WeakRef per
change for competitors that never existed on their hot path. The WeakRef targets survived scavenges
and V8 could pretenure every overlay context into old space (47.5 MB promoted against 2 MB in a
tracker loop). The tracker now keeps its open change in a plain field and moves it into the weak
registry only when a second change opens; stale semantics are unchanged. A draft dropped without
prepare or abort stays referenced until the tracker's next change or adopt (at most one per tracker).
`stored` and `rawNodes` are created on first use.
Chord's shared watch: listeners take a tagged call, { kind: "value", value, ops, retired } or
{ kind: "failed", error }; a recoverable failure keeps the watch open and the next value arrives as
[["r", value]]; WatchEnd is tagged by kind, and an aborted context ends a watch "aborted".
Durable: observers take the watch's shapes (advance(call, ctx), end(end)); onReport(report) with
report.kind and report.error, session_failed is storage_failed; subscribeCommits is observeCommits;
SessionEnd and ReportFrom are tagged by kind; FileChange is tagged and file watchers stop with
stop(ctx); CompactionFailure and CompactionStatus name their trigger (pi.live version 2 with a
migrate); the task_outdated report uses the scheduler's reasons; TaskInspection's ready is pending;
DocumentCommitChange is tagged. Stored formats are unchanged.
