diff --git a/AGENTS.md b/AGENTS.md index 04c9c32be..43a9cf8c9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,6 +27,20 @@ - Never hardcode key checks (e.g. `matchesKey(keyData, "ctrl+x")`). Add defaults to `DEFAULT_EDITOR_KEYBINDINGS` or `DEFAULT_APP_KEYBINDINGS` so they stay configurable. - Never modify `packages/ai/src/models.generated.ts` directly; update the catalog definitions in `packages/ai/catalog/` instead (see its README), then regenerate. Including the resulting `models.generated.ts` diff is always OK, even if regeneration includes unrelated upstream model metadata changes. The generated model data in `packages/ai/src/providers/data/` is gitignored: never stage it, and don't look for it in git history. +## Chord v2 Code Style + +Binding for `packages/chord/src/next/` and its tests, when writing and when reviewing. The model is pi-durable's source. + +- Files: one concept per file, named after it; no one-helper files, no grab-bag files. First line: a comment saying what the file is for. Order: imports, exported types, then the public entry point first and its helpers below it in the order they are used; constants right above their first user. A blank line between top-level declarations. Split files by concept, never by size: one concept may take a long file (up to about 1,000 lines is fine); a second concept in a file is the reason to split. +- Functions: every export has a one-line doc comment saying what it does or guarantees. Inline comments explain a non-obvious why (cost, race, decided rule), never what the next line does. Short functions with one job (about 40 lines needs a reason; one `switch` over a tagged union is fine). No boolean flag parameters that switch behaviour: two functions or a tagged value. Never write the same lines twice. Context last: required args, then `context`/`ctx` (always required, as in Durable; optional only on error-reporting paths), then optional args. +- Types and checks: tagged unions for every state and result. Casts only where TypeScript cannot express a decided invariant, each with a short why. Every remaining runtime check names the trust boundary it serves (wire input, schema-checked params and arguments, declaration policy the types cannot express). Same-host values are plain data, immutable by convention: never copied, frozen or walked to enforce it; freeze only Chord-owned metadata when a decision says so. No defensive code against broken adapters, callers or our own modules. +- Cost and memory: no hot-path work a decision did not ask for (walks, copies, compilations, per-call allocations). The function doing counted work counts it through its context; call sites never count for others. Caches are `WeakMap`s keyed by their owner, each with a GC test; no module-level strong caches. +- Names and API: the design's vocabulary (`params`, `binding`, `principal key`), no invented synonyms; decided `ChordError` codes and reasons. Export only what a later layer or app needs. No dev or test modes, no unrequested options. +- Tests: one test file per source file, one `describe` per mechanism, the layer spec's test list case by case; cost guards through the in-memory telemetry adapter; GC tests for every cache and binding. Every slice also adds one or more runnable examples in `packages/chord/test/next/examples/NN-topic.ts`, one topic each, numbered across slices, written as references for Chord's users (Durable's `test/examples/` format: header with what it shows and how to run it, commented steps, printed results). +- Slices: Chord v2 is built in vertical slices (`packages/chord/docs/next/slices.md`). Implementers and reviewers read the slice spec, the feature map in `slices.md` and the whole of `packages/chord/docs/next/design.md` in full, and check the work against all three; slice specs cite design sections for orientation, not as a reading limit. +- Deferred work: anything left to a later slice goes into the section "Carry-over" of `packages/chord/docs/next/slices.md` under that slice, in the same change; a slice's spec and build include every entry of its section. +- Before reporting: re-read every changed file top to bottom as a stranger would; each file must be understandable without jumping around. Then `npm run check` and the layer's tests. + ## Commands - After code changes (not docs): `npm run check` (full output, no tail). Fix all errors, warnings, and infos before committing. Does not run tests. diff --git a/packages/chord/CHANGELOG.md b/packages/chord/CHANGELOG.md index cf4082fa9..5724e1a42 100644 --- a/packages/chord/CHANGELOG.md +++ b/packages/chord/CHANGELOG.md @@ -13,6 +13,7 @@ ### Added +- Chord v2, experimental and not exported yet: the next object runtime in `src/next` (contracts, hosts and objects, documents, derives, links and relays, routines and jobs, the store), with its tests and runnable examples in `test/next` and its design in `docs/next` (`design.md`, the decision log `decisions.md`, the slice plan `slices.md`). Its API changes with every slice until it gets an entry point. - `NativeWatchable`: an in-process value that changes and can be watched (`value`, `watch(context)` returning a `NativeWatchHandle`), the in-process counterpart of `Watchable`. - `WatchHandle` and `WatchEnd`, the one watch API, moved from Durable: a serialized observation with `value`, `start(listener)`, `stop()`, and `closed`; the listener gets a `WatchChange` (`{ value, ops, retired }`) and the context. `WatchEnd` reasons are `stopped`, `cancelled`, `ended`, `listener_error`, and `failed`. - `Watch`, the one watch handle implementation, moved from Durable: a source creates one per watch and feeds it with `advance(change, context)` (`retired: true` for a retirement), `end()` or `end({ error })`, `endAfterDelivery()`, and `observeCancellation(signal)`. Frames that arrive while the listener runs (or before `start`) are delivered in one call with the newest value, their ops folded by `PendingOps` and the newest frame's context; past 10,000 pending ops after folding, the call gets one root replacement `[["r", newest]]` instead (replaces the collapse after 100 undelivered frames); termination drops the listener and every reference into the source. diff --git a/packages/chord/docs/next/benchmarks.md b/packages/chord/docs/next/benchmarks.md new file mode 100644 index 000000000..4add7d22b --- /dev/null +++ b/packages/chord/docs/next/benchmarks.md @@ -0,0 +1,492 @@ +# Chord v2 benchmarks + +Light benchmarks of hot paths, run by hand (not in CI), recorded per slice for later slices to compare against +(owner, 2026-10-22). Each section names its script, the machine and what the numbers include. + +## Slice 3a: remote objects over one link + +Script: `test/next/bench/links.ts`, run from `packages/chord` with +`node --conditions=source --experimental-strip-types --expose-gc test/next/bench/links.ts` (about 15 s). Node 24.4.1, +Intel Core i9-13900, one process: server and client host on one memory link (`queueMicrotask`), every case once per +codec, side by side. + +Payloads, modelled on p2: entries `{ id, text, author, at }` in one document of a local store, exposed by a remote +view; "1,000 items" are entries with 80-character texts (the notes list), "500 KB" are 500 entries of 960 characters +(a conversation document). Commits are server-side method calls (one transaction each) on the store. + +What the numbers include: + +- *first hold to live*: the first `client.get`, which also activates the view on the server; one sample, so it moves + by a few hundred µs between runs (JIT and collector state). +- *hold to live (mean of 100)*: further `client.get`s of the live view, each released: the `hold`, the snapshot, the + `bound` message encoded, decoded and checked, the replicas built (two link hops). +- *wire bytes*: one whole message as its codec writes it (memory links add nothing). +- *encode / decode*: that message through the codec alone (`decode` without the schema check), averaged over 10,000 + runs (1,000 for snapshots and the 1 MiB call), the fastest of five such loops (one loop of a microsecond-sized + message is at the mercy of the collector and the JIT). +- *heap per encode + decode*: heap bytes allocated per message while encoding and decoding it that many times, + measured with `GCProfiler` (the growth plus what the collections freed); `ArrayBuffer` backing stores (the message + bytes themselves) are outside the heap. +- *array buffer kept per encoded message*: the `process.memoryUsage().arrayBuffers` growth while 50 encoded copies of + the message are kept, per message: the backing store a queued or held message keeps. Freed buffers are not reported + reliably inside a loop, so allocation of temporary buffers is not measured. V8 keeps very small typed arrays on the + heap, so the smallest binary messages show 0 B here. +- *commit to frame*: from the server's method call to the client's watch listener, averaged over 1,000 commits: the + call and its transaction, the encodes, the hop, decode, check, apply, and the watch's delivery. +- *heap / array buffers retained per hold*: the heap and `arrayBuffers` growth of holding the 1,000-item view, after full collections, divided by the + holds; both ends (the provider's member encoders and subscriptions, the consumer's member decoders and replicas, + each its own decoded copy of the snapshot), median of seven trials. + +| Case | binary | text | +| ------------------------------------------------------------------------------- | --- | --- | +| notes list of 1,000 items: first hold to live (activation, one sample) | 0.958 ms | 1.041 ms | +| notes list of 1,000 items: hold to live (two hops, mean of 100) | 0.666 ms | 0.686 ms | +| notes list of 1,000 items: snapshot: wire bytes | 133.7 KiB | 133.7 KiB | +| notes list of 1,000 items: snapshot: encode / decode | 291.1 µs / 266.4 µs | 312.3 µs / 302.8 µs | +| notes list of 1,000 items: snapshot: heap per encode + decode | 441.5 KiB | 441.8 KiB | +| notes list of 1,000 items: snapshot: array buffer kept per encoded message | 137.7 KiB | 133.7 KiB | +| 500 KB document: first hold to live (activation, one sample) | 1.461 ms | 1.555 ms | +| 500 KB document: hold to live (two hops, mean of 100) | 1.138 ms | 1.309 ms | +| 500 KB document: snapshot: wire bytes | 496.5 KiB | 496.5 KiB | +| 500 KB document: snapshot: encode / decode | 615.4 µs / 390.8 µs | 687.5 µs / 528.3 µs | +| 500 KB document: snapshot: heap per encode + decode | 1511.2 KiB | 1511.2 KiB | +| 500 KB document: snapshot: array buffer kept per encoded message | 500.5 KiB | 496.5 KiB | +| one appended item: commit to frame | 0.037 ms | 0.031 ms | +| one appended item: messages per commit, encodes per hold | 1, 1 | 1, 1 | +| one appended item: state message: wire bytes | 165 B | 175 B | +| one appended item: state message: encode / decode | 1.4 µs / 1.4 µs | 0.9 µs / 1.1 µs | +| one appended item: state message: heap per encode + decode | 2607 B | 3034 B | +| one appended item: state message: array buffer kept per encoded message | 165 B | 175 B | +| one changed field: commit to frame | 0.036 ms | 0.031 ms | +| one changed field: messages per commit, encodes per hold | 1, 1 | 1, 1 | +| one changed field: state message: wire bytes | 27 B | 38 B | +| one changed field: state message: encode / decode | 0.6 µs / 1.0 µs | 0.5 µs / 0.7 µs | +| one changed field: state message: heap per encode + decode | 2005 B | 2527 B | +| one changed field: state message: array buffer kept per encoded message | 0 B | 38 B | +| one removed item: commit to frame | 0.031 ms | 0.029 ms | +| one removed item: messages per commit, encodes per hold | 1, 1 | 1, 1 | +| one removed item: state message: wire bytes | 23 B | 34 B | +| one removed item: state message: encode / decode | 0.6 µs / 1.0 µs | 0.5 µs / 0.7 µs | +| one removed item: state message: heap per encode + decode | 1704 B | 2575 B | +| one removed item: state message: array buffer kept per encoded message | 0 B | 34 B | +| streamed reply, 2,000 appends: time per append | 0.031 ms | 0.027 ms | +| streamed reply: bytes per append, total | 23 B, 45.0 KiB | 34 B, 66.5 KiB | +| streamed token: wire bytes | 23 B | 34 B | +| streamed token: encode / decode | 0.5 µs / 1.0 µs | 0.5 µs / 0.6 µs | +| streamed token: heap per encode + decode | 1696 B | 2527 B | +| streamed token: array buffer kept per encoded message | 0 B | 34 B | +| 1 hold(s) of 1,000 items: heap / array buffers retained per hold (both ends) | 154.1 KiB / 47 B | 159.7 KiB / 21 B | +| 1 hold(s): messages per commit | 1 | 1 | +| 10 hold(s) of 1,000 items: heap / array buffers retained per hold (both ends) | 175.2 KiB / 41 B | 175.2 KiB / 44 B | +| 10 hold(s): messages per commit | 1 | 1 | +| call, small argument: round trip | 0.016 ms | 0.015 ms | +| call, small argument: call message: wire bytes | 16 B | 23 B | +| call, small argument: call message: encode / decode | 0.6 µs / 0.4 µs | 0.5 µs / 0.6 µs | +| call, small argument: call message: heap per encode + decode | 1580 B | 1603 B | +| call, small argument: call message: array buffer kept per encoded message | 0 B | 23 B | +| call, small argument: result message: wire bytes | 9 B | 14 B | +| call, small argument: result message: encode / decode | 0.4 µs / 0.3 µs | 0.5 µs / 0.5 µs | +| call, small argument: result message: heap per encode + decode | 1219 B | 1509 B | +| call, small argument: result message: array buffer kept per encoded message | 0 B | 14 B | +| call, 1 MiB bytes argument: round trip | 0.192 ms | 0.159 ms | +| call, 1 MiB bytes argument: call message: wire bytes | 1024.0 KiB | 1024.0 KiB | +| call, 1 MiB bytes argument: call message: encode / decode | 113.7 µs / 1.4 µs | 123.5 µs / 1.5 µs | +| call, 1 MiB bytes argument: call message: heap per encode + decode | 3042 B | 3383 B | +| call, 1 MiB bytes argument: call message: array buffer kept per encoded message | 1028.0 KiB | 1024.0 KiB | + +Observations: + +- One message per commit and one encode per hold in every case (W1). +- The binary codec makes small messages 30-45 % smaller (a streamed token 23 B against 34 B; a changed field 27 B + against 38 B) and allocates less heap per message; both are about 1 µs per small message each way. Binary decodes + small messages a little slower than text: it makes one call per field (varints, strings, one `JSON.parse` per value) + where text makes one `JSON.parse` of the whole tuple; the difference is under 1 µs per message and not worth + optimizing. Snapshots are JSON in both and about the same size; binary decodes them faster, text encodes them a + little faster (binary encodes the JSON text, then copies it into the message once). +- A binary message keeps a right-sized backing store (a 500 KB snapshot keeps about 500 KiB, not the 1.5 MB a buffer + reserved for the worst-case UTF-8 length kept before the review). +- Kept backing stores of small binary messages (owner, 2026-10-22): a message is written into a 256-byte starting + buffer, whose exact size is not known before the message is written, so in-place encoding does not right-size it. + `finish` copies to an exact-size buffer when the spare room is at least 64 bytes and more than an eighth of the + message. The appended-item state message (165 B) now keeps 165 B instead of 256 B, for about 0.2 µs more per encode + (1.1 against 1.3-1.4 µs); messages up to about 100 B were copied before already (they show 0 B: V8 keeps such small + arrays on the heap); large ones end within their 4 KiB of room and are not copied (the 1 MiB call keeps 1028 KiB). +- Large snapshots (owner's profiling request, 2026-10-22): the earlier "hold to live" gap (2.67 ms binary, 1.67 ms + text) was one sample each; averaged over 200 holds both codecs took about 1.2 ms. The remaining real gap was the + encode (886 against 718 µs): binary encoded the JSON text into its own buffer, then copied it into the message. + Long strings (from 4,096 UTF-16 units) are now encoded straight into the message buffer behind a fixed five-byte + length (a padded varint), and short ones (up to 42 units, a one-byte length) too: binary now encodes a 500 KB + snapshot in 650 µs against text's 690 µs and holds it in 1.14 ms against 1.30 ms. +- A 1 MiB `bytes()` argument travels as 1 MiB plus a few bytes in both codecs (not base64's 1.33 MiB); its decode is a + view (under 2 µs), its encode the one copy into the message. +- Encoding is not the bottleneck at these sizes; no CBOR. +- Each hold's replica is its own decoded copy (about 160-175 KiB for 1,000 entries); two references from one host are + two holds by design. + +## Slice 3b: surviving drops + +Script and machine as for slice 3a (`test/next/bench/links.ts` now runs the 3b cases after the 3a ones, about 37 s). + +New cases, both on the 1,000-item notes list view (133.7 KiB snapshot): + +- *reconnect, N reference(s)*: N references held on the client (N holds of one provider object); the server's end + of the link closes, the client sees the drop, then a new link is accepted and connected. *Time to all live*: from + `connect` to every reference's `ref.state` being `live` again (the re-holds, N snapshots encoded, decoded and + adopted as one frame `[["r", value]]` per member); the mean of 20 reconnects for one reference, 5 for 100. *Bytes*: + what the new link carried both ways until then (the handshake included). +- *1,000 calls parked while down*: a small call (`ping`) made 1,000 times right after `connect`, before the re-hold is + bound, so every call parks; time from `connect` to the last result (the re-hold, then 1,000 calls sent in order and + answered), mean of 10; the order is checked at the provider and in the results. + +| Case | binary | text | +| --- | --- | --- | +| reconnect, 1 reference of 1,000 items: time to all live (mean of 20) | 0.870 ms | 0.791 ms | +| reconnect, 1 reference of 1,000 items: bytes down / up | 133.7 KiB / 63 B | 133.8 KiB / 110 B | +| reconnect, 100 references of 1,000 items: time to all live (mean of 5) | 63.1 ms | 66.4 ms | +| reconnect, 100 references of 1,000 items: bytes down / up | 13370.5 KiB / 4718 B | 13371.1 KiB / 6736 B | +| 1,000 calls parked while down, flushed on reconnect: time to the last result | 14.9 ms | 13.8 ms | +| 1,000 parked calls: run in the order they were made | yes | yes | + +Regression rows (the live paths of 3a, same run): + +| Case | binary | text | +| --- | --- | --- | +| notes list of 1,000 items: hold to live (two hops, mean of 100) | 0.750 ms | 0.682 ms | +| 500 KB document: hold to live (two hops, mean of 100) | 1.134 ms | 1.308 ms | +| one appended item: commit to frame | 0.037 ms | 0.031 ms | +| one changed field: commit to frame | 0.036 ms | 0.029 ms | +| streamed reply, 2,000 appends: time per append | 0.031 ms | 0.027 ms | +| 1 hold of 1,000 items: heap / array buffers retained per hold (both ends) | 154.6 KiB / 47 B | 160.1 KiB / 21 B | +| 10 holds of 1,000 items: heap / array buffers retained per hold (both ends) | 175.1 KiB / 41 B | 175.4 KiB / 44 B | +| call, small argument: round trip | 0.016 ms | 0.014 ms | +| call, 1 MiB bytes argument: round trip | 0.192 ms | 0.191 ms | +| messages per commit, encodes per hold (every commit case) | 1, 1 | 1, 1 | + +Observations: + +- The live paths are unchanged within noise (3a: 0.67 / 0.69 ms per hold, 0.037 / 0.031 ms per appended item, 154.1 / + 159.7 KiB per hold); message sizes and encodes per commit are identical. A live reference's call checks its state + once and makes no queue; `ref.state` is not made until read. +- A reconnect costs what the holds cost the first time: about 0.63 ms and one 133.7 KiB snapshot per reference of + this size (100 references: 63 ms, 13 MiB). The consumer sends only the `hold`s (47 B each binary). +- 1,000 parked calls take about 14 ms to flush and answer, a little less than 1,000 sequential round trips (16 ms): + they leave in one turn once the re-hold is bound. Each parked call keeps its arguments and one timer until sent. + +## Slice 4a: links and relays + +Script: `test/next/bench/relay.ts`, run from `packages/chord` with +`node --conditions=source --experimental-strip-types --expose-gc --no-flush-bytecode test/next/bench/relay.ts` (about +5 s; the forked workers are the same file run with the argument `worker`). Machine as for slice 3a. Binary codec +throughout. The topology is the one of slice 4a's example: a tab (a client host), a server with an address and a +route for `Session`, and per session a worker host whose `Store` (a local document) backs a remote `View` placed on +the session; the tab holds the view through the server, which relays (no part parsed). + +What the numbers include: + +- *call to frame*: the tab calls `stream` (one commit appending `" tok"` on the worker) and waits for both the result + and its watch frame; mean of 1,000 after 100 warm-up calls. Through a relay that is four hops for the call and four + for the frame and result (tab → server → worker → server → tab); the baseline is a tab directly on the worker. + Slice 3a's *commit to frame* started at the provider, so these are not comparable with it. +- *bytes per commit*: bytes the server sent to the tab, and received from the worker, per commit (the `state` + message and the call's result; acks and the call go the other way). From the worker the messages carry the tab + link's origin (two bytes each). +- *first hold, `resolve` included*: a `tab.get` of a new session: the route resolves (an in-process worker host is + created, connects, loads its facet with 1,000 entries and activates the view), the relayed hold, the 133.7 KiB + snapshot through two links. The IPC row's first hold includes forking a Node process with strip-types (one sample). +- *the throttled tab*: the tab's end of its link delivers at 64 KiB/s (whole messages, in order; idle time banks no + budget) in **virtual time**, 10 ms steps, so 60 s stalls run in well under a second. Per step: one token committed + on the worker (100 tokens/s), the throttle's delivery, then every host's turn. Phases: the first snapshot of the + 500 KB view (8 s of virtual time at 64 KiB/s), 2 s of streaming, the pause (a stall: the tab receives nothing, so + it acknowledges nothing; or the control: the same streaming without a stall), then 10 s more of streaming and + until the tab's text equals the worker's. *Caught up after the stall*: virtual time from the end of the pause to + the first step where the tab's view equals the worker's document; *bytes until then*: what the throttle delivered + meanwhile. *Heap growth during the stall*: the + process's heap after full collections at the end of the pause minus at its start (all hosts share one process; + compare with the control row, which is the same streaming without a stall). *Commit at the worker*: the mean time + of `store.stream` (the transaction, its flush: an encode when the window has room, a push into the pending queues + when it is full), outside the pause and during it. *Longest message handling at the provider*: the longest single + received message handled after the pause at the worker, the provider of the tab's state: its end of the tab link + directly, of the route link relayed (the `ack` that sends the folded `state`). +- *calls*: tab B's sequential `ping`s through the relay, alone and while tab A has issued 10,000 `ping`s at once on + the same session (two holds, two origins at the worker, each with its N = 256 running calls; the relay forwards at + once). + +| Topology (1,000 streamed tokens into the 1,000-item notes list view) | call to frame | bytes per commit: to the tab / from the worker | +| --- | --- | --- | +| memory tab, in-process worker (memory links) | 0.053 ms | 27 B / 29 B | +| memory tab, forked worker (IPC) | 0.097 ms | 27 B / 29 B | +| first hold to live, the fork included (one sample) | 240 ms | - | +| loopback WebSocket tab (`ws` serving, Node's `WebSocket`), forked worker (IPC) | 0.148 ms | 27 B / 29 B | +| no relay: memory tab on the worker (baseline) | 0.034 ms | 27 B / - | + +| Hold to live, the 1,000-item notes list view (133.7 KiB snapshot) | time | +| --- | --- | +| first hold, `resolve` of an in-process worker included (mean of 20) | 1.375 ms | +| hold of a resolved session through the relay (four hops, mean of 100) | 0.763 ms | +| no relay: hold on the worker (two hops, mean of 100, baseline) | 0.703 ms | + +| Throttled tab (64 KiB/s), 100 tokens/s into the 500 KB view | caught up after the pause | bytes to the tab until then | heap growth during the pause | commit at the worker, outside / during the pause | longest message handling at the worker after the pause | +| --- | --- | --- | --- | --- | --- | +| direct, 1 s stall | 0.02 s | 485 B | 28.0 KiB | 0.018 ms / 0.030 ms | 0.169 ms | +| direct, 10 s stall | 0.08 s | 4147 B | 134.2 KiB | 0.029 ms / 0.023 ms | 0.053 ms | +| direct, 60 s stall | 0.38 s | 23.7 KiB | 392.2 KiB | 0.028 ms / 0.017 ms | 0.138 ms | +| direct, 60 s without a stall (control) | 0.01 s | 23 B | 482.7 KiB | 0.025 ms / 0.017 ms | 0.005 ms | +| relayed, 1 s stall | 0.02 s | 485 B | 7.5 KiB | 0.025 ms / 0.025 ms | 0.057 ms | +| relayed, 10 s stall | 0.08 s | 4147 B | -25.3 KiB | 0.030 ms / 0.015 ms | 0.059 ms | +| relayed, 60 s stall | 0.38 s | 23.7 KiB | 513.5 KiB | 0.026 ms / 0.015 ms | 0.146 ms | +| relayed, 60 s without a stall (control) | 0.01 s | 23 B | 257.1 KiB | 0.023 ms / 0.021 ms | 0.003 ms | + +| Calls through the relay (in-process worker, memory links) | p50 | p95 | max | calls | +| --- | --- | --- | --- | --- | +| tab B alone | 0.012 ms | 0.018 ms | 0.600 ms | 1,000 | +| tab B while tab A has 10,000 calls in flight or queued | 4.851 ms | 10.701 ms | 10.708 ms | 27 | +| tab A: 10,000 calls at once, until the last result | 657 ms | - | - | 10,000 | + +Observations: + +- A relay hop costs about 0.019 ms per commit-to-frame on memory links (0.053 against 0.034 ms for call, commit, frame + and result) and adds no bytes to the tab: the server forwards the worker's message with the origin taken off (ids + unchanged, parts copied once); the origin costs two bytes per message on the route link. IPC to a forked worker adds about 0.04 ms, a loopback WebSocket to the tab another 0.05 ms. +- Hold to live through the relay is within 9% of a direct hold (0.763 against 0.703 ms): the relay never parses the + 133.7 KiB snapshot. A first hold that resolves an in-process worker (create, connect, load, activate) takes 1.4 ms; + forking a strip-types Node process takes about 240 ms, which is the app's to hide (start workers ahead). +- Directly, the provider's pending queue folds a stall of any length into one append (24 KiB of text after 60 s, + sent in 0.38 s at 64 KiB/s); committing while the window is full costs no more than committing with room (about + 0.02 ms), and the folded message is encoded in under 0.2 ms. The stall adds no heap beyond the control's, which is + the streamed text itself on the worker and the tab. +- Through the relay a stall now costs what it costs directly (decision "Relay flow control per origin"): the worker + keeps the tab origin's window and folds its commits into its pending queues, the relay forwards and acknowledges + per message. Before this rework the relay ran the tab's window itself and recovered with `reset`: every stall, even + 1 s, cost three full snapshots (about 1.5 MiB) and about 24 s to catch up under sustained streaming. +- Tab B's calls are their own origin at the worker, with their own N, but B's call still waits behind the up to 256 + calls of A running there (one worker, one event loop): about 4.9 ms median, 11 ms p95, against 0.01 ms alone. A's + 10,000 calls take 0.07 ms each (its window of 256 refills one reply round trip through the relay at a time). +- A first run of this script found about 350 B retained per call or commit: `AbortSignal.any` on Node 24.4.1 keeps + cleanup entries per call while its sources live (nodejs/node#62367, fixed in 24.16.0). Calls and gets now link + their signals only until they settle (`withLinkedAbortSignal`); the numbers above are after that fix (heap during a + 60 s stall: 2.3 MiB before, 0.4 MiB after). + +## Slice 4b: authority and housekeeping + +Script: `test/next/bench/compression.ts`, run from `packages/chord` with +`node --conditions=source --experimental-strip-types --expose-gc test/next/bench/compression.ts` (about 5 s). Node +24.4.1, Intel Core i9-13900, one process. + +Payloads: the 500 KB document of slice 3a (500 entries `{ id, text, author, at }` of 960 characters), but with +prose-like text (words from a small vocabulary in a fixed pseudo-random order, some with numbers: it compresses about +3.6x at level 1, as conversation JSON does; the `"x".repeat` texts of the earlier benchmarks would compress about +100x and say nothing), and a 1 MiB string as a call's result (a file export). Compression is option (b) as built: +`nodeCompression` (zlib's raw deflate at level 1) at the origin; the owner inflates with `nodeCompression` +(synchronous, a Node client) or `DecompressionStream` (asynchronous, a browser). + +- *deflate at the origin / inflate*: one part through the adapter alone, mean of 20, the fastest of three loops. +- *end to end*: a server and a client host on a memory link (binary codec); *hold to live* is a further `get` of the + live view (snapshot encoded, deflated, sent, inflated, parsed, replicas built), *call to result* a call returning + the 1 MiB string; *bytes received* is the one message as the client's link delivered it. + +| Payload | JSON | deflated (level 1) | deflate at the origin | inflate, sync (`nodeCompression`) | inflate, async (`DecompressionStream`) | +| --- | --- | --- | --- | --- | --- | +| 500 KB snapshot | 494.9 KiB | 137.4 KiB | 3.308 ms | 0.915 ms | 1.400 ms | +| 1 MiB result | 1024.0 KiB | 286.1 KiB | 6.856 ms | 1.757 ms | 3.113 ms | + +| End to end over a memory link (binary codec) | snapshot: bytes received | hold to live (mean of 20) | 1 MiB result: bytes received | call to result (mean of 20) | +| --- | --- | --- | --- | --- | +| uncompressed | 494.9 KiB | 1.197 ms | 1024.0 KiB | 1.939 ms | +| deflated, inflated synchronously (Node client) | 137.5 KiB | 5.632 ms | 286.1 KiB | 10.868 ms | +| deflated, inflated asynchronously (browser client) | 137.5 KiB | 6.356 ms | 286.1 KiB | 12.198 ms | + +Rerun after the review fixes (inflate only after admission by the part's owner, a tagged adapter result, byte counts +for the threshold): the same within noise (the asynchronous inflate of the 500 KB part varies between 1.4 and 2.0 ms +from run to run). + +Observations: + +- Level 1 cuts prose-like JSON to 28% (494.9 to 137.4 KiB, 1 MiB to 286 KiB) at about 6.7 ms per MiB deflated at the + origin and 1.8 ms per MiB inflated synchronously; `DecompressionStream` takes about twice as long as zlib's + synchronous inflate (3.1 against 1.8 ms per MiB) and adds its own turns, which the in-order holding of the link's + later messages absorbs. +- On a memory link (no transport time) compression only costs: hold to live 1.2 → 5.6 ms, the 1 MiB call 2.0 → + 10.9 ms. It pays where bytes take time: at 10 MB/s a 500 KB snapshot takes about 50 ms uncompressed and 14 ms + compressed (plus 4.3 ms of deflate and inflate); at 1 MB/s (a phone) 500 ms against 140 ms. On IPC to a worker + (gigabytes per second) it does not pay for itself on that hop: it pays on the tab's WebSocket, where the relay + forwards the worker's deflated part as it is (a test counts no inflate at the relay). +- Parts at or below 16 KiB (UTF-8 bytes) are never compressed (a state change, most call arguments), nor is a part + that deflating does not make smaller (sent as the bytes encoded for the attempt, never encoded twice), nor hold + params, nor call arguments sent up a parent link: the cost per message of the slices 3a-4a benchmarks is unchanged. The binary codec's part prefix (length doubled, plus one when deflated) costs + a byte more for parts between 64 and 127 bytes. + +## Slice 6a: the store + +`test/next/bench/store.ts` (run by hand: `node --conditions=source --experimental-strip-types --expose-gc +test/next/bench/store.ts` from `packages/chord`): Node 24.4.1, `chord/sqlite/node` on a fresh SQLite file per burst +for both sides (the same pragmas and file system), 7 bursts of 1,000 commits after a warm-up burst; per commit, the +median and p95 of single commits and the burst's mean. + +The gate (decision "The commit-step benchmark"): the store without outbox no slower than Durable's `SqliteStorage`. + +| Case | Durable | Chord store | Chord store + outbox | +| --- | --- | --- | --- | +| (a) one document delta per commit (step level) | 23.7 / 31.4 / 32.9 µs | 13.4 / 18.7 / 20.9 µs | 16.6 / 29.2 / 26.7 µs | +| (b) one table `put` (Durable: one entry append) | 19.3 / 27.1 / 29.5 µs | 13.2 / 23.6 / 20.6 µs | 17.5 / 28.1 / 26.3 µs | +| (c) whole transactions back to back, no watch (Durable `Session`, Chord host) | 35.0 / 61.1 / 44.4 µs | 18.1 / 27.2 / 25.7 µs | 20.3 / 38.8 / 30.7 µs | +| (c) the same with a watched document | 34.1 / 55.4 / 42.4 µs | 20.6 / 26.3 / 27.1 µs | 20.2 / 39.0 / 29.9 µs | +| (d) one in-memory document per transaction (no store) | | 2.5 / 2.9 / 2.7 µs | | +| (d) the same, watched | | 2.6 / 4.4 / 2.7 µs | | + +Every gate case passes. The outbox adds 3-5 µs per commit (one `INSERT` of the commit's writes as JSON). The heap after +GC at the end of the whole script is about 3 MiB above its start (the modules and statement caches loaded on the way). + +In-memory transactions against HEAD (before slice 6a), after the second fix round (`Transaction.draft` returns its +draft's promise as it is: an in-memory `change` settles in 5 microtask turns, HEAD 6, a test guards it): one `change` +per transaction through a host, 15 bursts of 1,000 per process with a timer around each `change`, HEAD's source +extracted beside this checkout, the same script on both, alternating processes pinned to one CPU (`taskset -c 4`), +three runs; median / p95 / burst mean: + +| Case | HEAD | now | +| --- | --- | --- | +| unwatched document | 2.52-2.55 / 4.96-5.02 / 5.41-5.44 µs | 2.56-2.60 / 5.02-5.06 / 5.69-5.71 µs | +| watched document | 2.96-3.07 / 6.44-6.54 / 6.81-6.90 µs | 2.96-3.01 / 6.35-6.42 / 6.88-6.95 µs | +| watched derive through a reference | 3.14-3.29 / 6.77-6.93 / 8.38-8.50 µs | 3.06-3.08 / 6.49-6.53 / 7.52-7.54 µs | + +What the differences are (the second review's measurements, Opus): + +- The change path itself is at HEAD's level or below. Without the per-operation timer, a tight loop of 200,000 changes + costs 3.67-3.69 µs against HEAD's 3.70-3.78 µs (and the loop of the timed harness 4.85-5.06 µs against + 5.15-5.24 µs). Allocation per change is lower (7,703 B against 8,035 B, measured with a semi-space of 128 MB so no + scavenge lands in a burst), and so are the collections (40 hosts of 2,000 changes: 34 scavenges against 38, 2 + mark-compacts on both). +- The median difference of the timed harness (+1-2% here, up to +3-7% elsewhere) comes from the per-operation timer: + it stays with zero collections in the bursts and disappears without the timer, so it is harness-sensitive (code + layout, inline caches), not work in the change path. +- The burst means differ by where a mark-compact lands: the worst burst per process coincides with a mark-compact over + a larger old space (67 MB against 45 MB before the collection, about 18 MB after on both; a pause of 5.2-7.1 ms + against 3.0-3.7 ms), which is heap-growth timing, not extra garbage. Host setup is about 3% above HEAD (21.9-23.2 µs + against 21.1-21.9 µs per host, create, load and get; close equal), which the burst means include too. + +Calls and references (`taskset -c 4`, five alternating pairs of processes; one host, 21 bursts of 3,000 operations +after 3 warm-up bursts, a timer around each; reads: 200,000 per burst, burst means). Every call admits its handler to +the drain and its cleanup (call count, `ctx.get` references) in one map entry with one reaction: 19 promises per call +against HEAD's 20, a test guards it. A reference's state members are its object's member sources (one per object and +member, re-bound when a new attempt goes live): a watch through a reference gets the member's frame passed on as it is +(one call more per frame), a read goes through one getter more. + +| Case | HEAD | now | +| --- | --- | --- | +| local call: median / p95 / burst median / burst p95 | 1.49-1.51 / 1.93-1.99 / 1.64-1.67 / 2.51-2.75 µs | 1.47-1.48 / 1.93-1.96 / 1.59-1.61 / 2.40-2.47 µs | +| `change` under a watch through a reference, derive member: median / p95; frames per second | 2.80-2.92 / 4.64-4.80 µs; 268,000-282,000 | 2.76-2.87 / 4.75-4.84 µs; 278,000-288,000 | +| the same, document member | 2.68-2.79 / 4.58-4.79 µs; 285,000-306,000 | 2.67-2.76 / 4.57-4.79 µs; 285,000-306,000 | +| `ref.member.value`, derive member | 9.24-10.82 ns | 8.53-8.57 ns | +| `ref.member.value`, document member | 10.02-10.10 ns | 10.30-10.32 ns | + +Calls are 1-3% faster than HEAD (bursts' p95 10% lower), watches through a reference at HEAD's level (p95 +1-2% for a +derive member, more frames per second), a derive member's read faster; a document member's read costs 0.25 ns more +(the member source's getter in front of the slot's). + +A local document member re-exposed by a remote object and held over a memory link (one `change` and its state message +per operation; Astra's third review harness, `taskset -c 4`, alternating pairs, 41 bursts of 10,000 after 3 warm-up +bursts): a member source passes its subscribers (holds, member sources re-exposing it) on to the member it follows, so +their frames take no extra hop. Burst median 7.80-8.38 µs against HEAD's 7.82-8.35 µs, burst p95 8.84-9.44 against +9.52-10.06 µs, median per operation 6.76-7.34 against 6.82-7.29 µs (before the pass-through, one run: 8.51 / 13.05 µs +against 7.99 / 10.36 µs). Host setup (+2-4%) and a document member's read through a reference (+0.25 ns) are accepted +as they are. + +After Delta's tracker stopped allocating a `WeakRef` per change (pi a92283056, merged as 95ec93a08), against HEAD +95ec93a08 (which has that change too) extracted at a path as long as this checkout's (44 characters). Sol's tight +harness without a per-operation timer (`taskset -c 4`, five alternating pairs of processes, 31 bursts of 10,000 after +3 warm-up bursts; medians of the burst medians, ranges in parentheses): + +| Case | HEAD | now | +| --- | --- | --- | +| unwatched `change` | 1.96 µs | 1.96 µs (0.0%) | +| `change` under a watch of the document | 2.24 µs | 2.19 µs (-2.1%) | +| `change` under a watch through a reference | 2.23 µs | 2.21 µs (-0.8%) | +| `change` under a watch of a derive through a reference | 2.33 µs | 2.26 µs (-2.9%) | +| local call | 1.36 µs | 1.33 µs (-1.9%) | +| a document member forwarded into a remote object, held over a memory link | 6.60 µs (6.54-6.63) | 6.52 µs (6.48-6.59) (-1.2%) | +| the same, yielding a macrotask every 1,000 operations | 6.67 µs (6.57-7.02) | 6.54 µs (6.42-6.79) (-1.9%) | + +Collections over one forwarded-link run (`--trace-gc-nvp`, 340,000 operations, two processes each; scavenges and +promotion were the same in both): + +| Run | Promoted | Scavenges | Mark-compacts | GC pauses | +| --- | --- | --- | --- | --- | +| HEAD | 6 MB | 158 | 2 | 154 ms | +| now | 7 MB | 148 | 2 | 147-148 ms | +| HEAD, yielding every 1,000 | 6 MB | 186 | 2-3 | 149-162 ms | +| now, yielding every 1,000 | 7 MB | 178 | 2-3 | 145-149 ms | + +History: the fifth fix round's measurement, before the Delta change (failure calls once per episode, watches on +failed sources, keyed pins still loading). Sol's tight harness without a per-operation timer (`taskset -c 4`, five +alternating pairs of processes, 31 bursts of 10,000 after 3 warm-up bursts; burst medians, HEAD's source at +`/tmp/chord-head4`, this checkout at its own path): + +| Case | HEAD | now | +| --- | --- | --- | +| unwatched `change` | 2.91 µs | 2.95 µs (+1.5%) | +| `change` under a watch of the document | 3.14 µs | 3.10 µs (-1.4%) | +| `change` under a watch through a reference | 3.10 µs | 3.08 µs (-0.4%) | +| `change` under a watch of a derive through a reference | 3.21 µs | 3.26 µs (+1.6%) | +| local call | 1.37 µs | 1.35 µs (-2.0%) | +| a document member forwarded into a remote object, held over a memory link | 7.80 µs | 8.32 µs (+6.6%) | +| the same, `--no-allocation-site-pretenuring` | 8.12 µs | 8.04 µs (-1.0%) | +| the same, yielding a macrotask every 1,000 operations | 7.17 µs | 8.61 µs (+20%) | +| the same, yielding, `--no-allocation-site-pretenuring` | 7.21 µs | 7.52 µs (+4.2%) | + +Allocation per operation, with a fixed semi-space of 256 MB so no scavenge lands in the measured 20,000 operations, is +below HEAD's in every case: 6,942 B against 7,317 B unwatched, 8,383-8,393 B against 8,764 B watched, 8,935-8,939 B +against 9,338 B for a derive, 8,405-8,413 B against 8,764 B through a reference, 2,885 B against 3,089 B over the +link. What stays after the link drains and a GC is equal (18.9 against 19.0 B per operation). + +The forwarded-link gap is not work or retention in 6a's code. It is a V8 decision that depends on heap layout: + +- With huge semi-spaces (no scavenges) the two take the same time (10.01 against 10.00 µs). Without pretenuring (the + flag above) they are equal too. +- An identical copy of this checkout's source runs at HEAD's level (7.1 µs, 7 MB promoted over the run) when its path + is at most 40 characters long (`/tmp/wexp/packages/chord/src`). From 44 characters on it flips: 46 MB promoted, 4 + mark-compacts instead of 2, about 9 µs. The repository's own path is 44 characters. HEAD's source does not flip at + any tested path length (28 to 62 characters). +- `--trace-deopt` shows the flip: V8 marks the allocation site of the delta tracker's per-change overlay context + (`createContext`, inlined into `beginChange` and `Transaction.#begin`) as tenured. From then on every change's context + is allocated in old space and dies there. +- The contexts survive scavenges because each change registers its context as a `WeakRef` target + (`TrackerImpl.#register`). A `WeakRef` keeps its target alive until the current job ends, and scavenges do not + clear it. So a site that allocates one context per change shows a high survival rate. +- Heap snapshots taken during and after the run are equal between the two trees, and no 6a object outlives its + change. +- pi a92283056 ("Delta's tracker holds its one open change without a WeakRef") removed the cause: the numbers above + it show the forwarded link at HEAD's level with 6-7 MB promoted and 2 mark-compacts on both. + +A retirement under a diamond of derives (`test/next/bench/derives.ts`: 40 derives in 13 layers of two derives joined +by a third, 2^13 paths from top to bottom, the bottom watched; one step retires and recreates the document in one +transaction): 9.8 µs median, 17.7 µs p95 per step with one visit per derive per batch; 349 µs without the visited set +(one visit per path). + +Binary `state` messages (`test/next/bench/links.ts`, against HEAD, alternating, pinned): an entry without retirements +costs no extra byte (165 / 27 / 23 B for one appended item, one changed field, one removed item, as at HEAD); encode +and decode are at HEAD's level (one changed field, 200,000 runs, best of 9: encode 1,167-1,203 ns against +1,172-1,275 ns, decode 692 against 685-705 ns); heap per encode and decode 1,592 / 1,631 / 1,363 B against 1,684 / +1,726 / 1,355 B; commit to frame within HEAD's range (0.112-0.166 ms against 0.111-0.164 ms). After the second fix +round (`taskset -c 4`, two runs each): commit to frame 0.086-0.087 / 0.097-0.099 / 0.065 ms against 0.084-0.089 / +0.098-0.099 / 0.064 ms, a streamed append 0.053 ms against 0.051-0.053 ms, the same bytes and heap. + +Other costs (one run each): + +| Cost | Result | +| --- | --- | +| 100 commits submitted at once: time waiting in the queue against the step itself | median wait 956 µs, step 11.3 µs | +| a query page of 100 rows in an index's order (10,000 rows, cursors) | 113 µs per page | +| one `updateWhere` over 10,000 rows | one step of 10.6 ms (every owner's commits wait behind it) | +| `put` with 10-character text cells / 10,000-character cells (2,000 rows, one commit) | 2.4 / 58.4 µs per row | +| a `rows` migration of 10,000 rows (100 chunks; `taskset -c 4`) | 69-71 ms in all, 7.0 µs per row; chunk commits: first 0.34 ms, last 0.21 ms, median 0.22 ms | + +Observations: + +- The commit step is cheaper than Durable's per commit: Chord keeps no generic entry log beside the documents engine + and its tables, and serializes writes before the queue (rule 8). +- Between chunks a migration yields one macrotask without the 1 ms clamp of `setTimeout(0)`: `setImmediate` where the + runtime has it, else a `MessageChannel` message (Workers, browsers). The same 10,000-row migration takes 191 ms with + `setTimeout(0)` (99 clamped yields) and 100 ms with messages in Node (0.27 ms per yield there, against 13 µs for + `setImmediate`). +- Late migration chunks cost what early ones do (paging forward by the last key keeps each chunk's read bounded); the + spec's estimate of about 15 µs per migrated row holds with room. +- A bulk write's step is the one place a single commit holds every owner's commits for milliseconds; the guide says so. diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md new file mode 100644 index 000000000..c87c860ca --- /dev/null +++ b/packages/chord/docs/next/decisions.md @@ -0,0 +1,2168 @@ +# Redesign decisions (input for target design v2) + +Decisions made with the owner after the plugin review round on `target-design.md` (6559589bd), in the order they were +made. Later entries supersede earlier ones; superseded entries are marked inline and point to their replacement. The target design v2 is `packages/chord/docs/next/design.md`, written from this log; new decisions land here +first and win until the design is updated. + +## Plan + +- **p3 next to p2, fully isolated.** The new design is built as a new app (`apps/p3`) with its own data directory. p2 + keeps running unchanged with its data. Plugins are ported to p3 one at a time. +- **No legacy data migration.** p3 starts empty: no import of p2's `catalog.sqlite`, `shares.sqlite` or subagents table. +- **Chord v2 lives in `packages/chord/src/next`** (export `@earendil-works/chord/next`), developed on `private` only + (owner override of the "pi first" rule for this directory) and upstreamed to pi once stable; then `next/` replaces + today's Chord and p2 is deleted. `next/` imports `../context`, `../delta`, `../json`, `../output` directly, so + Durable's `Context`, `Draft` and `JsonValue` are the same modules and types in v2 (no copies, no type conflicts). + (Replaces "Chord v2 lives inside p3".) +- **Build a playground before p3:** a small system with p3's shape (server, worker processes over IPC, clients over + WebSockets) and toy plugins, one per concept; it proves the concepts and reload and is the e2e test ground. + Location `apps/chord-playground` (private). Client: a web page first (a CLI client possibly later). Order: kernel + (hosts, links, objects, references, principals), then plugin loading and reload with the dev loop, then the web shell + with a private preview, then each feature built live as a plugin. Plugin changes reload; Chord changes restart the + hosts (fine for the owner). +- **Chord v2 is written fresh** in `packages/chord/src/next`, copying what fits from today's Chord (link, wire, cell, + delta and the like). It needs the same test coverage as today's Chord (tests under `packages/chord/test/next`). +- **Playground web stack:** React, built with esbuild (loader D) instead of Vite, if feasible. +- **Reload control in the playground's web shell:** when sources changed, a control pops up listing the changed + plugins (reload them) or base code (restart all hosts); the owner triggers it manually. The agent can trigger the + same actions (command or tool). Never automatic on save. +- **Preview:** `https://p3.mariozechner.at` on slayer (wildcard DNS already resolves), set up like + `~/p2.mariozechner.at` (Caddy container on `caddy-network`, unix-socket bridge, `run.sh` in tmux) with `basic_auth`; + the password is chosen by the owner and never stored in the repo. `p2.mariozechner.at` stays the live p2. + +## Natives + +- **Natives only in objects.** An object opens a native in its synchronous factory with + `self.open(async ({ signal }) => value, close)`; Chord opens it during activation (the object goes live after all + opens) and closes it when the object ends, in reverse order. The value never changes during the object's life: no + `self.resource`, `version`, `replace`, `deps`, reopen, stale handles or "read `.value` at use". +- **The factory principle:** synchronous, declares only (references, documents, natives, derives, routines, jobs, + contributions); Chord performs activation; the factory receives handles usable in everything it registers; handles + are valid for the object's whole life. +- **No natives across plugins:** other plugins reach a native only through the owning object's methods and state, or + through the owner's registry entries ("Registries and HTTP"). No callbacks as method arguments, also not on + `local` types ("Object kinds"). +- **The facet is a pure object.** (**superseded in part** by "Facet documents are removed" and "Facets only declare" (2026-10-26): no documents, no routines) It may hold documents, references, routines and contributions, and may not open + natives (`self.open` does not exist on the facet). Served objects of a facet may read and write the facet's documents; + a served object's transaction may span its own documents and its facet's documents (lock order: own line, then the + facet's). The facet may not reach into its served objects' documents. + +## API style + +- **Tagged unions, always,** instead of a status string plus fields that only mean something in some states ("field + soup"). Each state carries exactly its own fields, and consumers narrow on `kind`. Example: `ref.state` is + `{ kind: "failed"; cause; retryAt } | { kind: "live" } | ...`, not `status` plus `error` plus `retryAt`. + +- **Context last (owner, 2026-10-21):** parameters are the required arguments, then the context (`ctx` or a Durable + `Context`), then optional arguments, as Durable already does (`Storage.scanEntries(query, limit, cursor, context)`, + `transaction(change, context)`). Examples: `self.tx(fn, ctx)`, `handle.change(fn, ctx)` and keyed + `handle.change(key, fn, ctx)`, `handle.retire(ctx)` and keyed `handle.retire(key, ctx)`, `chord(fn, context)`, + `job.start(args, ctx, { key })`. Applied to design.md, the guide and the walkthrough; the frozen `step-*.md` and + `kernel.md` document the frozen code and keep its signatures. pi's `traceIn(context, options, fn)` (implemented in + `chord/context`) becomes `traceIn(options, fn, context)` in the same pi change as S1 (owner, 2026-10-21). + +## Hosts and placement + +- **Placement is a function** of the object's typed input, not data naming an input field (`hostFamily(prefix, + key)` looked a key up in the input untyped). A function is type-checked and composable. The caller runs it and + sends the result; the provider runs it too and refuses an object that is not its own. Placement is then no longer + part of the contract fingerprint; a changed rule surfaces as that refusal. +- **Incarnations are minted by Chord** (uuidv7, the code copied from pi-ai into Chord: no dependency), never passed + in; the link handshake carries it, so the other side sees a restart as a new incarnation of the same identity. + +- **The low-level host API** (no topology layer; Radius and DO show entry points vary, so the kernel has no + "listen"): + - host kinds are typed: `hostKind("server")`, `hostKind("session", Type.Object({ id: Type.String() }))`; an address + is `Session({ id: "S1" })`; placement functions return addresses; no host-name strings in app code; + - `createHost

({ address?, principal? })`: a host with an address is addressable and acts as `principal` on its + own host; no options is a client (not addressable, no own principal); the incarnation is minted inside; + - (**superseded in part** by "Host values are removed" (Review round 1): start data is a core object) `host.provide(Key, value)` before `load`; factories read host values with `self.value(Key)`; + - `host.load(facets)` takes imported facets (loaders are adapters); + - `host.connect(link, { principal })`: the link to the parent (at most one), `principal` = who the parent is; + today's duplicate `upstream` option is gone; + - `host.accept(link, { principal })`: a link from a client; every entry point (loopback/tunnel WebSocket server, + Radius channels, a DO edge, memory) is app or adapter code that authenticates and calls `accept`; + - `host.route(Kind, { authorize(principal, address), resolve(address, { signal }) => { link, principal } })`: + how to reach hosts of a kind; `authorize` runs before anything starts; Chord calls `resolve` once per address + while its link lives and again after a drop; the returned principal is stamped on that link (defined once, in + the parent; the app sends it to the child in its own start message, with any init data); + - starting processes, init data and keep-alive are app code (helpers later if apps repeat them). + Close to today's shape (`createHost`, `connect`, `accept`, `route`, `load`); gone: string host names and prefix + parsing, app-managed generations, `upstream`, `opaque` runtime objects, the loader inside the kernel. + +## Quick decisions on the design's open points (design.md section 26) + +- **Call-scoped natives (#4):** no `ctx.open`; a native that lives for one call is opened and closed in the method with + `try/finally` (documented pattern). Revisit only if it shows up often. +- **Documents across plugins (#6):** no plugin reads or writes another plugin's documents (Chord or Durable). Data + crosses plugins only through state and methods. ~~Within a plugin, the facet's documents are readable and writable + by the objects that facet serves (decided earlier).~~ (**superseded** by "Facet documents are removed") Data core enforces lives in a format core owns and is handed to + core through its API when the data is created (e.g. `createConversation` with `init`), never written into core's + documents directly. +- **`overlap` for routines (#8):** no `overlap: "latest"` option; a handler compares with the current value itself. +- **Async activation guard (#3):** none; authority checks live in the per-principal view's methods (and in + `route.authorize` before a host starts). +- **`close` with a reason (#7):** `self.open(open, (value, reason) => …)` with `reason: { kind: "reload" } | { kind: + "stop" } | { kind: "end" }`, decided by Chord (its plugin is in the reload set and the object is native / the host is + shutting down / the last reference ended or its type was removed). Owners decide what to do with it. Chord aborts + only the object's own calls and jobs; work elsewhere that uses the native (e.g. Durable tool executions using a code + mode sandbox) is the owner's to drain or end in `close` ("old code keeps running" applies to natives too). +- **Code mode in p3 drains its sandbox in `close`** (**superseded in part** by R16 (`close` detaches the sandbox and disposes of it in the background)) (not moved into Durable): the plugin keeps owning the + `pi-codemode` sandbox; on `reload` (and `stop`) `close` seals it (no new scripts), running scripts finish or hit + their deadline, then its VMs and worker threads end (`closeWhenIdle` in `pi-codemode`, or a counter in the plugin). + New scripts start on the new sandbox through the registry entry. `RunningCalls`, the removal message and + `abortTask` on reload go away; after a worker stop Durable replays as today. +- **Invariants across owners (#5):** the owner of the dependent data enforces the rule by reconciling from the other + owner's state with an idempotent routine; no cross-owner transactions, no "notify the other owner" calls that can + be lost. Example: ports' forward store watches the catalog and closes forwards of archived sessions (also after a + crash, from the current value); sessions does not know ports exists. + +## Wire (#15) + +- ~~**Compression:** large JSON envelopes are deflated on WebSocket links (relayed and mobile links benefit); byte + segments stay raw and uncompressed; IPC never compresses.~~ **Superseded** by R1 and "Compression at the origin". +- **Frame size limit per link adapter** (**superseded in part** by Q2 (limits per part from the header; `maxPayload` only as the hard cap)) (e.g. 16 MiB on client WebSockets, none on IPC): a frame over the limit fails + that call or member with `too_large`, the link stays up. Large uploads go through owner chunking anyway. +- **One batch per commit per link:** a commit that touches several holds on one link travels as one `state` frame; + the consumer applies every member update of it before listeners run. Remote consumers never see a torn commit, + matching local subscribers, who are notified once after every document of a commit is adopted. +- **Adapted bindings get diffs:** a consumer bound through `accepts` (an older contract version) is served like a + derive of the provider's state through the adapter: Chord keeps the last adapted value per binding, applies the + adapter to the new value and sends the diff (`diffRevisions`), never the whole adapted value per commit. + +## Gaps closed after the walkthrough rework + +- **Clients** serve only `local` types (not addressable); local calls have no principal; what a client calls on the + server is stamped with the client's authenticated principal. +- **Placement refusal** is `invalid/placement`; during a rollout it is handled like a mismatch (provider on another + revision: park; same revision: fail). +- **HTTP route signature:** `(req, ctx) => Promise`; `ctx.principal` is the request's principal (from + the request resolvers), so `ctx.get(Notes, …)` gives that principal's view. +- **Start barrier** (**superseded in part** by R17 (the harness starts after the barrier, which waits until every plugin loaded or failed)) counts facets and `lifetime: "host"` objects; an object that must contribute at start (the model + runtime) is `lifetime: "host"`. +- **`persist` only in `local` objects and `lifetime: "host"` objects** (**superseded** by P3: only in input-less `local` + objects) (identities without a principal); `storeKey` + is dropped (two principals' objects writing one document would be two lines on one document). Enforced at the type + level (`self.state` in a remote factory takes only non-persisted definitions) and as a declaration error at runtime + ("persist needs a stable identity: put the documents into a local core"). +- **Call deadlines:** a call ends through the caller's signal; there is no per-call deadline of its own. Parked calls + (link drop, loading, rollout) share one default deadline (60 s, like `readyTimeout`), then fail `unavailable`. +- ~~Lingering through `ref.release({ linger: true })`~~ **superseded** by D8 (no consumer-side linger). Was: a closing view releases with `ref.release({ linger: true })`; the hold stays, marked lingering, until + its timeout. A link drop is still a departure (presence details stay open). +- **Contributor order** (**superseded in part** by "Contributor order is sorted by plugin name (never load order)" in the design, section 17) is load order (alphabetical by plugin name), deterministic on every host and across reloads, + and carries no meaning: wraps in one phase must not depend on each other; meaningful order comes only from phases. +- **Revocation without field patterns:** the app defines a typed revocation type and a function + `createHost({ …, revokedBy: (p: P) => R[] })` (the same app code on every host); `host.revoke(r)` ends every + object and closes every link whose principal `p` has `r` in `revokedBy(p)`, and propagates `r` (a small JSON value) + to the children, which apply their own `revokedBy`. No `PrincipalPattern`, no local-only predicates, no re-stamping: + a changed grant (mode `write` → `read`) is a revocation of the old one; the client reconnects and authenticates + anew, and its references re-hold. +- **Per-session native owners:** (**superseded in part** by "Host values are removed": the session is read from the core object `SessionInfo`) the facet cannot `self.get` a type it serves, so a worker-wide native owner is a + `lifetime: "host"` object without input that reads its session from a host value (`self.value(SessionInit)`). + +## Documents step + +- **Plugin identity:** every plugin has a `package.json` whose `name` is its globally unique, stable plugin name (like + an npm package name); that name identifies its facets (~~persisted facet documents,~~ reload, diagnostics; **superseded in part** by "Facet documents are removed"). Loader D + reads it; static composition passes the same name. Never a directory name, never load order. +- **Document roots are `JsonObject`,** as in Durable (`defineDoc`); lists live in a field. Documents + hold JSON only. +- **In-memory documents end with their object** (last reference released, linger expired); they survive reloads and + native restarts of a live object. What must outlive the object is persisted or lives in a `lifetime: "host"` object. +- **Chord ships its own document storages, no adapter over Durable's storage:** (**superseded** by "The store" + (2026-10-21): Chord's persisted documents live in the per-session store, next to Durable's adapter and plugin tables) in-memory and SQLite (JSONL was cut later) + implementations of `DocumentStorage`, inspired by Durable's storage implementations (atomic commits, base plus + deltas, incarnations) but with Chord's own schema: ordered keys per owner and definition, a tag index. Durable's + storage is shaped for Durable (scans by id, no tags); Chord documents and Durable transcripts never commit together, + so they need not share a storage. Node-specific storages live outside the portable core; a Durable Object SQLite + storage can follow. (Replaces the earlier "adapter over Durable's node SQLite storage" for the playground.) +- **Durable's SQLite driver layer moves into Chord** (`@earendil-works/chord/sqlite`, `/sqlite/node`, + `/sqlite/cloudflare`): the `SqliteDatabase`/`SqliteExecutor` interfaces, the `node:sqlite` and Durable Object + drivers, the migration runner and schema-prefix helpers. Durable imports them from Chord (it already depends on + Chord, so no cycle) and re-exports them under its old paths so no Durable user breaks. Chord's SQLite document + storage uses the same drivers (`next/` imports `../sqlite` like `../output`). Done in pi first, then merged. + +## Derives and natives step + +- **`self.open` returns a handle** whose `.value` is the native (throws before activation); no forwarding proxy + (private fields and identity break through proxies). The value never changes during the object's life. +- **A partly failed activation** closes the natives already open (reason `end`) and opens all of them again on the + next attempt. +- **A broken part fails the whole object** (owner): a derive that throws, or a document whose hydration or `migrate` + fails, makes the object `failed`; an incomplete object could break invariants that rely on atomic commits across its + documents. (Replaces "member-local" errors for documents.) +- **`previous` of derives** is tested through failed/retry until the reload step exists. +- **`self.retryOn` may appear anywhere in the factory:** the factory specifies, the host orders and executes (watches + are installed before natives open). +- **Failure causes are tagged:** `{ kind: "transient" }` (an `open` throws, a reference is unreachable) gets the + automatic backoff; `{ kind: "broken" }` (factory, derive, document hydration or `migrate` throws; or an `open` that + throws `broken(…)` because retrying is pointless, e.g. invalid config) gets no automatic retries, only reload, + `ref.retry()` or `retryOn`. Chord decides by where it failed; only an `open` can choose `broken` itself. +- **Activation waits for its dependencies:** an object whose references are not live yet stays `loading` and goes live + as soon as they are (bounded by `readyTimeout`, then it fails `transient` with the dependency as cause); ~~a `broken` + dependency fails it at once. No retained references while failed.~~ **Superseded** by R18 (dependency failures are + `transient`, references are kept as an automatic retry trigger). +- **Timeouts** are host defaults with per-thing overrides: `closeTimeout` 30 s (per native), call drain on teardown + = `closeTimeout` (per type), `readyTimeout` 60 s (per `get`), park deadline 60 s (per type), routine handover 10 s + (per routine), job `finish` retry backoff 1 s to 60 s (per job), rollout install 60 s, linger per type; route + `resolve` is bounded by the app's signal. +- **Inbox path:** the notes plugin computes its own inbox path from generic start data (the data directory; host + values were removed later, start data is the core object `SessionInfo`). + Natives open sequentially in declaration order and close in reverse; a throwing `close` does not stop the others + (`host.close` rejects with all errors, background cleanup reports to `onError`). Type names `Native`, + `NativeCloseReason`. +- **`retryOn` sources** must live independently of the attempt (facet documents, registry values; host values were + removed later; + readables the host provides); a reference's member is not allowed (type error). Dependency waiting covers references. +- ~~**A facet that fails transiently in activation** retries automatically~~ (**superseded** by "Facets only declare": a facet cannot fail after `load`) with backoff, like an object; it does not wait + for a reload. +- **Park deadline per consumer:** `get(..., { parkTimeout })` overrides the host default for that reference; the + provider's per-type override applies to its own local bindings. +- **Playground:** the inbox watcher only opens and closes for now (the import comes with routines); a note-count derive + and the reader's redaction as a derive are visible to clients. + +## Telemetry + +- **Context:** (**superseded in part** by "Telemetry types and review cleanup": `Context.telemetry` is always a + `TelemetrySpan` and `withTelemetry` takes spans) Chord's `Context` gets `telemetry: TelemetryContext` as a fixed member, like `abortSignal` (never + undefined; `NOOP_TELEMETRY_CONTEXT` by default), with `withTelemetry(telemetry, context)` and + `traceIn(context, options, fn)` (child span, `fn` gets the derived context). `value()` and keys stay, forever, without + deprecation. Durable gets telemetry everywhere through the `Context` it already passes; principal and other app + values stay out of Durable's context (Durable work runs in the scheduler's contexts and resumes without a caller; + origin travels as data). In Chord v2, `Ctx` extends `Context`: `ctx.telemetry` is the current span, `ctx.trace(name, + attributes, fn)` gives `fn` a child `ctx` (principal and `get` kept), so calls inside are parented correctly. +- **One adapter per host:** (**superseded in part** by the adapter is a `TelemetryContext` with readonly `metrics`/`propagation` members, held only by hosts ("Telemetry types and review cleanup")) `createHost({ telemetry })`, the adapter implements `TelemetryContext` and optionally + `TelemetryPropagation` and `TelemetryMetrics` (no split). Default NOOP. +- **pi-telemetry additions** (pi first): `TelemetryCarrier = Record` (opaque, W3C-compatible); + `TelemetryPropagation { inject(context) → carrier | undefined; extract(carrier) → TelemetryContext }` (absent, + invalid or unknown carriers give the adapter's root, never throw; carriers are best effort: after an adapter change + only links are lost); span links (`SpanOptions.links`, required `TelemetrySpan.addLink(link)`, a link targets a + context); `NOOP_TELEMETRY_PROPAGATION`; the in-memory adapter implements propagation, links and metrics (carrier + snapshots, no forced W3C fields); conformance fixture with optional `propagation`. No `isRecording` for now (it would + break adapters, as would closure attributes). +- **Metrics** (in the same pi change): `TelemetryMetrics { counter, upDownCounter, histogram }` with `add`/`record` + (synchronous, cheap, never throw; histogram `record` takes an optional context for exemplars), `InstrumentOptions` + (description, unit), histogram boundaries, `NOOP_TELEMETRY_METRICS`, typed through `metrics` in telemetry schemas, + low-cardinality attributes only. +- **Across links:** (**superseded in part** by R5 (the carrier is a bounded field of the header schema)) `call` and `hold` frames carry the caller's carrier; the provider extracts it and runs the method + under it; the server adds a `chord.relay` span. Cancellation already travels as `cancel` frames. +- **Shared work** (an activation awaited by several getters, a worker start, the per-object line) is its own root span + with a link to its first trigger; every waiting span links to it. Work without a running trigger (routine runs, job + runs, eviction, backoff retries, background closes) is a root span, linked to its trigger where one exists. Chord + sets all of these; authors may add links through `ctx.telemetry.addLink`. +- **Tiers:** spans for calls, methods, relays, holds, activations, opens, closes, loads, revocations, resolves, routine + runs, job runs and finishes; events on the enclosing span for commits inside a method (`chord.commit` with docs, ops, + duration), retries and parking; metrics for background high-frequency work (state frames, bytes on WebSocket links, + commit durations, in-flight calls, active holds, open links, park durations, backpressure waits). Verbose per-commit + or per-frame spans are an adapter decision (filter by name); Chord, Durable and pi-ai only report. Sampling, + buffering and limits belong to the adapter. +- **No measuring for telemetry:** only sizes the code already has are reported (encoded WebSocket frames, `byteLength` + of bytes, lengths a storage computed while writing); nothing is serialized just to count it. +- **A Chord telemetry schema now** (typed span, event and metric names; used by tests and debugging). +- **Debugging:** (**superseded in part** by the SQLite telemetry store and `TelemetryStore` ("Telemetry store")) a JSONL trace adapter (bounded rotating file, batched asynchronous writes, drops when full, never + blocks) and a text renderer of one trace now; a web trace viewer in the playground (span tree, durations, hosts, + links, attributes, events; also usable standalone on a JSONL file) with the web shell. +- **Durable** (pi): (**superseded in part** by "Durable and AI spans" and "Telemetry implementation decisions" (phase spans under the stored carrier; every input submission starts its own trace)) `HarnessOptions.telemetry`; an optional `trace` carrier field on submission and task records (the + submitting call's or the parent task's carrier, rewritten with the record as other immutable fields are; JSON records, + no migration, no format bump, old records simply have no link); task runs are child spans while their trigger runs in + the same process, otherwise root spans linked to the stored carrier; Durable spans get their own schema. Persisted + Chord jobs store their start carrier the same way. +- **Own schema, no OTel GenAI (owner, 2026-10-18):** pi's AI and Durable telemetry uses its own typed schema + (`pi.ai.*`, `pi.durable.*`); the OTel GenAI conventions (development status, renames, streaming still `TODO`) are not + followed. An OTel adapter may translate pi's names to `gen_ai.*` at export. Model stream details are span events, never + per delta. The pi part of telemetry (propagation, links, metrics, typed metric schemas, `chord/context` telemetry + member, `HarnessOptions.telemetry` type) is approved as diffed in `/home/badlogic/pi-telemetry`. +- **Durable and AI spans (owner, 2026-10-18), partly open:** (**superseded in part** by pi-ai's option is `telemetry` (a span), not `telemetryContext` ("Telemetry types and review cleanup")) every span is short and in one process. Spans: + `pi.durable.submit` (admission), `pi.durable.phase` (one invocation of any task's phase; fixed name, `task.name`, + `phase`, `kind`, `attempt`, `resumed`, outcome as attributes), `pi.ai.request` (one model request). Events: + `pi.ai.first_chunk`, `pi.ai.block` (per `*_end`: type, index, chars, tool; never per delta), `pi.durable.entry`, + `pi.durable.submission`, `pi.durable.replay` (a codemode call answered from its journal). No content ever; entry ids + point to the transcript. Submissions and tasks store their creator's carrier; phase spans are children of it. Every + input submission starts its own trace, linked to its trigger (bounded traces; subagent report chains do not nest). + No hook spans for now. Metrics: `pi.ai.request.duration`, `pi.ai.first_chunk.duration`, `pi.ai.tokens`, `pi.ai.cost`, + `pi.durable.phase.duration`, `pi.durable.tasks.active` (ids never on metrics). `pi.ai.request` is emitted by pi-ai + in one wrapper around `lazyStream` in `models.ts` (both `stream` and `streamSimple`; reads the event stream, changes + nothing; deferred results get `pi.ai.poll` the same way); Durable passes `telemetryContext`. Background tasks + (e.g. compaction) are children of the creating phase like every task. A task picked up by a new process sets + `resumed=true` and stays under its stored carrier (same trace). +- **Telemetry implementation decisions (owner, 2026-10-18):** `TelemetryContext` carries optional `metrics` and + `propagation` members (every span from an adapter carries its capabilities), so one type is used everywhere + (`HarnessOptions.telemetry`, `createHost({ telemetry })`, `Context.telemetry`, pi-ai's option, renamed `telemetry`). + Entry and submission events only after a successful commit. Several queued inputs started as one run: the run takes + the first input's carrier and stores the others' as immutable creation-time links on its first phase span; a steer's + submit span links to the running run's carrier. A phase closed or cancelled without a committed state ends + `aborted`. Submit spans only on the public submit path; internal admissions inside another commit show as events. + ToolTask sets `tool.name`/`tool.call_id` on its own span. Write submissions are children of the caller; only input + submissions start traces. Polls count tokens and cost only for final results. Import budgets raised (pi-ai models + 18, Durable root 71). +- **Errors in telemetry (owner, 2026-10-18):** a failed span carries `status: "error"` with a fixed category name and the + error message truncated to 500 characters (`provider_error` for a model request, `task_faulted` for a faulted phase). + Cancelling is not an error (`stop=aborted`/`outcome=aborted` keep status ok). Durable's `onReport` stays unchanged; + every report is additionally a `pi.durable.report` event on the current span (its own short root span outside any + span) plus a `pi.durable.reports { type }` counter. (Superseded in part below: no root span for reports.) +- **Telemetry types and review cleanup (owner, 2026-10-18):** `Context.telemetry` is always a `TelemetrySpan` (default: + the NOOP span); the adapter (`TelemetryContext` with readonly `metrics`/`propagation`) is held only by hosts + (`createHost`, `HarnessOptions`) to start roots, never put into a Context; pi-ai's option is `telemetry?: + TelemetrySpan`. Code that only has a Context calls the generic `addEvent` (no typed-events helper, no casts, no + duck typing). A report is an event on the current span plus the counter and `onReport`; outside any span only the + latter two. `TaskOptions.trace?: TelemetryCarrier | "root"` (absent: inherit). No protection against broken adapters. + Telemetry never walks or reads data just to measure (checkpoint event compares the phase only). Instrument caches are + `WeakMap`s keyed by the adapter's metrics object. +- **Telemetry store (owner, 2026-10-18):** a SQLite adapter in Chord (`@earendil-works/chord/telemetry-sqlite`, reusing + `chord/sqlite`, Node and Cloudflare): settled spans one row each (events and links inline), metrics aggregated per + series per 10 s, a 1 s batched flush that also deletes rows older than the retention (7 days), a bounded queue of + settled spans (dropped and counted), no retries. A shared read interface `TelemetryStore` in pi-telemetry, modeled on + Jaeger's span reader: `findTraces({ name, attributes, errors, start window, limit, cursor })` returns complete traces + (newest trace start first, paged by an opaque cursor), `getTrace(id)`, `findMetrics({ name, window, attributes })`. + Spans carry global ids and a trace id (a root's own id), links carry trace and span id. Both the SQLite and the + in-memory adapter implement it; one renderer `renderTrace(spans)` (subpath `pi-telemetry/render`) prints links and + never follows them. Live spans, incoming-link search and duration search are left out. The store's metric read method is + `findMetrics` (`metrics` is the recording capability). Adapters trust typed inputs (no validation of malformed + caller values, no try/catch around recording); only carriers from another process are checked (keys present, + strings), else the root. + +## Review round 1 (Astra, Fable): kernel fixes and decisions + +Source: `p3-reviews/v2-review-summary.md` (K = kernel fixes, D = decisions). + +- **Variants (D11):** a member name has the same kind (state or method) in every variant that has it, and a method has + the same signature (arguments and result) in every variant; state schemas may differ per variant (redaction). + Enforced at the type level in `remote(...)` and at runtime for contracts arriving as data. K7 becomes that check. +- **Registry composition is a runtime state, not a staging check (D1; reverses "compose runs in staging").** + Contributions can be dynamic (only known once a native is open, or changing at runtime), so no conflict check can be + complete in staging. `compose(table)` is pure and runs whenever the table changes (after a commit, when a contributor + goes live or ends, when a dynamic contribution changes; contributions may be readables). The composed value is a + plain value for readers: `compose(table, { report })` always delivers a value, builds the fallback in (e.g. the + first by plugin name wins) and reports conflicts through `report` (they become owner notices and telemetry events); + readers never handle conflicts. Only if `compose` throws is the registry in an error state, and readers get an error + when they use it (like a failed reference). Staging rollback stays for code errors only (load errors, throwing factories). A contribution + is in the table exactly while its contributor is live (a facet while its plugin is loaded; an object from `live` + until it ends, fails or restarts). The HTTP router follows the same rule: a duplicate route is a conflict state, not + a rollback. +- **Code mode reload vs. a pinned Durable phase (D2):** (**superseded in part** by R16 (the old sandbox is detached and disposed of in the background, not drained in `close`)) the code mode tool entry looks up the current sandbox when + `execute()` starts (not the one it was created with), so a call pinned by a phase before a reload (e.g. waiting in + an approval hook) starts on the new sandbox; the old sandbox is still sealed and drained in `close` (running scripts + finish or hit their deadline, then it ends). +- **Natives that follow a relocation (D3):** documented pattern, no new mechanism: the native is a stable manager + (`self.open` once); a routine on the session location switches its internal resource (e.g. the watched directory) + and bumps a generation, so late events from the old location are dropped. Used for watchers, prompt resources and + OS probes in p3. +- **Jobs keep their owner alive (D4):** (**superseded in part** by R11 (revocation ends the objects whatever their jobs)) a job that is running, finishing or blocked in `finish` counts as a reference + to its object; the object ends only once all its jobs are settled. Starting a job and releasing the reference (an + HTTP route returning the job id, a tab closing) no longer aborts it. Keep-alive already counts running jobs. +- **Reloading native objects (D5):** stays as decided: native objects of a reloaded plugin restart. In practice this + costs little: the Durable harness is core (base code, restarts only with the host), terminal PTYs live in the + separate terminal host process (the server's native is only the connection), model runtimes, sandboxes and the + search index rebuild in milliseconds; only working on envs (running remote commands are killed) or radius (relayed + connections drop) hurts, and only on reloads of exactly that plugin. Keeping natives by name was rejected: natives + have no line, no transactions and no schema, so old and new code would interleave unchecked. +- **Old handles stay valid until their calls settle:** when a pure object is swapped in place, a call that started on + the old code (and its transaction) keeps working with the old handles until it finishes; only new calls get the new + handles. Matches "running calls finish on old code". +- **Core vs. plugin is decided by where the code lives:** `src/` (base code, imported statically by the start code + under a fixed name such as `@p3/core`, not switchable, a change restarts the host) or a plugin directory (found by + listing, loaded through loader D, named by its `package.json`, switchable, a change reloads it). Core is what the app + always needs and everything builds on (harness, catalog, access, session location); optional features are plugins + (code mode stays a plugin). +- **Host values are removed** (no `hostValue`, `provide`, `self.value`, no host-value versions). Start data (session + id, init, data directory) is a `local` core object with `lifetime: "host"` and constant state, served by the core + facet from the start message; platform bindings the host owns (a Durable Object's `state`/`env`) are core objects + whose `self.open` returns the binding and whose `close` does nothing. Plugins read start data through a reference, + in openers, derives, routines and methods (after activation); nothing reads it synchronously in a factory. A + long-lived native from the host is a `lifetime: "host"` object in a core facet. +- **`accepts` adapters map values (D6):** (**superseded in part** by R6 (provider-owned old schemas plus mappings)) per member, `state: { member: (newValue) => oldValue }` (pure; Chord wraps it + as a derive and diffs per binding) and `methods: { name: (oldArgs) => newArgs }` (plus an optional result mapping + back); members an old consumer does not know do not appear. Authors never build readables for adapters. +- **Revocation and work in cores (D7):** no revocation hook. A core that runs work on behalf of a principal stores the + grant (or device) as data with the work and reconciles from the grant owner's state (the sharing core's live grants): + a routine aborts jobs, withdraws questions and drops queued input whose grant no longer exists. The same pattern as + invariants across owners; it also holds after a crash. +- **Lingering and keep-alive (D8, decides open #1):** (**superseded in part** by R26 (the counts are one readable, `host.activity`)) the consumer-side linger is removed (`release({ linger })`, the + lingering mark on holds, the consumer linger cache): a closed tab closes its link, so it could never survive the + case it was made for. What remains is provider-side `serve(…, { linger })` (an object stays alive for a while after + its last reference, whoever held it; this is what keeps a just-closed conversation warm). Presence is the live + references (`host.references`). Worker keep-alive works as in p2 (the worker counts, the app's supervisor decides): + core code on the worker reports viewers (from `host.references`), active calls and running jobs (Chord's counters) + and Durable tasks (the core `Harness`), plus provider-side lingering objects; the server's supervisor stops the + worker when all are zero for the idle time (p2: 5 minutes, checked every second). Chord provides the counts, no idle + logic. +- **Validation cost (D9):** (**superseded in part** by R9 (no dev walks: one behavior everywhere) and R7 (arguments validated for every caller, inputs normalized)) document values are not walked against their schema in production. Instead each + document record stores the fingerprint of the schema it was written with (computed once per definition from the + structural schema, ignoring descriptions and titles; written at creation and with the new base after a `migrate`). + On access Chord compares the token's fingerprint with the record's: same version but a different fingerprint fails + the object `broken` ("schema changed without a version bump"). Always on, O(1). Every structural schema change needs + a version bump (an additive change gets a trivial `migrate`). Derives and readables are not validated in + production; in dev, values may be walked as a bug finder (documents at commit, state at publication). Inputs from + peers (method arguments, hold inputs, with the provider's schemas) are always validated. +- **Storage failure (D10):** Chord never ends the process. A storage error ends the store and closes the host, and + `onError` receives a tagged error (`{ kind: "storage_failed", … }`); the app's start code must handle it (log and + exit with its own code for a supervisor, or reopen). The design states that every app has to; the playground shows + how (`process.exit` with a distinct code). Fixes the server zombie (a closed host with a live listener). +- **Reading without starting a worker (D14):** (**superseded in part** by R20 (Chord stamps the caller's incarnation as `ctx.incarnation`)) plain pub/sub in an ordinary plugin, no Chord mechanism: the plugin's + worker facet records the summary locally and publishes it to the plugin's server object (a method call, from a + routine on the source); clients read the server object, which never starts a worker. The server object keeps the + latest value per session (a document if it must survive a server restart) and ignores reports from an older + worker incarnation. Session list, status, children, unread, artifact lists follow this. +- **The K list is confirmed** with these changes: + - K9: (**superseded in part** by R14 (`checkpointWhen` is required on every definition)) like Durable: storages do not validate deltas against content (the store already holds and validated the + authoritative value); a commit writes the delta; replay only when loading; checkpoints stay optional per + document, no default. + - K16: (**superseded in part** by "Flow control: an ack-clocked window" (call window N, not credits; no byte window)) no count limits that refuse; call credits as backpressure (throttle only, never an error); the byte window + applies to every frame including first snapshots; holds unlimited (protected by authorization and input + validation). + - K17: dropped (an IPC peer producing deep JSON shows up at once in development). + - K21: (**superseded in part** by "Only `tx.draft`" (`tx.get` is gone) and R12 (drafts are revoked; reads after the commit)) values are immutable by convention (Chord Delta's rule, as in Durable), documented for `tx.get`, + `handle.value`, `handle.initial()`; no copy, no freeze; the `keys()` index is fixed; the write on a read-only + transaction after a version bump is documented. + +## The state path: minimal compute and memory (after the cost review) + +Source: `p3-reviews/v2-cost-model.md` (before/after per hop, measured by Fable and Astra). Replaces the per-change +validation and copies the v2 kernel added around Delta. + +- **The model, end to end, as Delta and `replicatedState` (which Durable already uses):** (**superseded in part** by R1 and Q4 (parts encoded once at the origin per hold; relays rewrite headers; one binary format)) values are immutable + revisions with structural sharing; commits produce exact ops; subscribers on the same host get the same value and + ops by reference; derives publish through `diffRevisions` (identity short-circuit; bytes are atomic leaves; + equality never by `canonicalJson`); each frame is encoded once per link and sized from its encoded bytes; relays + read only the frame header and forward the body opaquely; receivers check a frame once by its header (`JSON.parse` + already yields strict JSON; bytes recognized in O(1)); consumers apply with `applyImmutable`. No validation of + state, no clone of snapshots or ops, no repeated walks. +- **A v2 readable is `replicatedState`:** today's `chord/src/state.ts` semantics (immutable value, ordered ops, + gapless attach with snapshot and cursor) is the one readable implementation, imported by `next/` like `../delta`, + so Durable's views and document watches are state members without conversion. +- **The store is a port of Durable's store design** (`durable/src/session/transaction.ts`, `session.ts`): one Delta + tracker per document, prepare and adopt by pointer swap, observers get the commit's ops, storages write the delta + (or a base when `checkpointWhen` says so) and check only metadata; replay only on load. +- **Only `tx.draft`:** it returns a draft of the current document, or a draft over `initial()` if the document is + absent; an unchanged draft (empty ops) commits nothing, so an absent document is never stored by being read. + `tx.get` is removed. +- **`derive` stays as it is,** documented with its footgun (a derive that builds new objects for unchanged data costs + O(value) per commit and per view) and a cookbook of efficient derives for p2's categories: select a field, look up + by key, small aggregates (just compute); filter/map a list (an identity cache per source element); redaction + (preferably separate documents per visibility, otherwise the identity cache); joins (identity cache plus the + looked-up source's revision); a window over a large list (reuse unchanged parts, as p2's chat window). No `project`, + no view algebra, no `memo` API. +- **Diffing and derived values come from Delta and today's Chord, not new code:** Delta's `diffRevisions` (tested, + identity short-circuit) gets bytes as atomic leaves (compared by identity, replaced when different), and v2 calls + it directly; `next/diff.ts` is removed. Derives reuse today's derived cells (`chord/src/cell.ts`) on top of the + readable. Delta gets a small, pure, tested helper `mapShared(f)` (an identity cache per source element: a new object + per changed element, the previous result for unchanged ones), used by derives that build a new object per source + element; the cookbook documents it alongside the patterns. +- **Same-host calls on `remote` types pass values by reference** (**superseded in part** by R9 (no freezing anywhere, no development mode)) (immutable by convention, as `local`); the dev freeze + catches mutation. (Replaces "validate and copy arguments and results also on the same host".) +- **Arguments and hold inputs are validated at the provider by the caller's trust:** (**superseded in part** by R7 (arguments validated for every caller; inputs always normalized)) whenever the originating + principal is not a host principal (guests, owners, any client, also when relayed over trusted IPC), once, with the + provider's schema; calls from server and worker code are not validated in production. Results are not validated in + production. Provider schemas are compiled lazily at their first use and cached in a `WeakMap` keyed by the schema + object; old tokens and their compiled schemas are collected by the GC after a reload. +- **Contract descriptions travel once per (link, fingerprint):** (**superseded in part** by R6 (no descriptions on the wire; strict fingerprints)) a hold sends the fingerprint; the provider asks for + the description once and caches the parsed token per link; the cache dies with the link. +- **Cost guards:** (**superseded in part** by R9 (the counters are ordinary telemetry metrics, always on; no test mode) and T7 (one encode per hold part at its origin, none at relays)) counting tests in the normal test run (in test mode Chord counts full-value walks, deep copies, + encodes, schema validations and materializations; per hop a test asserts the counts, e.g. one appended note to a + 500 KB document with 10 viewers: no walks, no copies, one encode per link, no validations) and a `vitest bench` + suite outside CI (500 KB and 1 MB values, 0/1/10 viewers, memory/SQLite, 1 MiB bytes) with results recorded + in `packages/chord/docs/next/benchmarks.md`. +- **Naming:** the pattern of a `local` object holding shared data behind per-principal views is the **shared store** + (not "core"); base code and plugins use exactly the same primitives. + +## Cuts and closed deferred points (review round 1) + +- **Cut:** the cross-relay reset coordination (`#collectReset`, `#resyncs`, `#resetRequests`): on an overflow a relay + releases the hold and re-holds, and the provider sends a fresh snapshot. The JSONL document storage (Chord ships + memory and SQLite). `MemoryDocumentStorage.prepareCommit`/`snapshot` (unused after K9). +- **Kept:** (**superseded in part** by R9 (no dev freeze)) the dev freeze (the safety net for passing values by reference). The Questions service stays a documented + pattern, not built. **Deferred:** the HTTP and channel components, middleware phases and the Durable Object profile, + until the kernel fixes and a first real proof (harness, one tool with an approval pause, a code mode reload) are + done; the playground's web shell serves its pages with plain app code until then. +- **Presence (#10):** a link drop is a departure (accurate and simple); smoothing (a phone switching networks) is the + UI's job, e.g. a debounced viewer count. Counts per person or origin are derived by the app from + `host.references` (principal, origin). Workers do not stop on a flicker because of provider-side linger (D8). +- **Optional dependencies (#2):** a rule, no mechanism: an optional dependency is a registry owned by the dependent + (ports contributes its forwards to `radius.forwards`; without ports the table is empty). No `optional` on `get`. +- **Rollout fixes (#12), built with the rollout step:** (a) a host checks its own install status before waiting on a + provider (a rejected own install fails at once with the reason); (b) installs are serialized per host; (c) a + revision also carries the code-set digest (the hash over the facet bundles' content hashes that loader D computes + anyway; no module analysis), and hosts compare the digest on reconnect, so a server restart with unchanged code + triggers no reinstall. +- **UI reload vs. leaving (#13):** "leaving" is never derived from an unmount, only from explicit navigation or + presence (a dropped reference); a component unmounted by a reload has not left. + +## Process (after review round 2) + +- **Layer by layer, as pi-durable was built:** the existing `next/` code (about 10k lines, kernel steps 1–6) is frozen as + a reference and not built on; Chord v2 is rebuilt bottom-up in small layers (data and contracts; readable and derive; + documents and store; objects; references and `ctx`; protocol and links; relay, routes, revocation; routines and + jobs; registries; reload and rollout; telemetry). Per layer: a one-page spec with data flow, cost per hop, a line + budget, footguns and what is reused, approved by the owner; a minimal build with counting and GC tests; a cleanup + loop with fresh reviewers and the owner until he is happy; only then the next layer. Target: 6–7k lines without + tests. The handoff for continuing after a compaction is `apps/p2/docs/p3-handoff.md`. + +## Review round 2: decided items + +- **R1, the message protocol and relays:** (**superseded in part** by Q4 (one binary message format on every link), R6 (no `describe` kind) and compression at the origin) a message is a header plus opaque parts, one part per hold. A part's payload + is encoded once by its origin with that hold's own Delta encoder (path interning per hold, end to end between + provider and consumer), packed with its byte segments, compressed at the origin above a threshold (`codec`). WebSocket: + `u32 header length | header JSON | part bytes`; IPC: structured clone of `{ header, parts: Uint8Array[] }`; memory: + the object. Kinds: `hold`, `release`, `call` (1 part: arguments), `cancel` (consumer to provider); `bound` (1 part: + snapshot), `state` (n parts: one per hold the commit touched on this link), `result` (1 part), `gone`, `describe` + (provider to consumer). Relays parse and rewrite only headers: map hold and call ids per link, stamp the principal on + `hold` and `call` (a principal field from a client link is always overwritten; only a trusted parent link may carry + one), group parts per destination link (each link gets only its own holds' parts); payloads are never parsed. One + `state` message per commit per link. Encoding happens when sending: while a link has credits a commit is encoded and + sent at once; under backpressure its ops are collected per hold and encoded into one part when credits return (still + atomic for the consumer). Flow control counts the parts' sizes from the header. A re-hold sends a fresh snapshot and + restarts that hold's encoder and decoder. Two holds of the same principal (two tabs) encode the same ops separately + (O(change) each). Replaces "relays forward bodies opaquely" from "The state path". + +- **R2, frame size limits (owner, 2026-10-19):** (**superseded in part** by Q2 (limits per part; `maxPayload` only as the hard cap)) configurable per link kind: the compressed (wire) size through the + WebSocket server's `maxPayload`, the decoded size through zlib's `maxOutputLength` in the one decode function, + wherever decoding happens (relays do not decode). Defaults for client WebSockets 16 MiB compressed, 64 MiB decoded; + none on IPC. An oversized message fails only its own call (`too_large`); the link stays up. +- **R4 and R5, protocol header schema (owner, 2026-10-19):** (**superseded in part** by Q1 (unknown ids are ignored; only a header failing the schema closes the link)) the message header is a TypeBox schema, a tagged union + over `kind` (`hold`, `release`, `call`, `cancel`, `bound`, `state`, `result`, `gone`); protocol types derive from + it, and headers from untrusted links are checked with the schema compiled once (trusted links such as IPC are not + checked, as for arguments). No kind allow-list: every message resolves against the receiving link's own tables; a + header that fails the schema, or an unknown kind, hold or call id, is a protocol error that closes that link. The + trace carrier is an optional bounded field (key count, string lengths) of `call` and `hold` in that schema; relays + read it from the header. Payloads stay opaque. +- **R7, inputs and arguments (owner, 2026-10-19):** (**superseded in part** by "Params instead of inputs": flat required + params, no normalization) object inputs are always normalized (defaults, unknown fields + removed, a canonical identity key), whoever the caller is, so equal inputs give one identity; inputs stay arbitrary + (nested objects, arrays, records allowed). (**Superseded in part** by Q12: closed schemas, unknown fields rejected.) + Method arguments are always stripped to the provider's schema and + validated with it (compiled once), whoever the caller is: trusted principals are not type-checked code (plugins are + bundled by esbuild without type checks, agents write JavaScript plugins). Bytes are leaves: `instanceof Uint8Array` + plus an optional length bound, never walked. Results and state are not validated (they come from the provider). + Replaces "arguments validated only for callers that are not host principals". +- **R8, migration output (owner, 2026-10-19; `initial()` part **refined** by "Simplification pass after 4a" D3: once per definition):** the result of `migrate` and of `initial()` is validated against the new + schema once, when produced (per document per version change, on load, off the commit path); a failure refuses the + access, keeps the old data and fails the object. Commits are still never validated. +- **R9, no development mode (owner, 2026-10-19):** one behavior everywhere; the host's `dev` option is removed. Values + are immutable by convention and never frozen (deep freezing was too costly in Delta), and state is never validated + (schema changes are caught by fingerprints, migration results by R8). The cost-guard counters (walks, copies, + encodes, validations, materializations) are ordinary telemetry metrics, always on; tests assert them through the + in-memory adapter. Replaces "values immutable by convention, with a dev freeze". +- **R10, revocation window (owner, 2026-10-19):** accepted. Work started on behalf of a revoked principal stops when + the reconciling routine runs; a step started in between still runs. No extra checks before tool calls or approvals; + there is always some window. +- **R11, jobs and revocation (owner, 2026-10-19):** "jobs keep their owner alive" stays for the normal end (the last + reference released: a closed tab does not abort a running job). Revocation takes precedence: revoking a principal ends + its objects at once, whatever references or jobs remain; their jobs get the abort signal and the normal + `closeTimeout`. A job runs as its object's principal. +- **R12, returning data from transactions (owner, 2026-10-19):** option B, as in Durable: drafts are revoked when the + transaction settles; transactions write and return at most plain values; data is read after the commit from the + document (`handle.value`, `handle.get`). No `tx.read`. +- **R13, existence inside transactions (owner, 2026-10-19):** as in Durable, no existence check: `tx.retire` of an + absent document is a no-op, `tx.draft(…, { tags })` creates a missing document with its tags, existence is read + outside the transaction. The R12 and R13 patterns are documented in `design.md` (section 9). +- **R14, checkpoints (owner, 2026-10-19; **refined** by "Slice plan and slice-1 review decisions": required only for persisted documents):** no default, but `checkpointWhen` is required on every document + definition, in Chord and in Durable (a pi breaking change), so forgetting it is a type error. Durable's task-scoped + documents (nested call, codemode run and journal) return false (they retire with their task); the codemode store + checkpoints after 64 deltas; p2's small settings documents checkpoint on every write, its growing ones after 64. +- **R15, old calls and migration (owner, 2026-10-19):** as in Durable: running calls keep their old handles until they + settle, except for documents whose version rose; old code touching a document migrated past its version throws, the + document stays intact, the call fails once. +- **R16, native restarts (owner, 2026-10-19):** no Chord change: a restart closes the old native, then opens the new. + Code mode's `close` detaches the old sandbox, returns at once and disposes of it when its scripts finish or after + the default `closeTimeout` (30 s, no override); its tool entry finds the current sandbox at `execute()`. Documented + as two patterns for natives: drain in the background, and swap through a registry contribution. +- **R17, harness start (owner, 2026-10-19):** (**superseded in part** by "The harness is an ordinary object") the harness is not part of the start barrier; it starts after it (the + barrier waits until every plugin loaded or failed, so the registries hold everything that will be there). No list of + required contributions and no paused state: a missing tool gives its call an error result; a missing model makes the + generation report "model unavailable" and wait until a model appears; failed plugins are reported with their error. +- **Note for p3, moving a session between instances (owner, 2026-10-19):** copying a session's database to an instance + with other plugins already works as Durable does: missing tools, sections and hooks are skipped; tasks without their + definition stay blocked (`missing_task`) until installed or aborted; documents of missing plugins stay untouched; + stored selections are never rewritten. The UI must show blocked tasks with their reason and offer to abort them (a + turn waiting on such a task hangs until aborted). +- **R18, dependency recovery (owner, 2026-10-19):** a dependency that fails (or does not become live in time) fails + the dependent `transient` with the dependency's cause inside, never `broken`. The dependent keeps its references + while failed; their state is an automatic retry trigger, so the dependent is retried the moment the dependency goes + live (backoff cancelled; it stays as a safety net). Invisible to factory authors. After `live`, a dependency leaving + `live` leaves the dependent live; its calls using the dependency get its error or park while it loads. +- **R19, flow control and large snapshots (owner, 2026-10-19):** (**superseded** by "Flow control: an ack-clocked window") snapshots are always admitted, even when larger + than the window (they consume it); overflow is decided on changes queued after them; after an overflow a re-hold + waits until the credits cover the last snapshot's size, with a minimum interval. +- **R20, incarnation fence (owner, 2026-10-19):** Chord stamps the calling host's incarnation (from the link handshake) + on calls arriving over a link: `ctx.incarnation` next to `ctx.principal`, not part of the principal or its key; a + server object ignores reports from an older worker incarnation by comparing it. Nothing is sent as data. +- **Testing the core (owner, 2026-10-19):** every mechanism decided in review round 2 gets exhaustive, individual tests + with its edge cases; every layer spec lists them (design section 28). +- **R21, who contributes to registries (owner, 2026-10-19):** only facets and `lifetime: "host"` objects, enforced by + the type of the factory's `self` (only theirs has `contribute`). Reading registries stays open to every object. +- **R23 (owner, 2026-10-19):** `constant(value)`, a readable that never changes, used for start data. +- **R22 (owner, 2026-10-19):** a host-lifetime object without input gets session-scoped objects with `ctx.get` in a + routine or method (documented pattern); a `get` with an input computed after activation only if needed. +- **R24 (owner, 2026-10-19):** the container-width cost of structural sharing is documented (one array slot copy and + one element visit per element per change); a benchmark of long arrays sets the limit; large collections are keyed + documents or bounded windows. No new mechanism. +- **R25 (owner, 2026-10-19):** rejected are registries created at runtime and entries resolved through mutable + indirection; a contributor may change its contribution at any time (a readable contribution recomposes on change), + and readers may look up the current value at use. +- **Changing contributions, `self.writable` (owner, 2026-10-19):** a contribution is fixed for its contributor's life + or a readable; Chord subscribes to readable contributions and reruns `compose` on every change; use sites follow the + composed readable or read its current value. `self.writable(initial)` is a host-local readable of any in-process + value (identity-compared, no JSON, no wire) with `set`, for contributions such as MCP tool lists. Tools, sections, + hooks, wraps and tasks reach the harness as one Durable `Extension` per plugin; skills and other lists likewise. +- **Removing a contribution (owner, 2026-10-19):** no remove call; an entry leaves when its contributor ends or a reload + stops contributing; a live contributor with nothing to offer sets its readable to the registry's empty value (no + `undefined` entries). +- **R26, drift (2026-10-19):** fixed in the text (the description frame is gone with R6; §9 no longer says values read + in a transaction are shared). Keep-alive counters: one readable `host.activity` (`{ references, calls, jobs, lingering }`; owner). +- **Q1, unknown ids (owner, 2026-10-19; review round 3):** unknown hold and call ids are ignored on both sides (races + such as `result` crossing `cancel` or `call` crossing `gone`); a consumer receiving `gone(h)` rejects its own pending + calls on `h`. Only a header that fails the schema closes the link. Replaces "an unknown hold or call id closes the + link" (R4). +- **Q2, size limits per part (owner, 2026-10-19):** limits per part from the header sizes, before decoding; an oversized + part fails only its hold or call, other parts apply; an oversized snapshot fails its hold without automatic re-hold; + the admitting link's decoded limit is stamped on relayed work and enforced where it is decoded; `maxPayload` is only + the transport's hard cap (closes the connection). Replaces R2's per-message wording. +- **Q3, re-hold after overflow (owner, 2026-10-19):** (**superseded** by "Flow control: an ack-clocked window") re-hold once the previous snapshot's debt has drained (credits at + the full window), with the minimum interval; replaces "once the credits cover the last snapshot's size" (R19), + which stalls for snapshots larger than the window. +- **Q4, one message format and window (owner, 2026-10-19):** (**superseded in part** by "Wire protocol" (codecs, not the JSON header framing) and "Flow control: an ack-clocked window" (no byte window)) every link carries the same binary message + (`u32 header length | header JSON | part bytes`, one `Uint8Array`): WebSocket as a binary frame, IPC through + `serialization: "advanced"`, memory delivered asynchronously. Encoding, framing, checks, ids, flow control and limits + are one shared protocol layer; adapters only move bytes; one byte window everywhere from `byteLength`. Replaces the + IPC frame-count window and R1's per-adapter formats ("IPC: structured clone of { header, parts }; memory: the + object"). Tests on memory links run the production path. +- **Q5, dependency cycles (owner, 2026-10-19):** rejected before activation waits: on one host by following the + waiting edges of activating objects; across hosts through the activation chain of uuidv7 instance ids carried in the + header of holds made during an activation (bounded; ignored from untrusted links). A cycle fails the activation + `broken` naming it; no references are kept; checked again after reloads. +- **Q6, Chord from Durable tools (owner, 2026-10-19):** (**superseded in part** by "Context last": `chord(fn, context)`) p3's core puts the host into the harness's base context under + a typed key; `chord(context, fn)` opens a call scope (host principal, the call's abort and span, references released + when `fn` settles) in tools, hooks and sections; no wrapping, no p3 `defineTool`. Needs good docs: the pattern, + lifetime and principal, and the Questions service as the example (tools reach users only through owner state; + clients are not addressable). Replaces "hooks open a call scope with `host.root`" (plugins never hold the host). +- **Q7, code mode on reload (owner, 2026-10-19):** R16 stands: the detached old sandbox is disposed when its scripts + finish or after 30 s at most; a reload during a longer script aborts it and that turn fails. After a reload at most + 30 s of old code keeps running. +- **Q8, registry ownership and versions (owner, 2026-10-19):** one authoritative declaration per registry, from the + defining facet (`compose`, version, `accepts`); other tokens only identify it; a reload replaces it atomically. + `accepts` stays (third-party plugins are an explicit goal of p3), owned by the defining facet, pure, following the + owner-adapter pattern of documents, object types and jobs; a contribution without a matching version or adapter is + left out and reported, not a load error; contributions are not schema-checked. Every use site of a contribution + catches throws and rejections, reports naming the contributor, and continues. Replaces "newer ones are refused at + load". +- **Q9, job dedupe and retention (owner, 2026-10-19):** dedupe covers unsettled runs only; after `finish` commits, + Chord drops the run's record and key (a persisted record in the same commit) and releases waiters; dedupe after + completion is the owner's, through the result in its own document. Chord keeps nothing beyond a run's lifetime. +- **Q10, contract evolution (owner, 2026-10-19/20):** forward evolution in code, no build step: a base written in + full with an explicit version (`remote()`/`local()`; `local` keeps its own kind with flat `members`, C1 rejected), + `evolve(prev, e => …)` always continues the history with structured edits (`state`, `addState`, `removeState`, + `args`, `result`, `method(name, args, result)`, `addMethod`, `removeMethod`); every version is materialized and + fingerprinted at declaration. Input or variant-set changes start a new history (a new declaration with an explicit + version; no older accepts). Mappings only with the provider, in the pure `checkAccepts(contract, accepts)` (no + `serve` in this layer): required at type level and at runtime for every member of the old version that changed or + was removed; removed state from the whole current variant state or `"refuse"`. Fingerprint: `hash64` over a + schema-aware data-only normal form (`NORMAL_FORM_VERSION`; union and intersection order kept; `required`/`enum` + sorted; TypeBox keyword whitelist; bytes as a `{ type: "bytes" }` data marker); history private, deep-copied and + frozen (**superseded in part** by "L1 rewrite review": nothing is frozen). Schema helpers `str num int bool nul lit oneOf union obj arr map opt bytes unknown` + `type Static`. + Implemented and reviewed (Fable, Astra: thumbs up) in `packages/chord/src/next/contracts/` (`7d2bb8ea8`). +- **Q11, absent singletons (owner, 2026-10-21):** a singleton always has a value: `handle.value` is the written value + or `initial()`, never `undefined` (an exposed singleton is typed `T`); `undefined` only for absent keyed documents + (`handle.get(key)`); singleton existence has no API. A deliberate difference from Durable's `snapshot()`. +- **Facets typed by the app's principal (owner, 2026-10-21):** base code exports `defineFacet` bound to its `P` + (`bindFacet

()`, working name) next to the principal union; every facet imports it from there; `host.load` + accepts only facets typed for the host's `P`. No runtime check. +- **`placement`, not `host` (owner, 2026-10-21):** (**superseded in part** by "Params instead of inputs") the contract field naming where an object lives is `placement` + (a typed function of the input); `input` stays (the object's key among objects of its type, part of its identity). +- **Q-B2, in-memory documents across input changes (owner, 2026-10-21):** not carried over. An input change of a + type ends its live objects on reload; their in-memory documents, flows and jobs are lost; documented ("in-memory + documents survive a reload only while the object identity is unchanged; persist what must survive"). +- **The store (owner, 2026-10-21; backends **superseded in part** by W14: no IndexedDB; direction accepted, details under review in `p3-reviews/store-proposal.md`):** + P1 recorded: a session is self-contained (its store plus its directory); export from the session alone, import + into another instance, sync to Radius and continue elsewhere; the server only mirrors (catalog) and keeps instance + data. One transactional table store per session, layered backend (SQLite, memory, IndexedDB) / store (tables, a + small query DSL, controlled writes, commit queue, seq, outbox, export/replay) / adapters: Durable's `Storage` as a + new `StoreStorage` adapter (Durable's existing storages stay unchanged), Chord's persisted documents, and plugin + tables (`self.table`), each table owned by one object like a document. Writes are only `put`/`delete` + (`deleteWhere` resolved to keys), recorded as row operations `{ table, key, v, row }` in the same backend + transaction as the rows, so a replica or an import needs no plugin code; indexes are local (**superseded** by "Index + definitions are recorded"); migrations run only + where the owning code runs and ship as ordinary puts. Lines stay per Chord object and per Durable Session; the + store has one short commit queue per store that assigns the seq; Durable's seq is the store's seq (Durable sees + gaps; it only compares seqs). Rejected on the way: derived-only plugin data, opaque plugin directories, export + hooks, plugin record logs, per-store outboxes with a global position counter, a SQLite-changeset or SQL-statement + store, raw SQL reads, an IndexedDB-only API, Chord documents inside Durable's `Storage`. Open: physical layout + (generic rows or real tables), DSL scope, migration timing, `next_id`, backend details, the ordering review + ("seq 40 after seq 50"). +- **S1, adoption inside the commit step (owner, 2026-10-21; store review):** every owner's in-memory adoption runs + synchronously inside the store's commit step, after the backend commit and before the next commit starts, so memory + always shows a complete prefix of the log. Durable's `Storage.commit(writes, adopt, context)` gains a required `adopt` + callback (pi breaking change): every storage calls it synchronously once the commit is durable and before its next + commit; the Session catches its own adoption error and fails; a rejection after adoption fails the Session without + `tx.discard()`; `writeAtClose` passes a no-op; `StoreStorage` forwards `adopt` into the store's step; Chord objects + adopt (pointer swap) the same way. Listeners stay where they are: after adoption, in each owner's continuation + (unchanged), so their order across owners may differ from the seq order while every listener reads a complete + prefix. Checked against Durable's code (memory, JSONL, SQLite, GuardedStorage, writeAtClose); the pi change and its + test run come with the store work. +- **S2, references to later commits (owner, 2026-10-21; store review):** no mechanism and no new rule. Ids are minted + inside the transaction that creates them (Durable `tx.appendEntry` → `mintId`) and reach other code only after that + commit, so a reference to a later commit only arises when code leaks data out of an uncommitted transaction, which + is already wrong today (the transaction may still fail). The guide states it once: ids and data leave a transaction + only after it commits; reference what is committed. +- **S3, commit completion (owner, 2026-10-21; store review; IndexedDB parts **superseded** by W14):** the commit step finishes only on the backend's terminal + success (SQLite: `COMMIT` returned; IndexedDB: the transaction's `complete` event); only then adoption (S1) and the + next commit run. The memory backend stages a commit and swaps it in at the end. An ambiguous outcome (an error during + `COMMIT`) stops the store (`storage_failed`, D10), as Durable's `GuardedStorage` does. After a restart a local head + below the uploaded head is normal: the driver pulls its own uploaded batches back before writing. SQLite keeps + `synchronous=NORMAL` (no extra cost: a crash loses nothing, a power loss the last commits, recovered from Radius); + IndexedDB uses `durability: "strict"` where supported. Condition (owner): SQLite must not get slower. +- **S4, consistent reads (owner, 2026-10-21; store review):** an export pauses commits (holds the commit queue) while + it reads, so the archive is the state at one head; streaming from a snapshot comes only if exports get large. Reads + that need several queries (a document from its base and revisions) run in one backend read transaction, as + Durable's SQLite storage does. Ordinary reads in object code (separate awaits) are not snapshot reads; the guide + says so; no global snapshot guarantee. +- **S5, replica apply (owner, 2026-10-21; store review; a sync-layer rule):** a replica applies a whole batch in one + backend transaction together with its applied head. Uploads may run in parallel (owner); the driver resubmits a + batch until Radius acknowledges it and trims a batch from its outbox only after its acknowledgement. Every batch + carries `from` and `to`; a replica applies a batch only when its `from` equals the replica's applied head, so it + never applies past a gap: a later batch waits until the missing one has arrived. +- **S6, compaction (owner, 2026-10-21; store review):** (**superseded** by W9: no compaction at first) covered by S5: a compacted batch is applied as a whole, so + its intermediate states are never visible; takeover happens at batch boundaries. Nothing to add. +- **S7, losing the driver role (owner, 2026-10-21; store review):** no rollback (tools ran, files changed, natives and + derived data followed the tail). When an upload is refused for an old epoch, the local copy becomes a separate + session with a new id and keeps running; commits in flight land in it; the old id's sync stops there; the new session + may sync on its own from a full snapshot; the two never sync again. The catalog shows both (the split-off copy marked + with its origin and seq). No `not_driver` error. To work out with sync: changing the id (store metadata, session + directory, catalog), links from other sessions keep the original id. +- **S9, S12, S14, S17 (owner, 2026-10-21; store review):** S9: the store persists its head with every commit; the + prototype's `Sequencer` and safe cut go away (prototype bug, nothing to decide). S12: the query DSL starts without + `join`, `groupBy` and `select`; plugins aggregate in JS over indexed scans; added when a real case needs them. S14: + tables and documents of an object are written in one transaction API, `self.tx(fn, ctx)` (no separate + `table.transaction`). S17: bulk operations are designed now, together with migrations, after the remaining S items + (full-row puts make a 10k-row rewrite about 40 MB in the outbox; revision pruning is one record per row). +- **S8, work that finishes out of order (owner, 2026-10-21; store review):** an app concern, no store or job mechanism. + §14 states that the handler calls of one routine never overlap; the guide shows two patterns: chain through the + routine (await the run's outcome, skip values that are no longer current) or drop stale results in `finish` by + comparing the source position inside the destination's transaction. +- **S10, ids (owner, 2026-10-21; store review):** no persisted reservations. Durable's `mintId` contract is narrowed to + "greater than every id minted since this storage was opened, and every id stored, also across reopen" (an id from a + transaction that never committed was never seen outside it, S2; Durable's transactions are serialized on the + Session line). In the store each owner's id counter starts at open above that owner's highest stored id (one index + lookup per table; a replica taking over gets it the same way). Durable's existing storages already behave so; the + comment changes in the same pi change as S1. +- **S11, copying documents into a fork (owner, 2026-10-21; store review):** `StoreStorage` reconstructs the source + documents of a `document.copy` before entering the commit step (reads of committed history outside the step) and + enters it with plain creates carrying the base; Durable is unchanged. Safe because the source is a past state and + only Durable's Session line writes Durable's tables. The outbox only ever carries the resulting rows. Forks of forks + and selecting an entry in the parent work as today (rewindable documents keep their history). +- **S13, one documents engine (owner, 2026-10-21; store review):** the store contains one documents engine used by + Durable's `StoreStorage` and directly by Chord (Chord keeps no Durable dependency). Address `(namespace, owner, kind, + key?)` with an opaque `owner` string chosen by the adapter (Durable: `session`, `conversation:7`, `task:31`; Chord: + host/object type); records per incarnation with `history: "latest" | "full"`, `createdAt`, `retiredAt`, opaque + `meta`; revisions (base with version and optional fingerprint, deltas). Reads outside the step: `find(address, at, + context)`, `load(id, at, context)` (reconstructs base plus deltas once; no cache), `list(query, limit, cursor, + context)` ordered by id or key. Writes inside the step: `create(record, base)`, `change(id, content)`, `retire(id)`, + with the engine's checks (one live per address, create before change, delta on a base of the same version, nothing + after retire); `latest` prunes revisions on a new base and on retire. Callers decide base or delta + (`checkpointWhen`) and keep their in-memory trackers as today (Durable's Session caches, pins and evicts unchanged). + Fork copies stay in `StoreStorage` (S11). Chord's frozen storage layer and its conformance suite are not ported. +- **S15, Durable's existing storages (owner, 2026-10-21; store review):** kept, as decided; no plan to remove them. + Revisit whenever it becomes worth it. +- **S16, layout and text order (owner, 2026-10-21; store review, Astra's design; **superseded in part** by W8 (every text + column), W14 (no IndexedDB) and "Text is plain UTF-8, in code point order" (the whole text order and encoding)):** real tables per logical table + (definitions and changes recorded in the outbox, the store renders DDL); keys and indexed columns of type text or + integer, the rest of a row in a JSON column (**superseded in part** by "Flat columns"). Text order is JavaScript's (UTF-16 code units) on every backend: memory + and IndexedDB use plain strings natively; SQLite stores key and indexed text columns as `BLOB`s of UTF-16 code + units, high byte first (`Buffer.from(s, "utf16le").swap16()`), so its bytewise comparison equals JS order; text that + is not indexed stays `TEXT`. Lossless (lone surrogates, NUL round-trip), no comparator, no normalization ("é" and + "e\u0301" are two keys). The SQLite backend encodes writes and query bounds and decodes results; outbox records, + exports and cursors keep logical strings. Prefix ranges come from one store helper (increment the last code unit + below `FFFF`). Every index ends with the primary key; cursors are opaque `{ dir, last }` with logical values and are + refused by a scan in the other direction; a row whose indexed field is missing or null is not in that index. Plugin + rule: text compares exactly like JavaScript strings; page with the store's cursors. Verified on Node 24.4: lossless + round trip, SQLite BLOB order equals JS sort, ~0.4 µs per encode plus decode. Cost: indexed text shows as hex in + `sqlite3` and takes two bytes per code unit. Rejected: generic JSON rows, store-maintained index entries, code point + order with `toWellFormed()` (Fable), WTF-8, text equality-only. +- **S17, bulk operations and migrations (owner, 2026-10-21; `p3-reviews/store-bulk-migrations.md`; JSON-part wording + **superseded in part** by "Flat columns"):** writes are `put` + (a whole row), `update(table, key, set)` (some columns of one row), `delete`, `updateWhere` and `deleteWhere` (many + rows, one record). Bulk operations are recorded as operations with the driver's affected-row `count`; every replica + runs the same operation (one SQL statement, or a loop in the memory backend) without plugin code; a different count + stops applying that store and restores it from a snapshot. `set` expressions touch real columns only: `value`, + `col`, `add`, `sub`, `max`, `min`, `concat`, `replacePrefix`, `coalesce`, `when`; no JSON-field operands (added + when a real case needs them); no subqueries, joins, aggregates or insert-from-query (read first, then `inArray` or + puts); no case folding, `substr` on unindexed text or floats; one conformance suite for both backends. Migrations: + shape changes of the JSON part migrate on access (each row stores its version `v`; `migrate(row, from)` on read; the + next write stores the new shape; old code refuses newer rows); changes to key or indexed columns run eagerly when + the owning object starts on the driver, before it goes live: recorded table changes (`alter` add/rename/drop, a + rebuild for key or type changes), then either an `update` step (one `updateWhere`) or a `rows` step (scan in key + order, `tx.update` per row, chunks of N rows per commit; table version `migrating` → `ready`; a crash resumes the + scan). Indexes are local and created after `ready` (**superseded in part** by "Index definitions are recorded": the + index changes after `ready` are recorded). +- **Blobs and derived data (owner, 2026-10-21):** blobs (artifact versions, uploads, sent files) are app layer: files in + the session directory named by the hash of their content, never changed, written (tmp, fsync, rename) before the row + or document that names them; any process may write one. Export copies `blobs/` with the store snapshot; import + copies them back and checks each hash. Sync: before uploading a batch, the sync layer uploads every blob file present + at that moment that has not been uploaded yet (no reference tracking; a replica never sees a hash without its file). + No blob is deleted before its session is deleted (an unreferenced blob only costs disk). Derived data (full-text + indexes, caches, a plugin's search database) is never exported or synced and is rebuilt where needed, typically by a + catch-up routine comparing it with the store. +- **Index definitions are recorded (owner, 2026-10-21):** index contents are never recorded (every backend maintains + them from the rows), but their definitions are: when the owning object starts on the driver, the store compares the + declared indexes with the existing ones and creates or drops the difference, recorded as `{ op: "index", table, name, + columns }` and `{ op: "dropIndex", table, name }` (after a migration's `ready`); every replica applies them. Every + copy of a store, an export and the file seen by outside tools is then identical and self-describing. Indexes never + affect results (queries order by index columns plus the primary key; rows without the field are excluded by rule). + Cost: replicas without the plugin maintain indexes they never use; adding an index builds it during apply. +- **P3, where persisted data lives (owner, 2026-10-21):** persisted documents and tables only in input-less `local` + objects (one per host and type); their address is (host, object type, name, key), with no input, so an input change + can never orphan persisted data. Per-entity data (per conversation, per session) is keyed documents or table rows in + such a store with an app-chosen key (data, not identity); objects with an input keep only in-memory state and read + persisted data from the input-less store by key. Not adopted (proposals of 2026-10-21): `previous`/`rehome`, the + split question, identity hashes as storage addresses. A persisted store's type and definition names stay stable; + renaming one, or changing how keys are formed, is an explicit one-off migration in the store. p2 fits: its four + stores with an input persist no Chord state. Enforced at the type level (`self.state(…, persist)` and `self.table` + only in input-less local factories) and as a declaration error at runtime. +- **Flat columns (owner, 2026-10-21):** `defineTable` takes `columns`, a flat record: each column is a scalar (`str`, + `int`, `bool`, `num`) or `json(...)`; no nested `obj`/`arr` outside `json(...)` (refused by the types and at + declaration). Every column is a real SQLite column; scalars can be filtered and ordered; keys and indexed columns are + `str` or `int` (S16); a `json(...)` column is one JSON `TEXT` column, opaque to queries, typed when read. Supersedes in + part S16 ("the rest of a row in a JSON column") and S17 ("shape changes of the JSON part"): adding, renaming or + dropping a column is a recorded `alter`; shape changes inside `json(...)` columns migrate on access. +- **P2, in-memory documents are instance state (owner, 2026-10-21):** an in-memory document is attached to the live + object: no storage, no address, no seq, never in the outbox. Same machinery as persisted documents (drafts, `self.tx`, + atomic commits, ops for subscribers and the wire); its commit is prepare and adopt. A transaction mixing both kinds + adopts the in-memory part inside the store's commit step (S1). It lives from activation to the end of the object: + survives plugin reloads (pure objects swapped in place) and native restarts of the live object, ends with the object + (last reference, linger, an input change on reload, Q-B2). Deletes the memory `DocumentStorage` for in-memory + documents and identity hashes as their keys. +- **Query runner (owner, 2026-10-21):** a query is immutable data built with `query(Table).where(…).orderBy(…)`; + `limit` is an argument, not part of the query; the same `where` serves `updateWhere`/`deleteWhere`. The table handle + from `self.table(Def)` runs it: `page(q, limit, ctx, { after })` returns `{ rows, next? }` (an opaque cursor with + direction and last key as logical values, bound to the query's order, refused with another query), `rows(q, ctx)` + iterates all pages, `get(key, ctx)`, `count(q, ctx)`. Outside a transaction it reads committed state, one query at a + time (S4). Reads inside `self.tx` (owner, 2026-10-21): `tx.page(table, q, limit, { after })` and `tx.rows(table, q)` + return committed rows, stable for the whole transaction (only the owning object writes its tables, P3, and its + transactions run one at a time on its line), without the transaction's own writes; `tx.get(table, key)` overlays + the transaction's own pending writes (put, update, delete) on the committed row (**superseded in part** by W10: after + a bulk operation on the table, `tx.get` throws `read_after_bulk`). + Documented rule: queries inside a transaction see the state before it; own writes become visible after the commit. +- **P4, plugins' own databases and files (owner, 2026-10-21):** state written by code lives in a store: the session's + store for session data, the server's store for instance data (p2's `catalog.sqlite` becomes a rebuildable mirror table, + `shares.sqlite` server tables, `subagents.sqlite` part of the session's metadata, the code-written JSON files such as + `models-store.json`, `forwards.json`, `owner-revoked.json`, `radius.json`, `device.json`, `prefs.json`, `envs.json`, + `voice.json` server documents or tables); a plugin's own SQLite file holds only derived data (`search.sqlite`); blobs + are files named by hash; files a person edits by hand (`settings.json`, `models.json`, `auth.json`) stay files + (open point #21). Logs are not data. +- **Params instead of inputs (owner, 2026-10-21):** an object type's `input` is renamed `params`: the fields that select + an instance among objects of its type (like route params), part of its identity. Params are a flat record of + required fields, each `str()`, `int()`, `bool()` or a union of string literals (`oneOf("chat", "agent")`); no + defaults, no optional, nested or unknown fields (a `get` with a missing or extra field fails `invalid/params`); the + identity key is the field values in declared order (no canonical JSON walk, no normalization). p2's 78 object types + fit (52 empty, the rest one or two required scalars). Supersedes in part R7 ("inputs stay arbitrary", "inputs are + always normalized") and the placement entry ("`input` stays"); Q12 (stripping) now concerns method arguments only. + "Input-less" in P3 and Q-A reads "with empty params". +- **Q-A, lifetime from kind and params (owner, 2026-10-21; both reviewers up):** no `lifetime` option and no lazy flag. + A `local` type with empty params lives as long as the host (started eagerly at host start, never ended by reference + count; it may hold persisted data, P3, and contribute, R21); every other `local` type and every `remote` type lives + while referenced (`linger` only for those). No remote object has host lifetime (its identity includes the + principal; long-lived parts go into a local shared store). A lazy flag was rejected for now: a lazy store would + contribute and run routines only after first access and escape the start barrier; p2's 11 such stores open only + cheap natives. Every "`lifetime: \"host\"` object" in earlier entries reads "`local` object with empty params". +- **The harness is an ordinary object (owner, 2026-10-21; Astra's C17):** no harness exception in the start barrier. + The harness is a host-lifetime object (`local`, empty params) that opens Durable during the barrier with scheduling + paused (Durable opens paused until `resume()`); base code calls `harness.resume()` after `await host.load(...)` and + before the worker connects to its parent (**superseded in part** by W6: connect first), so no request arrives earlier and every registry is filled. Chord knows + nothing about Durable's start. +- **Q12, closed argument schemas (owner, 2026-10-21; review round 3):** every object schema in a contract is closed + (`obj()` sets `additionalProperties: false`); an unknown argument field fails the call with `invalid/args` instead of + being stripped; the provider validates the caller's object in place, never copies or mutates it; arguments have no + defaults (optional fields are `opt()`, handlers write `args.limit ?? 50`). Supersedes in part R7 ("arguments are + always stripped") and the earlier "undeclared argument fields are stripped" (step 2); `checkArgs` is one compiled + validation. +- **Q15, `local` arguments are validated (owner, 2026-10-21; review round 3):** one rule for every method call: the + provider validates arguments against its closed schema (Q12) for `local` types too (callers are often other plugins, + bundled without type checks; `local` stores hold persisted data, P3). Never copied. Supersedes the `local` column + "neither validated nor copied" (design §5) and the `local` part of "Object kinds". +- **Q13, origin incarnation on relayed calls (owner, 2026-10-21; review round 3):** relays stamp the origin host's + incarnation into the `call`/`hold` header, taken from the link the call came in on and overwriting any value a client + sent (like the principal), so `ctx.incarnation` always names the host where the call started. One header field. + Refines R20. +- **Q14, exposable state (owner, 2026-10-21; review round 3):** a state member of a contract may be backed only by a + JSON readable: a document handle (persisted or in memory), a derive, `constant(...)` or a state readable + (`replicatedState`, e.g. Durable's views). `self.writable` (host-local, any in-process value, for registry + contributions) is refused as a state member: by type, and when the object is served (a kind tag check, no value + walk; the object fails `broken` naming the member). Values are still not validated (R9). +- **`reset` instead of release and re-hold (owner, 2026-10-21):** (**superseded in part** by "Flow control: an ack-clocked window": consumers never fall behind a window; only a relay sends `reset { hold }` upstream, for a downstream that fell behind; no debt or minimum interval) when a hold falls behind (the link's window + overflows), the consumer or relay sends `reset(hold)` on the existing hold, never a release: the reference count is + unchanged, so a slow connection can never end an object. The provider answers with `bound` (a fresh snapshot) and + restarts that hold's encoder; the consumer drops everything for the hold that arrived before the new `bound` + (per-direction order). Q3's rules stay (one snapshot in flight per hold, after the previous debt drained, minimum + interval). A link drop remains a departure. Supersedes "releases the hold and re-holds" (R19, design §3). +- **Routines get the changes since their last run (owner, 2026-10-21):** (**superseded** by "Watchable and one watch API": per-commit frames) `self.routines.on(readable, (value, changes, + ctx) => …)` runs one call at a time with the latest value; `changes` is `{ kind: "ops", ops }` (every op since the + previous run, in order, the commits' op arrays kept by reference, never re-diffed) or `{ kind: "reset" }` (the first + run, or more than N ops piled up). Same rule as Durable's watch overflow and the link's `reset`, applied per run; + routines that only want the value ignore `changes`. +- **Watchable and one watch API (owner, 2026-10-21; **superseded in part** by W1 (flush after the step), W2b (routines + start with the current value), W4 (two kinds) and "One pending queue for slow consumers of state" (the 100-frame + overflow)):** v2 has no `replicatedState`. Everything observable is a + `Watchable` = `{ value; watch(ctx): Promise> }` with `T` limited to JSON plus bytes at the type + level: document handles (singletons; keyed per key), derives, reference members, `constant(v)`, and Durable's + conversation view, task graph and documents. Durable's `WatchHandle` and `WatchEnd` move into Chord as the one + observation API and Durable imports them: `watch()` atomically takes the current value and buffers later commits; + `start(listener)` installs the one listener; each commit is delivered as `(value, ops, ctx)`, in order, async + listeners awaited; after 100 pending frames the backlog becomes one root replacement `[["r", newest]]`; a retired + document ends with `[["r", null]]`; `stop()` is idempotent, drops pending frames, never aborts or awaits a running + call, and returns the end; `closed` resolves to the same. `WatchEnd` reasons are generic: `stopped`, `cancelled`, + `retired`, `ended` (source closed: object ended, reference released, Session closed), `listener_error { error }`, + `failed { error }` (source failed); Durable adopts them (was `session_closed`, `session_failed`). `routines.on` is a + watch owned by the object (per-commit frames; Chord awaits a running call on reload), replacing "Routines get the + changes since their last run". Durable's state getters (`documentState`, `viewState`, `taskGraph()`) are removed + (unused by p2). Derives and the wire encoder use an internal synchronous hook inside the commit step so a derive's + change travels in the same wire batch as its source (no torn state); this needs exhaustive tests. Memory: each watch + holds at most 100 frames by reference; every watch ends (object end, reload, ctx, source end); GC tests count + retained revisions. +- **W4, two watchable kinds (owner, 2026-10-21; both reviewers agree; review round 4; JSON-plus-bytes **superseded in part** by "L2 spec decisions": watchables are JSON only):** Delta's `Op` is typed over + JSON, so natives never ride in ops. `Watchable` (JSON plus bytes): `value` + `watch(ctx)` → + `WatchHandle`, listener `(value, ops, ctx)`, overflow → one `[["r", newest]]` frame; documents, derives, reference + members, `constant`, Durable's views; the only kind that can be a contract state member or go on the wire. + `NativeWatchable` (any in-process value): `value` + `watch(ctx)` → `NativeWatchHandle`, listener `(value, + ctx)`, a change is a new value by identity (`!==`; mutating an existing native value notifies nobody), overflow + collapses to the newest value, never ops; `self.writable(initial)` (with `set`) and composed registry values. Both + share one lifecycle implementation (`watch`, `start`, `stop`, `closed`, `WatchEnd`) and carry explicit runtime kind + tags; a `NativeWatchable` is rejected as contract state even when its current value happens to be JSON. Derives may + read either kind (mixed sources) and always return JSON plus bytes: a native change marks the derive dirty without + ops; the derive recomputes and its result is diffed as usual. Native changes use the same synchronous dirty-marking + hook (W1) with the same reentrancy flag; `writable.set` never goes through the commit queue and has no seq; + registries recompose in the owner's flush, never inside another owner's commit step. Supersedes the JSON limit in + "Watchable and one watch API" and the earlier `Writable` interface. +- **W1, adopt and mark in the step, one synchronous flush after it (owner, 2026-10-21; review round 4):** the commit + step writes, adopts (pointer swap of every changed value: the owner's documents; Durable's views and documents + touched by the commit, dispatched by conversation/document id) and marks (dependent derives dirty; ops appended by + reference to each touched hold's batch and each watch's queue), O(1) per dependent. Right after the step, before any + other code runs and without any await, the owner's flush recomputes the dirty derives that someone holds or watches, + in dependency order (derives are synchronous pure functions), encodes one part per touched hold (a hold covers all + members of the held object's variant) into one state message per link, and queues watch callbacks. A derive that + throws fails only its member (`broken`); the commit and its sources' parts still go out. A dirty derive nobody + flushes recomputes lazily on its next read. During adopt and flush, `self.tx`, `watch` and `release` throw + `reentrant`; a derive cycle throws once. The same adopt/mark/flush boundary runs without a store seq for in-memory + commits (P2), `writable.set`, registry changes and inbound wire batches. Durable calls the adopt synchronously + inside its commit step (no microtask). Async work belongs in routines writing in-memory documents. Supersedes + "derives and the wire encoder run inside the commit step" ("Watchable and one watch API"). Exhaustive tests: a + throwing derive, a chain A→B→C where B throws, two owners alternating with a derive over both, reentrancy, a watch + started after 150 frames, stop during a running call, a retire with a derive over it, a reload during a routine call, + GC counts of retained revisions. +- **W2, W3, watch fixes (owner, 2026-10-21; review round 4):** W2a: retirement is an explicit source event (last frame + `[["r", null]]`, then the end `retired`), never inferred from `value === null`; in Chord and in Durable alike (a pi + change; Durable's `CommittedWatch` stops treating `null` as retirement). W2b: only `routines.on` delivers the + acquisition value: it calls its handler once with `watch.value` (ops `[["r", value]]`), then starts the watch and + drains the buffered commits; a direct `watch()`/`start()` user keeps Durable's pattern (use `watch.value` first, then + `start`), unchanged. W3: termination clears every reference a watch holds (listener, detach/report/replace closures, + cancellation hooks); a running call keeps only what it needs; GC tests cover stopped but still referenced handles. +- **W6, worker start order (owner, 2026-10-21; review round 4):** connect to the parent first, then `load`, then + `harness.resume()`. A link carries nothing until a hold or call; a host-lifetime store that references a server type + in its factory no longer waits 60 s for a missing link; requests arriving during `load` park like any call to a + loading object; no turn starts before every plugin loaded. +- **W7, store addresses carry no host instance (owner, 2026-10-21; review round 4):** automatic: Chord builds the owner + string of persisted documents and tables from the host kind and object type (`session/notes.store`), never from a host + address (no session id), so a split (S7) or an import under a new id keeps every document reachable; each session has + its own store, so the session is the store, not part of each address. Durable's owner strings (`session`, + `conversation:7`, `task:31`) already qualify. Plugin code never writes addresses; the guide says keys need no session + id. +- ~~**W8, every text column as UTF-16 BLOB (owner, 2026-10-21; review round 4):**~~ (**superseded** by "Text is plain + UTF-8, in code point order") every `str()` column, indexed or not, + is stored as a UTF-16 `BLOB` (high byte first), so text has one order everywhere (JavaScript's), adding an index never + changes results, and lone surrogates and NUL survive; `json(...)` columns stay JSON `TEXT`. The SQL generator casts + `concat`/`replacePrefix` results back to `BLOB` and uses only even byte offsets (whole code units); conformance tests + cover emoji prefixes, NUL and lone surrogates. Cost: twice the bytes for ASCII text, hex in `sqlite3`, ~0.4 µs per + encode. Supersedes S16's "only key and indexed text". Revisit note (owner): Armin is not happy with the UTF-16 + encoding; no better option is known yet. +- **W9, no compaction at first (owner, 2026-10-21; review round 4):** batches carry the ordered operations exactly as + committed; a replica applies them in order, as a whole, in one transaction (S5). "Keep the last write per row" is + dropped (it loses rows with `update` and bulk operations). A later compaction must merge `update`s into the preceding + `put` and treat bulk operations as barriers. Supersedes S6's compaction note and the book's compaction text. +- **W10, no reads after a bulk operation (owner, 2026-10-21; review round 4):** once a transaction has an + `updateWhere` or `deleteWhere` on a table, `tx.get` on that table throws `read_after_bulk` (no JS evaluator of `where` + and `set` on any backend); other tables and reads before the bulk operation are unaffected. Like Durable's + `ReadAfterWrite` (spec §4: no table read after the first table write), but narrower. The overlay for `put`, + `update` and `delete` stays (owner): helpers doing read-modify-write compose within one transaction. Durable keeps its + stricter rule (point reads could get the same overlay, scans could not; not needed today). +- **W11, two version levels for migrations (owner, 2026-10-21; review round 4):** each `json(...)` column has its own + version and a value-only migration (`json(Schema, { version, migrate: (old, from) => new })`), stored with its value; + it migrates on access, and a write of that column stores the new shape and its version (writes of other columns never + touch it). The row's `v` means only "eager steps applied up to v": only an eager step and a full-row `put` set it; + `update`, `updateWhere` and `deleteWhere` never do. An eager `rows` step applies its row function to rows with + `v < target` in key order with a cursor, writes the changed columns plus `v`, one commit per chunk; after a crash it + scans from the first key again, skipping rows with `v >= target` (one pass, no temporary index, no progress record). + No ordinary writes happen between chunks (the owner is not live until `ready`); old code refuses a newer row `v` or a + newer JSON column version independently; replicas apply the shipped writes and alters without code; a new index is + created after `ready`. Key changes are a recorded `rebuild`. Supersedes in part S17 ("each row stores its version `v`; + `migrate(row, from)` on read") and "Flat columns" (shape changes inside `json(...)`). +- **W12, W13 (owner, 2026-10-21; review round 4):** an in-memory commit (P2) takes the owner's next publication + number, never a store seq, never enters the commit queue, and runs adopt/mark/flush (W1) when it happens. Durable's id + counter at open is the maximum over all five of its tables (S10). One message per link per commit comes from the W1 + flush, also when one Durable commit touches several holds on a link. `count(q)` stays in the query runner although + S12 starts without aggregates. +- **W14, cuts (owner, 2026-10-21; review round 4):** (a) no IndexedDB backend: the browser runs SQLite in WebAssembly + (official build 3.54.1: `sqlite3.wasm` 879 KB, ~406 KB gzipped), persisting into IndexedDB (a storage adapter such as + wa-sqlite's) or the browser's private file system; serverless targets use SQLite where it exists (Durable Objects, + Turso, D1) and a Postgres backend (a second SQL generator for the query language) when a real target needs it; a + backend needs atomic transactions, ordered indexes, bytewise binary keys and one writer per session. FoundationDB was + considered (Armin): the right model (ordered key-value with transactions) but the wrong shape (an operated cluster, + no embedding, 5 s / 10 MB transaction limits, no self-contained session files); at most a later backend for a hosted + multi-tenant service. (b) The `replicatedState` family (`replicatedState()`, `ReplicatedStateSource`, + `AttachedReplicatedState`, the microtask scheduling in Durable's source attachment) is removed as public API in Chord + and Durable. (c) `stopped` and `cancelled` stay two end reasons. (d) Derive `previous` is removed (`mapShared` and + lazy recompute cover it). (e) The per-row `rows` migration step is the explicit exception (done by W11). +- **No document tags (owner, 2026-10-21; Fable's consistency check, Astra earlier):** `defineTag`, the `tags` option of + `tx.draft` and `retireByTag` are removed. They existed only to delete a session's data; with one store per session, + deleting the session deletes its store and directory, and the documents engine (S13) has no tag index. Instance data + on the server that refers to a session is a table with a `session` column, cleaned up by its owner (`deleteWhere`, + reconciling from the catalog, "invariants across owners"). +- **Principals stay unversioned (owner, 2026-10-22; L1 part 4):** the app's `P` is a plain base-code type with no + schema or version: Chord never persists principals, base code changes every host at once, links stamp principals. + Objects persist a projection with its own schema, never the principal itself (p2: chat's `Author { kind, id, name }` + from `authorOf(principal, ownerName)`, stored in every input entry); the guide states the rule. +- **Direct mappings stay; params checked by TypeBox (owner, 2026-10-22; L1 build):** `accepts` maps each accepted old + version directly to the current one (no chains in Chord; one argument mapping per call and one state projection per + commit per old version); when a new version arrives, the author updates every still-accepted version's mappings, and + the guide shows composing step functions (`(old) => v2to3(v1to2(old))`). Params are checked with a compiled TypeBox + schema (a closed object), so constraints such as `str({ pattern })` or `int({ minimum: 0 })` are allowed and + enforced; the type level and the declaration check allow only scalar helpers (`str`, `int`, `bool`, `oneOf`) without + nesting. Refines L1 part 2 ("field by field, no TypeBox"). +- **`bindProvider` (owner, 2026-10-22; L1 build):** `serve` (L4) calls `bindProvider(Type, accepts)` once per load and + reload: it runs `checkAccepts` once and keeps the validated mappings (frozen); per call `checkArgs(provider, method, + raw, boundVersion)` checks with the bound version's schema and maps directly to the current version. Condition + (owner): no leak. The binding is held only by the served type's registration and by calls in flight; no global + strong cache keyed by contracts or bindings (compiled schemas only in `WeakMap`s keyed by schema objects); a GC test + shows a binding, its mappings and compiled schemas are collected once dropped (reload) and the last call settled. +- **L1 review round 1 decisions (owner, 2026-10-22):** same-host arguments are not checked for plain data (no + prototype, getter or `unknown()`-subtree checks): the documented convention is that callers pass plain data (JSON + plus bytes), immutable by convention. Params are never copied or frozen: the object keeps the caller's params + object, immutable by convention (no freezing anywhere). A method without arguments takes `null`: dispatch passes + `null`, the README says so. Cuts (owner): the error reason type is a closed union of the decided reasons (later layers + add theirs; plugin errors never use Chord reasons, shape decided in L4); `failure.ts` and its activation helpers stay; + the `copyJson` wrapper goes; `Provider` keeps only its runtime brand; `revocationKeys` stays (**superseded** by "Simplification pass after 4a": cut until 4b). +- **L1 review round 2 decisions (owner, 2026-10-22):** error reasons are part of the link protocol: adding a reason + bumps Chord's protocol version (hosts with different versions refuse each other, an old tab reloads), so a receiver + keeps rejecting unknown reasons; a test pins the reason list with the protocol version. Failed checks take TypeBox's + public error list (the internal first-error wrapper is removed). The provider's dispatch table is private (a + provider-keyed `WeakMap`). Accepted versions' argument validators compile on first use. The closed-object argument + check runs once per declared or edited method. Owner rule: no runtime checks of values TypeScript already guarantees + (AGENTS.md); declaration checks of decided rules the types cannot express stay. +- **Cost counters (owner, 2026-10-22):** the function that does the work counts it through its context (validation, + compilation, canonical JSON, a future counted `copyJson` with an optional context added in pi); call sites never + count work done elsewhere. Counters stay plain totals without site labels: mainly a test guard (a test isolates one + operation), plus a rough production number; attribution, if ever needed, comes from an event on the context's span. + The error rebuild from the wire does not copy (fresh decoder output) and checks only the envelope. +- **Code style and L1 cuts (owner, 2026-10-22):** all Chord v2 code follows the section "Chord v2 Code Style" of the + root AGENTS.md (one concept per file, top-down order, doc comments, no flag parameters, no typed-value checks, weak + caches with GC tests), so every agent has it in context. Cuts: the params descriptor pre-check is removed (params and + arguments follow one rule: the compiled closed schema; wire params are parsed JSON, same-host callers pass plain data + by convention); `bindProvider` keeps the app's accepts table as is (immutable by convention), no copy or freeze. +- **L1 rewrite questions (owner, 2026-10-22):** `checkArgs` returns a tagged value, `{ kind: "method"; args; result? }` + (args in the current shape, `result` maps back for an old-version caller) or `{ kind: "call"; args; call }` (the + checked old args and the provider's replacement, invoked by L4); no second lookup function. Failed-check paths are + TypeBox's instance path unchanged; keys containing `/` or `~` make it ambiguous (a diagnostic; not worth recovering). + The JSON Schema `oneOf` keyword is rejected (the `oneOf()` helper emits `anyOf`); local accepts tables and + `Implementation` are flat. +- **L1 rewrite review (owner, 2026-10-22):** nothing in L1 is frozen: contract handles, schema snapshots, params + schemas and providers are Chord-owned data, immutable by convention like everything else; the snapshot copy stays + (it keeps validation in sync with the fingerprint when an app changes its schema objects). Supersedes "frozen" in Q10 + ("history private, deep-copied and frozen") and the frozen metadata in the round-1 and round-2 entries. Size accepted + (about 1,900 lines, 1,450 without blanks and comments; about 420 type-only). Failed checks say "unknown field" and + "missing" for those two cases, TypeBox's message otherwise. A malformed error envelope from a peer throws a plain + `Error`. `Principal` is a type alias (apps declare principals as type aliases, not interfaces). +- **One watch-handle implementation, upstream first (owner, 2026-10-22; L2 spec):** Durable's `CommittedWatch` moves + into Chord's shared `src/watch.ts` (pi change, upstream first), generalized to serve native watchables too + (`NativeWatchHandle`, listener `(value, context)`, overflow to the newest value); Durable uses it and its + `observation.ts` goes away; Chord v2's L2 builds on it. Publications, derives, sinks, `constant` and `writable` stay in + `next/` (v2 only). Durable's synchronous view-change hook (so a derive over a Durable view is marked in the commit + step) is a later pi change, with L3d. +- **L2 spec decisions (owner, 2026-10-22):** watchable state is JSON only (no bytes in documents, derives, reference + members or any watchable; bytes only in method arguments and results and chunked transfers; Delta stays JSON-only; + supersedes W4's "JSON plus bytes" for watchables). `mapShared` lives in `next/watch/` until v2 goes upstream. C10 is + rejected (`constant` and `writable` stay two kinds, Q14). One publication object per host, passed to its sources; + no module-level singleton. No CBOR: the wire stays JSON plus raw byte segments (R1, Q4), revisited only if L6 + benchmarks show encoding as a bottleneck. +- **Contexts are required (owner, 2026-10-22):** as in Durable (356 required context parameters, none defaulted; + optional only on error-reporting paths), every Chord v2 function that takes a context requires it: no + `= BACKGROUND_CONTEXT` defaults, no `context?`. A getter that cannot take one (a lazy derive read) uses its owner's + context (the publication's). Applied to L1 and L2; written into the AGENTS.md style section. L2 review: the + publication resets its phase and queues in a `finally`, so a throwing sink cannot leave it stuck; L2's size (about + 680 lines) is accepted. +- **Carry-over ledger (owner, 2026-10-22):** obligations deferred to later layers are recorded in + `packages/chord/docs/next/carry-over.md`, under the layer that must do them, by whoever defers them, in the same + change; a layer's spec includes its section; done entries are deleted. The AGENTS.md style section states the rule. +- **Vertical slices instead of layers (owner, 2026-10-22):** building strictly bottom-up made L2 grow generic machinery + for callers that did not exist yet (nodes, sinks, deliveries, "publications"), hard to follow; its code was deleted, + not kept. From now on Chord v2 is built in vertical slices: each slice is the thinnest end-to-end path through + every layer it needs (slice 1: one host, one local object with `self.state`, `self.tx`, `self.derive`, a checked + method call and a watch), so every API has a real caller. Same process per slice: spec by this session, owner + approval, an Opus 5.5 build, review, owner review, commit. L1 stays (stateless checks, committed with required + contexts and an `index.ts`/`internal.ts` split, `ff4f4456d`). Kept from L2: Chord's shared watch handle + (`src/watch.ts`) and the decisions (JSON-only state, one batch per host, no CBOR, C10 rejected). Plain names from + the design's vocabulary or common usage (a signals library's `batch`), never invented internal jargon in APIs. + Plan: `packages/chord/docs/next/slices.md`. +- **`Implementation` maps state members to watchables (owner, 2026-10-22; slice 1):** L1's `Implementation` types a + state member as `Watchable>` (the document handle, derive or reference member the factory returns), + not a getter `() => value`: a getter cannot tell Chord when a value changed (design §11). +- **Slice plan and slice-1 review decisions (owner, 2026-10-22):** slice order 1 local objects, 2 failures and + natives, 3 two hosts, 4 topology, 5 routines and jobs, 6 persistence, 7 reload and rollout, 8 registries and the + harness, 9 telemetry (routines and jobs before the store and reload). A feature map in `slices.md` assigns every + design feature to a slice; implementers and reviewers check the slice spec, the map and the design (AGENTS.md). + Slice process includes reviews by Astra and Fable. An object that ends rejects new calls, aborts its running calls' + contexts and makes its `self.tx` throw `released` (a running method can change nothing after the end); no drain + without natives (the drain, design §3, is decided with natives and C13 in slice 2). Keyed in-memory documents, + `mapShared` and `defineFacet

` stay in slice 1 (base infrastructure), each with a real use in its example. + `checkpointWhen` is required only for persisted documents (refines R14: in-memory documents are never written, so + checkpointing cannot apply). `retryOn` waits for the first independently living source (registries, slice 8); + slice 2 has `ref.retry()` and dependency recovery. +- **Slice 1 approved (owner, 2026-10-22):** cycle rejection on one host (Q5) is in slice 1; watches belong to the + reference they came through and end when it is released; at `get` the consumer's contract version and fingerprint + must equal the provider's (`contract_mismatch` otherwise); `then`, `release`, `state` and `retry` are reserved member + names, refused at declaration; cut C14 rejected (the `handle.change`/`handle.retire` shorthands are built); cut C16 (**superseded**: accepted on 2026-10-26, "Facet documents are removed") + rejected (facet documents are built: served objects may draft them in their own transaction). +- **Callers when an object ends (owner, 2026-10-22; slice-1 review):** as design §3 says, callers are rejected at + once: when an object ends, every caller of a running call gets `released` immediately; the handler keeps running in + the background under an aborted context and cannot commit (`self.tx` throws `released`). A call whose caller context + is already aborted is rejected before its handler runs. Each call runs under `withAbortSignal(objectEndingSignal, + callerContext)` (one controller per object; a caller's cancellation aborts only its own call). The frozen v2 kernel, + its tests, the playground built on it and the kernel step docs were removed (`4a54fc38b`); `Watchable` went upstream + into `src/watch.ts` (`a2fd25f56`). Facets are typed by exactly the host's principal type (invariant). +- **No derive names, no `null` for no-argument methods (owner, 2026-10-22; slice-1 review):** `self.derive(sources, + compute)` takes no name: the name existed for `previous` (removed, W14); on reload exposed derives are matched by their + member name, internal ones are rebuilt (slice 7 adds matching only if a real need appears). A method without + arguments omits them in the API: references call `ref.bump(ctx)` and implementations are `bump: async (ctx) => …`; + the shape is fixed per method by the contract; `null` remains the internal and wire form. Refines the L1 round-1 + decision "no-argument methods take `null`". +- **Slice 2 approved (owner, 2026-10-22):** natives with per-native readiness, ordered opens and reverse closes, close + reasons and a close context aborting at `closeTimeout`; failure causes by where they failed; every attempt bounded + by `readyTimeout`; backoff per identity (a map holding no object); only R18 failures keep references; gap-free + dependency handover on retries; the start barrier waits for each host-lifetime object's first outcome; `host.get`: + broken rejects at once, transient waits up to the getter's own deadline; the drain waits for handlers' own promises + before natives close. Cuts: C5 rejected (no natives in facets); C6: `broken()` kept, `retryOn` in slice 8; C7: + `linger` kept, built in slice 3; C13: two host timeouts (`closeTimeout`, `readyTimeout`), the drain uses + `closeTimeout`. `ref.state`, `ref.retry()` and `host.hold` move to slice 3; the plugin error shape is decided in + slice 3. Openers take a `Context` (not `({ signal })`). +- **Slice 3a approved (owner, 2026-10-22):** slice 3 is split into 3a (remote objects over one link) and 3b (surviving + drops: re-holds, parking, `ref.state`, `host.hold`, `ref.retry()`, `linger`, `host.root`, `state: false`). Guiding + rule for every choice: simple, fast, correct. A remote factory returns `{ variant, members }`; `ref.variant` is the + discriminant (C15 accepted: no `inVariant`/`expectVariant`; `variant` reserved). Calls are checked against the + binding's variant. A hold's state is one object with one Delta encoder; failed members travel in the header. A + release rejects the calls made through that reference, locally and over links (refines slices 1-2). `$bytes` is the + one reserved key in values crossing a link (byte leaves travel as raw segments referenced by `{"$bytes": i}`). + Plugin errors: `PluginError(code, message, details?)`, `internal` for anything else, never a stack. Headers are + checked on every link until IPC (slice 4). Hot-path slices add light benchmarks (timing, memory, wire size) recorded + in `benchmarks.md`. +- **Wire protocol (owner, 2026-10-22; slice 3a review):** the protocol works on a typed `Message` (one TypeBox + schema, `MESSAGE`, also checked after every decode); codecs are pluggable per link: `binaryCodec` (kind byte, varints, + raw ids, JSON values; about 23 B for a streamed token instead of 98) and `textCodec` (Delta-style JSON tuples, no field + names), compared by benchmarks for wire size, compute and memory. Member, method and variant names travel as indexes + into the hold's bound version; a reload that drops that version ends its holds first (slice 7). Checks by layer: the + codec's structural decode, the `MESSAGE` envelope, the bound version (ids, indexes, order), then the owners + (`checkParams`, `checkArgs`, Delta's decoder). The memory link delivers with `queueMicrotask` (no Node APIs). A + 2,000-token reply drops from about 196 KiB to about 46 KiB. Supersedes R1/Q4's JSON header framing. +- **Slice 3b approved (owner, 2026-10-22):** references survive link drops (`loading`, values readable, calls queued + with their own deadline and sent in order, sent calls fail), one owner of remote references across links re-holding + on every `connect`, a re-hold may bind another variant; `ref.state` (`loading`/`live`/`failed`/`released`) is a + `NativeWatchable` (interface added to `src/watch.ts` upstream first); `host.hold`; remote retries through two new + messages `retry { hold }` and `loading { hold }`; parked calls wait at most `readyTimeout` (C13 keeps two host + timeouts); `linger` for live objects; C8 (`state: false`) cut; `host.root` moves to slice 8. +- **Flow control: an ack-clocked window (owner, 2026-10-23):** supersedes Q4's byte window, R19/Q3 (snapshot admission + and debt), C12's credit accounting and K16's call credits. Measured by Astra and Fable (Node 24, `ws` 8.22, separate + processes, proxy at 64 KiB/s): a `bufferedAmount` cap plus latest-state coalescing still delivered state 9.4 s old + (kernel and proxy buffers hide megabytes); application acks with 1-4 messages in flight plus coalescing delivered + 0.0 s old state, heap +0.4 MB, 98% fewer messages; no control buffered 7 MB (+18.5 MB heap; a client that stops + reading: 23.4 MiB, RSS +121 MB in 15 s). IPC `send()` returns false from about 64 KB yet 623k sends cost +1,035 MB; + its callback acknowledges the write, not the handler. Rules: + - At most K=2 unacked `bound`/`state` messages per link; the consumer sends a small `ack` after applying each one. + - Ops queued meanwhile are merged per hold; a snapshot replaces them only when the merged ops exceed the member's + value size (2,000 appends: 46,024 B one by one, 26,021 B merged, 8,035 B as a snapshot; but a member with 500 KB + of history would cost 508,035 B as a snapshot). + - Calls: a call window N (default 256) announced in `hello`; the caller parks extra calls in 3b's parked-call queue, + FIFO, refusing nothing (K16 kept). Exceeding N is a protocol error, enforced by the provider. 100k calls of 10 ms: + a caller-side cap of 64 took 17.1 s, +4 MB, another client's p95 0.66 ms; unlimited ran 43,824 at once (+56.5 MB, + p95 129 ms); a provider semaphore queued 96,736 on the provider. + - Holds are unlimited by count; size limits stay a separate check. Dropped: the byte window and credit messages, + snapshot admission and debt, C12's credit accounting, an adapter `backlog()` API, `reset` from the consumer. + - **Relays ack per hop (option b):** the server acks upstream on receipt. When a downstream tab falls behind, the + server drops that tab's queued parts and sends `reset { hold }` upstream; the provider answers with a fresh + snapshot of that hold (a second `bound`). One slow tab never coarsens the other tabs of the same worker link + (option a, end-to-end acks, rejected for that). About 40 more lines than (a). + - Open for the slice-4 spec: Astra's circuit breaker (a hard transport-buffer cap plus a stall or heartbeat + deadline that closes the link). + Explainer: `packages/chord/docs/next/flow-control/` (Radius artifact 01m5n82vsbe5rtn4nm6wzaq0mp). +- **One pending queue for slow consumers of state (owner, 2026-10-23):** refines the snapshot fallback of "Flow control: + an ack-clocked window" and supersedes "Watchable and one watch API"'s "after 100 pending frames the backlog becomes + one root replacement" for state watches. Measured by Astra and Fable (`/tmp/fc2-*.md`, `/tmp/fc3-*.md`; real p2 entries plus synthetic stress + cases): size-based rules fail ("snapshot bytes + sent op bytes" is no bound: an interned path lets a value grow + faster than its ops; it drifts 1,254x under churn; exact size needs per-node bookkeeping), a time trigger punishes a + stalled stream into a large value, and a count of unfolded commits resends whole histories. Folding alone came + within about 10% of an exact oracle. Rules: + - Pending ops stay raw (never encoded per commit while waiting); `r` drops everything before it. + - **Leaf index:** a map from path key to the last pending leaf op (`a`, `s` of a primitive); a new leaf op on a + path in the map merges into that op in place (`a`+`a` joined, `s` replaces `s` or `a`, `a` after a string `s` + appends into it). Any other op (`p`, `m`, `d`, `t`, `s` of an object or array) clears the map. Correct because + leaf ops on different paths commute, except an ancestor primitive `s`, after which the descendant cannot be + written until a structural write clears the map (fuzzed: about 1.8M tracker and `diffRevisions` ops, arrays with + splices and reorders, adversarial cases; a broken index without the clear on `p` fails). Cost about 0.2-0.4 us and + 60-80 B per pending leaf path. Ops come only from Chord's tracker and `diffRevisions`: no defences against + hand-written ops (no path-key normalization). No folding of list appends (1% gain, quadratic copies) or trims. + - **Cap: 10,000 pending ops after folding;** past it only "replace with the current value" stays pending (nothing + more accumulates). Worst case about 2.2-2.7 MB per member per slow consumer. No size or text caps, no time trigger. + - Used by link hold members while the K=2 window is full (encoded once when an ack frees a slot) and by state + watches (`Watch` over `(value, ops)`): a slow listener gets one call per batch with the folded ops and the newest + value. Measured, listener taking 1 s: 438 calls and 3.39 MB of ops become 171 and 371 KB. The watch change is in + Chord's shared `src/watch.ts` (pi first); tests and `examples/04-watches` change with the contract. + - Unchanged: Durable's event stream (`watchEvents` batches are events: they keep their own queue until removed), + `NativeWatch` (newest value), relays (drop + `reset { hold }`: parts are opaque). + Animation: `packages/chord/docs/next/flow-control/fold/` (Radius 01m5nc9s24e9kscppen2fxgwye). +- **Slice 4 split and first decisions (owner, 2026-10-23):** slice 4 is split into 4a (links and relays: IPC and + WebSocket links, trusted links, parts as opaque bytes, flow control, host kinds and placement, routes, relayed holds + and calls) and 4b (revocation, cycles across hosts, `host.references`, `host.activity`, size limits, compression); + draft with structural changes and races: `packages/chord/docs/next/slice-4.md`. Decided: trusted links are an + explicit option (`trusted: true` on `accept` and in `resolve`'s result; default untrusted; stamping only from the + parent link). ~~Resets and queue overflow travel as `[["r", value]]` ops in an ordinary `state` message~~ (**superseded** by + "Slice 4a review decisions": resets are a list-`bound`; overflow stays an `r` op): a relay's + `reset { holds }` (a list) is answered with one message carrying every reset hold's members as `r` ops (snapshots + taken in one step, applied in one batch), entries flagged `reset: true` in the header for relays; a second `bound` + only for slice 7's restarted objects. ~~Relayed calls are sent round-robin per origin while the worker's call window + is full (each origin keeps its order).~~ (**superseded** by "Relay flow control per origin": a relay forwards calls at + once; each origin has its own N at the provider). The Durable conversation view's size on re-hold is a known cost until pi + #11327 lands. +- **Relay flow control per origin (owner, 2026-10-24):** supersedes per-hop acks, `reset { holds }`, relay recovery + and id renumbering ("Flow control: an ack-clocked window" relay part, "Slice 4a review decisions" reset part). The + provider keeps the window (K=2), its pending queues and its call queue per **origin**; the relay never drops, folds or + resets. The relay assigns each downstream link an origin number (never reused on an upstream link; 0 = its own + work); upstream messages keep their ids and carry `origin` (read only on the parent link); the provider keys holds, + calls, windows and acks by (origin, id); replies carry the origin. The relay keeps per downstream hold its upstream + link and its original `hold` message (routing calls and releases; re-holds after a worker restart), and per downstream + link a FIFO of the upstream link of each message in flight (each in-order `ack` goes to the right worker). A new + message "origin closed" releases an origin's holds and calls when its link closes. A tab served by W workers may have + 2W messages in flight. Measured motivation: a slow tab through the relay needed ~24 s and 3 resets to catch up after a + 1 s stall (provider folding: 0.02-0.38 s). After the rework and the review fixes: a simplification pass with a budget + of 1,200 lines of net growth for 4a (owner). +- **Docs (owner, 2026-10-24):** `design.md` becomes the one current statement of all rules, updated in the same commit + as each decision; `mechanisms.md` folds into it, `carry-over.md` into `slices.md`; the decision log stays complete as + the history and the reasons; slice specs stay as history after their slice. Nothing is dropped in the move. +- **Slice 4b: `host.activity` is a watchable (owner, 2026-10-24):** a JSON `Watchable` of `{ references, calls, jobs, + lingering }`, published at most once per microtask turn with changes (design §3's "one readable"). +- **Docs consolidation (owner, 2026-10-24):** applied as decided in "Docs": `design.md` now states every current rule + (`mechanisms.md` folded in section by section, its 17 stale points fixed, then deleted); `carry-over.md` is the + section "Carry-over" of `slices.md` (AGENTS.md updated). Settled by what slices 1-4a built, as approved: resets do + not exist (per-origin flow control); trusted links skip only the `MESSAGE` check; a parked call waits its + reference's `readyTimeout` (no per-`get` `parkTimeout`); the object backoff is per identity; `host.get(Type, params, + context, options)` takes a context and `host.hold` returns at once and never rejects; a native's `close` gets + `(value, reason, context)`; a local object's handlers act as the host's principal. Still open, listed in design + §26 (#24-#29): the timeouts of later mechanisms, `host.root`'s signature, how a migration commit reaches consumers, + compression on browser links, the circuit breaker, cuts C3/C4/C11. +- **Simplification pass after 4a (owner, 2026-10-24):** two cutters (Opus 5.5, GPT-6.1 Sol) read all of `src/next` + without a line target (a target is not meaningful); the merged small cuts, two bug fixes (relays strip a client's + principal fields on every path; a closed link delivers no queued message) and D1-D4 are applied by a fresh builder. + D1: the unreachable "binding back to pending" branch of a served hold is cut from 4a; slice 7 builds it with reload + (carry-over). D2: L1's unused `revocationKeys` is cut; slice 4b writes revocation keys with their first user. D3: + `initial()` is checked once per definition at its first value (refines R8's "per document": `initial()` takes no + input, so a wrong shape shows at the first call); a failing keyed `initial()` fails only its transaction, a + singleton's fails the activation. D4: object instance ids (Q5) come with slice 4b, their first user; one-host cycles + follow the waiting objects without ids. +- **Slice 5 spec drafted (owner, 2026-10-24):** straight from the discussion to `slice-5.md` (the walkthrough + `routines-and-jobs.md` deleted, owner); design §14 and §15 state routines and jobs as decided; jobs in memory in + slice 5, persistence in 6, reload rules in 7 (carry-over); jobs as objects (#9) rejected. `tx.start` returns the + run's id (R12), `job.outcome(id, ctx)` waits by id; `unavailable/busy` is a new error reason. + On the draft: routines start right after the object went live (same turn); `abort(id)` drops a blocked run (no + `finish`, reported); `running` carries every run's args (views filter). Process: spec reviews, then the build. + After the spec reviews (Astra, Fable, Opus, Sol): `tx.start` returns a `Run` handle (its outcome rejects if the + transaction is discarded) instead of an id; `ctx.state.change(fn)` takes no context (changes while stopping after an + abort still commit); running and finishing runs are a count, not references, and an object without holders ends + (after `linger`) when its last run settles, dropping blocked runs; `finish` may chain the same job (its own run is + excluded from the concurrency check); a `run` throwing after an abort ends `aborted`; `job_failed` reported; + ~~routines also in facets~~ (**superseded** by "Facets only declare"); `persist: { resume: "rerun" | "abandon" }` is the final shape. + After the build reviews: the builder's ten points accepted (every object end reports dropped blocked runs; their + waiters get the end's error; `accepts` needs both mappings per older version; job args without bytes; routine labels + per kind; `start` before live is `not_ready`; a run cancels a linger; design fixes; rejected handles keep their stack + until read, documented); native `on` sources also get only the newest value after a busy call; the finishing run is + excluded from dedupe too (`finish` may start the same job with the same key); the retirement frame in `on` is a + slice-6 carry-over. +- **Routines and jobs open points (owner, 2026-10-24; walkthrough `routines-and-jobs.md` and its reviews):** at the + `concurrency` limit `start` rejects; the routine handover is bounded by `closeTimeout`; a persisted job's state + document gets the deferred persistence option (as in Durable), and authors throttle frequent state changes themselves + (Chord does not coalesce them: every change is a commit and a message). Jobs keep `accepts` (owner: everything + versioned has adapters; the walkthrough's "no job adapters" was never decided): a run started on an older version + finishes on its old code, its args and result are mapped to the current version for `finish` and for `running`, its + state document migrates through its own `migrate`; without an adapter the run is `blocked` and reported until a + reload brings one. So a version change aborts nothing and discards nothing. + Crash handling for persisted jobs (owner, with all four reviewers): a restart limit as Durable's (five crash + interruptions, counted and persisted, clean closes not counted; then parked as `blocked`, cleared by `abort` or a + reload); a `finish` that keeps throwing is retried with backoff and then `blocked` after five consecutive failures, + its outcome, state and key kept, retried on a reload; how a persisted run resumes after a crash (rerun, replay-safe, + or abandoned with an aborted outcome, wording aligned with Durable) is required, with no default. +- **Routines for slice 5 (owner, 2026-10-24; C4 decided):** two kinds, declared in the factory, no names and no + `routines` namespace: `self.on(source, fn)` (first call with the current value, W2b; then one call per change, calls + never overlap, changes meanwhile folded; a running call is not aborted by a change; a throwing call is reported and + called again with the current value after a backoff of 1 s doubling to 60 s, a change during the backoff calls at + once; the source ending ends the routine) and `self.onLive(fn)` (called once when the object goes live; returning + ends it until the next reload or activation; throwing is reported and restarts it after the same backoff, reset + after a run of a minute; an abort is no error). Both start when the object is live and stop when it ends (`ctx` + aborted, then the drain); on reload all old routine calls of the object finish (bounded) before the new routines + start: a handover per object, which needs no names. Routines are idempotent reconcilers; Chord keeps no state for + them; what must survive is a document. Helpers `sleep(ms, ctx)` and `untilAborted(ctx)`; routines and handlers wait + only through calls that take `ctx`. Errors: `HostError` `routine_failed { object, routine, error }` (`routine` = + kind and position). Routines do not count in `host.activity`. Dropped: `every` (an `onLive` with `sleep`; a due + time that must survive is a document), `custom` and `run(fn)` (an `onLive` with `try`/`finally`), names, the due + time across reloads. Kept separate from jobs (different lifecycle, reload behaviour and output). Open: the + handover's bound (`closeTimeout`, whose default may drop from 30 s to 10 s, or its own). +- **Slice 6 first decisions (owner, 2026-10-26):** (1) **Durable's internal commit observers run inside `adopt`** + (pi change, upstream first, built and reviewed before the slice): the Session's `observeCommits` listeners (view + mounts, task graph, scheduler, submissions) move from `#publish`, after `Storage.commit` returned, into the `adopt` + callback, so every Durable view changes inside the storage's commit step; with `StoreStorage` Durable's `adopt` runs + inside the host's batch, and Chord follows Durable's views as sources marked in the step (W1). Reason: today a + derive over two Durable views (conversation view and task graph), fed by two watches, computes between their frames + of one Durable commit ("tool call issued, no task running"). `adopt` stays throw-free (a listener's throw is kept and + fails the Session after `Storage.commit` returned, as today); `subscribeCommits` and telemetry events stay in + `#publish`; nothing changes for users of Durable's own storages. (2) **Persisted singletons load sequentially** right + after the factory, all singletons of an object in one read transaction (S4), before its references are bound; a + storage error is `storage_failed`, a fingerprint mismatch, a throwing `migrate` or an invalid migration result is + `broken`. (3) **Durable's ids in the store:** `StoreStorage` keeps a `record_ids` table (Durable's one namespace + across its five tables); the counter at open is `coalesce(max(id), 1) + 1` over it (1 is the root conversation). (4) + **Few statements per commit:** the store keeps seq and head in memory (read at open, the head persisted with each + commit, S9) and folds the documents engine's checks into the writes (a unique `record_ids` key, a partial unique + index on the live address, `UPDATE … WHERE id = ? AND version = ?`), so a delta commit is about three statements + (revision, version, outbox); `StoreStorage` maps constraint errors to the messages Durable's conformance suite + expects. A benchmark compares `SqliteStorage` and `StoreStorage` on the same workload. Still open: the split of slice + 6, and retirement in watches (point 3 of the discussion: no `null` frame, Chord documents that retire). + **Refined (owner, 2026-10-26), after four reviews of the first build:** the first build moved only Durable's internal + observers into `adopt` and deferred host-waking work (`afterPublish`); the reviews found host-visible regressions in + the gap between `adopt` and the end of `Storage.commit` (watch listeners before `subscribeCommits`, `waitForIdle` + early, a late subscriber missing a commit, the active-task metric drifting after a rejection). Under review now, + nothing built: "publication at adoption" (the whole publication, host `subscribeCommits` listeners and telemetry + events included, inside `adopt`; no `afterPublish`; `adopt` runs no task code, so a marked run's `abort()` is + queued). **No Durable special cases in Chord's store** (owner): the store never knows or waits for Durable's + continuation. + **Decided (owner, 2026-10-26), after four checks of the proposal (`/tmp/observers-proposal-*.md`, explainer + `packages/chord/docs/next/durable-observers/`):** publication at adoption in Opus's variant. Inside `adopt`, in + HEAD's order: the trackers, the telemetry events, the queueing of one microtask that dispatches the host + `subscribeCommits` listeners (the listener list read inside that microtask), the internal observers. Microtasks run + in order, so host listeners still run before every wait continuation, abort and `onWake` of the commit (before + every watch delivery too, as first stated: **superseded**, see "Superseded (owner, 2026-10-26)" below), but after the storage's synchronous step, so never inside Chord's batch (Chord calls from them work; Chord + needs only its step and flush to be synchronous). `afterPublish` is dropped. `adopt` runs no task code (a newly + marked run's `abort()` is queued) and is fully throw-free (listeners, reports, metrics and telemetry contained; an + internal observer's error fails the Session after `Storage.commit` returned). Accepted changes from HEAD: abort + listeners fire after host listeners; a commit the storage rejects after `adopt` is published (it is durable; a + replica stays equal to storage; spec invariant 2 changes) and its waits resolve before the Session fails; with a + storage that works after `adopt` (JSONL) the publication runs before `Storage.commit` resolves; a throwing telemetry + call is reported as `host_error` with callback `"telemetry"` and no longer rejects a durable commit. Nothing in pi or + p2 depends on the old orders (checked: p2's three `subscribeCommits` users). + **After the reviews of the second build (owner, 2026-10-26):** one delivery rule for every watch: a watch delivers a + frame in a microtask queued after the frame arrived, also when its listener was busy (Chord's shared `src/watch.ts`: + after a call the drain reschedules instead of taking the next frame in place). Without it a busy watch's + continuation, queued before `adopt`, delivered a commit before its host listeners (Astra, Opus). Cost measured: + about 170 ns per back-to-back delivery with a promise, less with a plain `queueMicrotask`; only busy watches pay it. + Also: a host listener added between `adopt` and the dispatch gets the commit only when a listener existed at + `adopt` (documented, accepted); a rejection after `adopt` keeps the storage error as the failure and still reports an + observer's error. + **Superseded (owner, 2026-10-26), after the third reviews:** the delivery rule did not hold (Astra, Opus: a delivery + already queued folds frames that arrive later, so a watch can still deliver a commit before its host listeners, also + in a shared commit step chained through promises); a complete fix (a late delivery queues itself again) would stall + watchers behind a source that changes in every microtask. Decided instead (option b): **no ordering promise between + host `subscribeCommits` listeners and watch listeners** (independent observers; nothing in pi or p2 depends on their + order); waits and aborts still follow the host listeners by construction; the hop in `src/watch.ts` is reverted. +- **Durable's synchronous observers (owner, 2026-10-26):** `observeConversation(id, observer, ctx)`, + `observeTaskGraph(observer, ctx)` and `observeDoc(token, …, observer, ctx)` become Durable's public primitive: the + observer's `advance(change, ctx)` (the listener bag `{ value, ops, retired }`) runs synchronously inside `adopt`, + `end(failure)` when the Session closes or fails; attaching returns the current value and `detach`, taken atomically + on the line. `watchXXX` stays, built on it (a `Watch` is such an observer), for app code; `observeXXX` is documented + for adapters (record only: no awaits, no Session calls, no throws, quick). Chord's `durableView` (slice 6b) is a + source over it, marked inside Chord's batch in the store and grouped into one batch per Durable commit on the next + microtask outside it. + **After the reviews of its build (owner, 2026-10-26: fix everything to HEAD's level of correctness, performance and + memory; the owner delegated the details):** `onWake` is no longer ordered after host listeners (same cause as + watches); `Watch.endAfterDelivery()` exists (a task document's watch delivers its retirement, then ends `ended`), + detaching only when the watch really ends and never while a listener call runs; the `observeXXX` context covers the + attachment only; observers get `end(undefined)` when their task document's task ends; the change object a listener + gets carries no context (the context is a separate field of the watch); a watch keeps only its address, never a + loaded document; `observeXXX` rejects like `watchXXX`; Durable's file-watcher type in `/env` is renamed + `FileChange` so `WatchChange` names only the listener bag. +- **One retirement mechanism for Chord and Durable, watches follow the address (owner, 2026-10-26; option A):** + supersedes W2a's last frame `[["r", null]]` and the end reason `retired`, and Q11's silent reset as the only meaning + of retiring a singleton. A document lives at an address (Chord: definition and key; Durable: kind, scope and key); + every document, singleton or keyed, Chord or Durable, is watchable; a retirement is signalled to the watch listener + as one last call with the last good value, no ops and a `retired` flag (no `null` anywhere); the watch goes on, and a + later write delivers the new incarnation as `[["r", value]]` on the same watch. A watch ends when its owner ends + (`ended`): a Chord document's object, a Durable task document's task (the retirement call, then `ended`, in the same + commit), the Session; and by `stop`, cancellation, failure or a throwing listener. Durable's watches change from + incarnation to address (breaking; nothing in pi or p2 watches a document that retires). Leaks: none new for task + documents (they end with their task and invocation); a forgotten watch of a session or conversation document that + is retired for good stays until its taker's context, `stop` or the Session ends it (GC tests). Still to decide: the + shape of the flag, the value an absent address shows to members, derives and `.value`, and the per-key source in a + factory. + **Details decided (owner, 2026-10-26):** (1) a watch listener (and an `on` handler) takes one object before the + context, `(change, ctx)`: `{ value, ops, retired }` for JSON watches, `{ value }` for native ones (a breaking change + of the shared `WatchHandle`/`NativeWatchHandle` and of `self.on`, accepted). (2) A retirement call carries the + address's value after the retirement: `initial()` for a Chord document (as Q11 reads an absent singleton), so + `watch.value` equals the source's value for every reader; for Durable, which has no value for an absent document, + the last value with no ops. (4) Watching an absent address gives `undefined` (`notes.watch(key, ctx)` as Durable's + `watchDoc`). The wire carries a `retired` list per hold entry in `state`, like `failed`. (3) **A per-key source in a + factory:** `handle.at(key)` declares a singleton per key: loaded at activation with the object's singletons, it reads + `initial()` while the key is absent, and is a state member, a derive source and an `on` source; its watchers learn + of a retirement through the flag. Telling "absent" from "empty" is `await handle.get(key)` (`undefined`) or the + flag. Not adopted: the key as an activation dependency (fails when absent, ends `gone` on retirement). +- **Text is plain UTF-8, in code point order (owner, 2026-10-26; reviews `/tmp/text-review-{astra,fable,opus,sol}.md`):** + supersedes S16's JavaScript order and W8's UTF-16 BLOB. `str()` columns are plain UTF-8 `TEXT` (`STRICT` tables; + strings bound and read as they are), and text compares by Unicode code point on every backend (SQLite `BINARY`, + Durable Objects, sqlite-wasm; Postgres `COLLATE "C"`, MySQL `utf8mb4_0900_bin` later). Written values and every + query operand (`where` values, `inArray`, prefixes, cursors, `set` operands) of a `str()` column go through + `toWellFormed()`: a lone surrogate becomes U+FFFD (what `TextEncoder` and Node's SQLite binding do), documented; a + value containing NUL fails `invalid/args` (Node 24.4.1 truncates text at NUL on read; Postgres refuses it; content + goes into `json(...)` columns, which escape both). A leading U+FEFF stays allowed: sqlite-wasm's reader drops it, so + its backend reads values starting with `EF BB BF` as bytes and decodes them keeping it. The store exports + `compareText(a, b)`; the memory backend uses it for ordering, ranges, `max`/`min` and `when`; a prefix's upper bound is + the next code point after its last (portable); `substr` offsets count code points; a SQLite file must use the UTF-8 + encoding (checked at open). The guide states the rule: store order differs from JS `sort()` only for U+E000-U+FFFF + against characters above U+FFFF; sort store text with `compareText`, page with the store's cursors, use `prefix()`, + cut strings with `[...s].slice()` or `toWellFormed()`. Durable's strings in `StoreStorage` (`requestId`, kinds) keep + Durable's JSON encoding in indexed columns, so Durable stays lossless. Rejected: CESU-8 in `TEXT` (a read path of + special cases on best-effort SQLite behaviour; Postgres refuses it), CESU-8 in a BLOB, keeping W8. +- **Slice 6 is split (owner, 2026-10-26):** 6a, the store with Chord's persisted documents and tables (backends, + tables, query runner, writes, commit queue, outbox, documents engine, index definitions, migrations, export and + import, `self.table`, persisted `self.state`, P3, the host's store option, `storage_failed`, W12, keyed watches and + retirement); 6b, what builds on it (persisted jobs, `StoreStorage` with Durable's conformance suite, the Chord source + over Durable's views). The API points (the host option's name, how a Chord transaction enters the commit step, the + table API in `self.tx`, the Durable view source, the commit benchmark) are proposed in the 6a and 6b specs. +- **The host's store and how it is opened (owner, 2026-10-26; slice 6a point 1):** the host option is `store` + (`createHost({ …, store })`), not `storage`: it is the store itself, and in 6b Durable's `Storage` is `StoreStorage` + over the same store. One entry point, `openStore(backend, ctx)`, with `sqliteBackend(db)` over any `SqliteDatabase` + (Node through `chord/sqlite/node`, Durable Objects, sqlite-wasm later) and `memoryBackend()` for tests; the backend + interface stays internal; `importStore(backend, archive, ctx)` takes the same values. `openStore` runs the store's + own migrations, checks the file is UTF-8, and reads the head, the definitions and the id counters once. The app owns + the store's lifetime: the host never closes it (in 6b the host and the Durable Session share it); the app calls + `store.close()` after `host.close(ctx)`; on `storage_failed` the store stops, the host closes, and the app closes or + reopens. No `openNodeStore(path)` shortcut. + + ```ts + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(dir, "session.sqlite"))), ctx); + const worker = createHost

({ address: Session({ id }), principal, store, onError }); + … + await worker.close(ctx); + await store.close(); + ``` +- **A Chord transaction enters the commit step as Durable's does (owner, 2026-10-26; slice 6a point 2: "if this is + Durable's proven flow we should use it here too"; checked against Durable's `#runCommit`):** the object's line ~~(and + the facet's)~~ (facet documents removed the same day) is held from the callback until the step returned; an abort or error before the step discards + everything; then every draft is prepared, `checkpointWhen` picks base or delta per persisted document, and + `await store.commit(writes, adopt, ctx)` runs without the caller's abort signal (once admitted, a commit runs to its + end). `adopt(seq)` is one `host.batch.commit(...)` adopting the persisted documents, the in-memory documents and the + staged changes together (Chord's addition: Durable has no in-memory documents; adopting them apart would show half + a transaction, or publish an in-memory change of a commit SQLite then refused). An error before `COMMIT` returned + rejects the transaction with nothing adopted; an error during or after `COMMIT`, or a failed `adopt`, is + `storage_failed` (the commit stays published if it was adopted). A transaction with only in-memory writes never + enters the queue and commits synchronously as before (W12): no extra await or promise. + + ``` + line -> fn -> (abort/error: discard) -> prepare, checkpointWhen + -> await store.commit(writes, adopt) // queue; BEGIN; revision, version, rows, outbox, head; COMMIT + adopt(seq) = host.batch.commit(() => { persisted; in-memory; staged }) + -> release line(s) -> settle staged -> drafts revoked + ``` +- **Facet documents are removed; persisted data stays in objects (owner, 2026-10-26; slice 6a point 5; reviews + `/tmp/facet-persist-review-{astra,fable,opus,sol}.md`):** cut C16 accepted (supersedes its rejection in "Slice 1 + approved"): a facet has no documents (`FacetSelf.state` goes), so no facet line, no lock order across two lines, no + `Documents.facet` and no `Transaction.#facetLine`; a served object's transaction covers only its own object. Shared + plugin state lives in a `local` store object with empty params, reached through its methods (its own failure domain, + R18 recovers its dependents). Point 5 (persisted facet data) is moot: persisted documents and tables stay only in + `local` objects with empty params (P3 unchanged). Why: the four reviewers found no stated reason in the history (the + frozen kernel let facets persist; P3 was written for objects only), but a facet cannot fail at runtime (slice 2: a + facet setup or `initial()` that throws rejects `load`, closing the host; C5 was rejected because a plugin's always-on + parts would share one failure domain), and persisted data fails at runtime (fingerprint mismatch, throwing `migrate`, + a newer version after a rollback, no store on the host): one plugin's stale document would close the worker, or need + a second failable lifecycle for facets. In-memory facet documents had no real use either: none of p2's 46 `self.state` + calls is in a facet; their only ability, one transaction over an object's and its facet's documents, is a + cross-owner transaction, which the design rejects elsewhere (§9 "Rules across owners"). Other findings kept for a + later facet lifecycle: reads of shared tables need the writer's line; an owner string `facet:` collides with + type names. Built in slice 6a (carry-over). +- **The table API in `self.tx` (owner, 2026-10-26; slice 6a point 3):** (a) the first argument is the table handle + from `self.table(Def)`, never the definition (the handle belongs to one object, so `tx` refuses another object's + table, as it refuses its documents); (b) keys are objects with exactly the key columns, `{ path, entry }`, not tuples + (two swapped `str` columns of a tuple are silent at the type level; other or missing fields fail `invalid/args`, a + trust boundary); (c) reads are async (`await tx.get(t, key)`, `tx.page(t, q, limit, { after? })`, `tx.rows(t, q)`), + writes synchronous and buffered like a draft's ops (`tx.put(t, row)`, `tx.update(t, key, set)`, `tx.delete(t, key)`, + `tx.updateWhere(t, where, set)`, `tx.deleteWhere(t, where)`): they check the row and stage it, no I/O, no line, so a + bad row throws at its own call; the buffer goes into the commit step's backend transaction or is dropped on failure, + like aborted drafts (writing at once would hold SQLite's one write transaction for the whole callback). As Durable + (`setTask`, `settleSubmission` synchronous, reads async) and Chord's `tx.start`. (d) No write shorthands on the table + handle (`edits.put(row, ctx)`) until real code repeats the one-liner. Outside a transaction the handle reads + committed state: `edits.get(key, ctx)`, `edits.page(q, limit, ctx, { after? })`, `edits.rows(q, ctx)`, + `edits.count(q, ctx)`. `tx.get` exists for tables only (documents keep `tx.draft`). + + ```ts + self.tx(async (tx) => { + const old = await tx.get(edits, { id }); + tx.put(edits, { ...row, at: now }); + tx.updateWhere(edits, (e) => prefix(e.path, from), { path: replacePrefix(from, to) }); + }, ctx); + ``` +- **Chunks of `rows` migrations (owner, 2026-10-26; slice 6a point 6):** 100 rows per commit, a fixed constant. + Measured on slayer (`node:sqlite`, WAL, 20,000 rows of about 3 KB, a primary-key `UPDATE` plus an outbox `INSERT` + per row): the total is flat from 100 to 1000 rows per chunk (about 15 µs per row); the step's median is 0.94 ms at + 100, 2.4 ms at 250, 9 ms at 500, 14.5 ms at 1000, 32 ms at 2000, and every other owner's commit waits for it. The + store benchmark measures it again with the real store. Can change later (owner), e.g. a table definition naming its + own chunk size; not built until a case needs it. +- **`lastSeq` instead of "head" (owner, 2026-10-26):** the seq of a store's last commit is called `lastSeq`, never + "head" (Durable's "head" is a conversation's head: head markers, `findLatestHeadMarker`). Every "head" of the store + in earlier entries (S3, S4, S5, S9, "Slice 6 first decisions" (4), "The host's store and how it is opened", the + commit-step trace) reads `lastSeq`; a replica's "applied head" is its `lastSeq`. Renamed in design.md, the store + book and the slice 6a spec. +- **Export archives are operations, the encoding is pluggable (owner, 2026-10-26; slice 6a point 7):** + `store.export(ctx)` yields `AsyncIterable`: the state after its `lastSeq`, written in the outbox's + operations as one synthetic batch from 0 to `lastSeq` (a manifest `{ kind, format: 1, lastSeq, schema }`; `table`, + `index`, `put` per row; `document.create` per live record with its base, plus its later `document.change` records + for `history: "full"` ones (Durable, 6b); an `end` record with counts). Retired documents, blobs, the outbox and sync + state are not in it. `importStore(backend, records, ctx)` refuses a non-empty backend, an unknown format, records out + of order and a missing or wrong `end`, and applies everything in one backend transaction with the replica's applier + (import, bootstrap and batch apply are one code path). Export holds the commit queue until the iteration ends, + throws, is closed or `ctx` aborts (a slow consumer stalls commits; the guide: export to a local file, then upload; + snapshot reads on a second connection only if exports get large, S4). The encoding is pluggable, as the link's + codecs: the store yields and takes records only and ships plain JSONL (`encodeJsonl`, `decodeJsonl`) with no size + optimizations; gzip or a Delta-style dictionary encoder come later as formats without touching the applier (owner: + "then it doesn't matter that the first thing we do is plain JSONL"). Sync is not export: it ships outbox batches as + commits happen (S5); a full archive is needed only where no history can be replayed (a session moved as a file, a + split-off session, a replica restored after divergence). Open for the sync layer: a session that never syncs keeps + every outbox record (trimmed only after Radius acknowledges it), so its outbox grows without bound. +- **State and job names are restricted too (2026-10-27, slice 6a fix round, under the owner's delegation):** + `defineState` and `defineJob` names follow the pattern of object type and table names, `[a-z0-9@/._~-]+`, lowercase, + checked at declaration: they become the documents engine's `kind` and keys of job records, where NUL or lone + surrogates would be truncated or replaced by SQLite. +- **Slice 6a build reviews (2026-10-27, under the owner's delegation to the reviewers' majority):** two rounds of four + build reviews (`/tmp/s6a-build-review{,2}-{astra,fable,opus,sol}.md`), merged into two fix lists + (`/tmp/s6a-fix{1,2}-brief.md`, 28 and 16 items) by majority, a point raised by one reviewer taking that reviewer's + recommendation. Rules they settle beyond fixes of the spec: (1) **members across attempts:** a local object's state + member is followed by its references and by other objects' derives through one member source per object and member + name; when a new attempt goes live every member re-binds by name (one diff frame from the last value, consumers' + derives recompute, a member the new variant lacks fails `invalid/member`), as a remote reference re-holds and gets a + fresh snapshot (Opus B1); supersedes slice-6a §1.2's "derive members' watches end with their attempt". (2) While an + object that was live is failed or activating again, same-host reads and new watches return the last values and get + no frames (was `not_ready` while activating again). (3) A live failure freezes the members only after the commits + admitted before the seal adopted and published once; a retry's factory and hydration wait for that, so no old commit + lands in a new attempt. (4) Re-exposed and forwarded members freeze at the re-exposing object; the provider's source + stays untouched. (5) A transaction cut at the seal while queued for the step rejects `released` at once; line jobs + queued before the seal are refused. (6) The seq, document id and revision counters stop at + `Number.MAX_SAFE_INTEGER` (`storage_failed`). (7) `evolveTable` checks each edit when it is made; import checks + `meta` with the text rules; an export never started rejects `released` after `store.close()`. (8) The migration + yield between chunks: `setImmediate` where the runtime has it, otherwise a `MessageChannel` message, never + `setTimeout(0)` (clamped to 1 ms in Node; a message costs about 0.27 ms per yield there: a 10,000-row migration takes + 70 ms against 100 ms and 191 ms); supersedes round 1's `setTimeout(…, 0)`. (9) Round 1: keyed keys follow the + `str()` rules (the empty key valid); same-host reads of a failed live object return the last value; stuck handlers' + counts, references and presence released at teardown; refusing a new job version on a live object kept for 6a + (slice 7 carry-over); in-memory keyed slots migrate on access. Measured against HEAD (`benchmarks.md`): calls 1-3% + faster (19 promises per call against 20, guarded), watches through a reference at HEAD's level, a document member's + read through a reference 0.25 ns slower, host setup about 3% slower. +- **Derives heal on a re-bind (owner, 2026-10-27: "a all the way, self healing is good"; Opus's F2 in the third + build review):** a consumer's derive that failed on a provider member's earlier error recomputes when that member + re-binds to the provider's new attempt (and every derive downstream of it), so it recovers once its source works + again; watches over it get the recovered value as a fresh frame. Before, a failed derive ignored later changes (as at + HEAD) and recovered only when its own object restarted. +- **Failures are watch calls; derives heal on every change (owner, 2026-10-27: "B", "beautiful, let's do it"):** + extends "Derives heal on a re-bind". (1) A failed derive recomputes on every later change of its sources (and on a + re-bind), so it heals as soon as it gets a value it can compute. (2) A watch stays open through a recoverable failure: + the listener gets one tagged call `{ kind: "failed", error }`, and the next good value as `{ kind: "value", value, + ops: [["r", value]], retired }`; the call is a tagged union (`WatchCall`: `value | failed`; native watches + `NativeWatchCall`: `{ kind: "value", value } | { kind: "failed", error }`), so the compiler makes every listener + handle the failure (an optional `error` field would let `({ value }) => render(value)` render the old value). + Supersedes the listener bags `({ value, ops, retired }, ctx)` and `({ value }, ctx)` of "One retirement mechanism" + (1). (3) Applies to derives, replica members a hold lists as failed, a member a new variant lacks (it comes back with + a later attempt), registry values whose `compose` throws (slice 8). (4) A watch ends only for good: its owner ended, + `stop()` or a cancelled context, a throwing listener, `storage_failed`; Durable's watches fail only for such causes + and only take the new shape. (5) `self.on` handlers are not called for failures (a reconciler acting on an error call + would act on the old value); the routine waits for the next good value; the failure is reported where it happens: + a derive's owner reports `derive_failed { object, member?, error }` once per failure episode (the first failure after + a value or after creation; a heal ends it), a new `HostError` kind (2026-10-27, slice 6a fix round 4: there was no + report for a throwing derive besides a routine's `routine_failed` on its ended watch). One failure call per + episode (a value turning into a failure), not one per source change while failed; a hold keeps only a member's + newest pending failure. Watching a failed source (2026-10-27, slice 6a fix round 5, under the rule "the routine + waits for the next good value"): `watch(ctx)` opens at the last good value and delivers one `failed` call first; + a source that never had a good value rejects `watch()` with its error; `self.on` on a failed source waits for the + first good value instead of ending. + (6) `ref.state` stays a separate watchable of reachability (`loading | live | failed | released`; `ended` since + "One vocabulary, groups 2 and 3"), same words: a + value's watch says whether it can be computed, `ref.state` whether its object is reachable (values stay readable + while loading). No new leaks: an open watch on a failed source holds what it held before and ends with its owner, + reference or context (GC test). Built: the shared `src/watch.ts` upstream in pi first (Durable's listeners adapt), + then slice 6a (points 1-3 except registries; registries in slice 8). +- **One vocabulary, group 1: the shared watch (owner, 2026-10-27: "yes"; vocabulary audit `/tmp/vocab-audit.md`, + A1, A3, A8):** built in the same upstream change as the failure calls. (a) `WatchEnd` is tagged by `kind`, not + `reason` (`end.kind === "failed"`). (b) A watch whose context is aborted ends `aborted`, not `cancelled` (one word + for a requested end, as `AbortSignal`, the `aborted` code and `JobError.aborted`). (c) Durable's synchronous + observers take the watch's shapes: `advance(call, ctx)` with a `WatchCall`, `end(end: WatchEnd)` (Durable never + sends a failure call: its failures are final). (d) A member a new variant lacks behaves the same locally and over a + link: a failure call, the watch stays open (today local `failed`, remote `ended`). (e) Durable's `FileChange` is + tagged: `{ kind: "paths", paths } | { kind: "overflow" } | { kind: "failed", error }`. The audit also found that + "Slice 6a small items" (1) is wrong: Durable's `Storage.close(context)` takes a context (decided with group 2). +- **One vocabulary, groups 2 and 3 (owner, 2026-10-27: "lgtm"; Durable's stored formats stay the same):** + Chord v2 (built in slice 6a): (1) `ref.state` `{ kind: "ended", error }` replaces `released`; the `released` code + gets reasons `ended` (object ended), `closed` (host, store, native), `settled` (`ctx.get` after its call), + `version` (an old handle), `discarded` (a transaction cut at the seal); a holder's own release stays `released` + without a reason. (2) Internal results are `done | failed { error }`; the attempt's `Outcome` is `Activation`, + the jobs' `Ended` is `Settled`. (3) `DocumentSlot.remove` is `retire`; `StoredChange` is `create | change | + retire`. (4) The wire's `cancel` message is `abort`; a native's close reason `stop` is `close`; ~~`job.abort(id, ctx)` + returns `"marked" | "terminal"` (Durable's words)~~ (**superseded**, owner 2026-10-27: Durable's `abortTask` stores a + mark and joins the run, Chord's 6a abort only fires a signal in memory; `job.abort(id)` stays in 6a, and becomes async + and stored with persisted jobs in 6b, its result chosen there). (5) Homonyms: the store's `Staged` is `StagedWrite`, its `Phase` + `WriteOrder`, the host's `KeyedMember` `KeyedBinding`, `KeyedMembers.hold` `pin`, `Presence.origin` `incarnation`. + (6) Context last: `runState.change(fn, ctx)`, `host.route(Kind, options, ctx)`, `store.close(ctx)` (as Durable's + `Storage.close(context)`; supersedes "Slice 6a small items" (1)'s "no context"). (7) `failed` and `gone` carry the + wire error format (`WireError`). (8) `HostError`: `failure` where the value is a `FailureCause`, `error` otherwise; + positions `{ kind: "on" | "onLive" | "native", position }`. (9) Store types as Durable's: `history: "latest" | + "rewindable"`, pages `{ items, next }`, a singleton's address has no key (`""` only in the column). (10) The + listener parameter is `call`; holds `served | relayed`; `put.v` is `version`. Durable's public API (pi upstream, + with group 1): (11) `onReport(report)` with `report.kind` and `report.error`; `session_failed` is `storage_failed`. + (12) `subscribeCommits` is `observeCommits` with the observer shape (`{ value, detach }`); a file watcher stops with + `stop(ctx)`. (13) `CompactionFailure.reason` and `CompactionStatus.reason` are `trigger` (`pi.live`'s definition + bumps its version with a `migrate`). (14) ~~One word for a definition that cannot take the stored state: + `incompatible`~~ (**superseded**, owner 2026-10-27: `task_too_old` is stored in orphaned outcomes and settlements, and the + report's `incompatible_task` covered two cases): the `task_outdated` report uses the scheduler's own words, + `missing_task | task_too_old | migration_failed` (newer without `migrate`: `migration_failed`); Chord's jobs name + their blocked reasons in slice 7. + (15) `TaskInspection` uses `TaskState`'s words (`ready` is `pending`). (16) `DocumentCommitChange` is tagged + (`{ kind: "retire" }` instead of `value: null`). Not done: Durable's stored tags (`status`, `type`; group 4), Durable + throwing Chord's codes (6b), one retirement value ("One retirement mechanism" (2) stands). + Added (owner, 2026-10-27, "lgtm"): Durable's `SessionEnd` and the `ReportFrom` argument of `runtime.report()` are + tagged by `kind` too (neither is stored). +- **Slice 6b D1: `StoreStorage` and `durableView` live in Durable (owner, 2026-10-27: "they are only used if durable + is used"):** in `packages/durable/src/next/`, private-only like `packages/chord/src/next`, importing + `@earendil-works/chord/next` (Chord's `./next` export returns, private-only). No package cycle: Durable already + depends on Chord, Chord imports nothing of Durable (S13). The folder goes upstream with Chord v2. +- **The performance and memory bar, and converging reviews (owner, 2026-10-27: "yes"):** supersedes "as good as + HEAD in everything" for Chord v2 slices and pi changes. CPU: each hot path (calls, changes, watches, links) within + about 5% of HEAD, measured in bursts with alternating processes pinned to one CPU; worse is fixed or goes to the + owner with numbers. Memory: allocation per operation and memory retained after GC no worse than HEAD beyond noise, + with GC probes for everything that must be collected (ended objects, released references, closed hosts, watches on + failed sources). No super bad regressions; small differences within noise are not chased. Reviews converge: a fix + round adds no new decisions or features; the same reviewers (restored, with their context) re-review their own + findings; when only small items are left, the slice is committed and they are carried over. +- **Size is not a goal by itself (owner, 2026-10-27):** no cut pass after slice 6a: "if the user-facing API is great + and the underlying implementation fast and correct, we're ok". `src/next` is about 13.2k lines without blank lines + and comments after 6a's build (about 8.5k before), against the earlier target of 6-7k for all of Chord v2, which this + supersedes; the bar stays a great API, HEAD-level speed and memory, and correctness. +- **Second spec review: the majority (owner, 2026-10-27: "pick the majority vote for everything, get building"):** the + second review of slice 6a (`/tmp/s6a-review2-{astra,fable,opus,sol}.md`) is decided by the reviewers' majority; a + point raised by one reviewer takes that reviewer's recommendation; ties take the simpler single mechanism. Folded + into slice-6a.md (section 9 and section 15, 27 rules) and design.md. The main points: (1) **migrations by + `evolveTable`** (all four): a base `defineTable` and one `evolveTable(prev, edits)` per version, each materialized and + fingerprinted (`TABLE_FORM_VERSION`), edits `rename add drop index dropIndex json` and at most one data step (`rows` or + `update`), the editor's types ordering them, per version an (A) commit (DDL, `migrating`) and a (Z) commit (last + chunk, later edits, `ready`), recorded as `alter`/`index`/`dropIndex`/`table` writes; fresh tables created at the + newest version; shared with contracts only `hash64`, the normal form per column, `Increment` and the editor-return + check (supersedes the derived adds and drops of "The migration API: the reviewers' consensus"). (2) The store's + admission check through `store.commit(writes, { admit, adopt }, ctx)`. (3) An admitted commit's staged job changes are + adopted into the object (supersedes "The seal fence"'s "staged job changes of the sealed attempt abandoned"); a run cut + off by its attempt's seal ends `aborted { reason: "restart" }`. (4) A transaction at the seal is discarded; pending + async `tx.*` operations tracked as in Durable. (5) Key columns immutable in `set`. (6) Writes serialized before + `BEGIN`; canonical `set` vocabulary with an explicit null for clears. (7) `int` columns kept in the safe range by a SQL + `CHECK` (an out-of-range result in the step is fatal, accepted), the JS evaluator checking the same. (8) The outbox + floor (`read(afterSeq)` valid from the floor to `lastSeq`). (9) The manifest carries `nextDocumentId`. (10) A bounded + LRU for `chord/sqlite/node`'s prepared statements (pi change). (11) Slots keep `valueVersion` and `storedVersion` (as + Durable's `LoadedDocument`). (12) An admitted commit of a failing object adopts and publishes once (Sol, Fable). (13) + Retiring a `latest` record deletes its row. (14) One host per store. The rest: section 15 of the spec. Carry-over to + slice 7 recorded in slices.md. +- **Slice 6a small items (owner, 2026-10-27: "all yes"; from the four spec reviews):** (1) `store.close()` (no + context, as Durable's `Storage.close()`) waits for a running step, rejects queued commits, running exports and reads + `released`, closes the backend; idempotent, also after `storage_failed`. (2) The store tells its host of + `storage_failed` through an internal listener, once; the host reports once and closes itself. (3) An omitted outbox + option keeps the persisted setting; switching needs an explicit `outbox: true` or `false` (supersedes in part "The + outbox is optional": "opening with the other setting is that explicit switch"). (4) `store.outbox` is `undefined` + without an outbox (typed); `read(afterSeq)` before the first stored seq and `trim` beyond `lastSeq` reject; `trim` is + idempotent. (5) **No hand-written memory backend:** tests run on SQLite `:memory:` through `chord/sqlite/node`; one + semantics (`NULL`, text order, the `set` evaluator, ordering), no conformance suite across two backends; every real + target is SQLite (supersedes the memory backend of "The store", S3 and "The host's store and how it is opened"). + (6) Export takes the queue at its first `next()`, an abort listener releases it while the consumer is suspended (a + later `next()` rejects `aborted`); it reads page by page with plain reads, not one long read transaction. (7) Parts in + order: manifest; per table its `table`, `put`s, `index`es; the `document.create`s; `end`; parts after `end` refused; + `importStore(backend, parts, ctx, { outbox? })`, the outbox starting after `lastSeq`. (8) A live object failing is + reported as `object_failed { object, cause }`. (9) `keys(ctx)` is in code point order, in memory too (`compareText`). + (10) A migrated base is written back by the next commit that changes the document (supersedes design §9's "a + read-only transaction right after a version bump writes the migrated base"); until then a load re-migrates (once per + object). (11) Document ids are assigned in the step, monotonic with seq, given to `adopt`. (12) A run waiting for a + live attempt keeps its object alive only while an attempt is live or activating. (13) Chord's own loads take no read + transaction (the object holds its line, so plain reads are consistent); S4's read transaction stays for reads outside + a line (Durable's, 6b); supersedes "Slice 6 first decisions" (2)'s "in one read transaction". (14) `put` is an upsert + (the whole row replaced); `update`/`delete` of an absent key change nothing and are recorded with their count (0). + (15) At write `int` is a safe integer, `num` finite, `bool` stored 0/1; one compiled closed check per table. (16) The + column helpers are L1's (`str int bool num opt`), a restricted subset. (17) Default order by key ascending; the key is + appended to every `orderBy` as the tie-break; one direction per query in 6a. (18) Cursors are opaque strings bound to + table, order columns and direction; decoding them is a trust boundary (`invalid/args`). (19) Generated SQL has a + bounded number of texts (the facade caches every prepared statement): `limit` and cursor values bound, `inArray` as + one JSON parameter through `json_each`. (20) Cut to 6b: the engine's reads at an older seq (`find`/`load` with `at`) + and exporting `history: "full"` (the format defines the fields now). +- **Cold keyed loads on the line; Chord aligns with Durable (owner, 2026-10-27; slice 6a spec review blocker 9):** + a cold read of a keyed document (`get`, `watch`, an `at` subscription) is a job on the object's line, as Durable's + `snapshot` loads on the Session line: it waits for the line, loads, installs and releases it; cached reads take no + line; `tx.draft` loads inside its transaction, which holds the line; one load in flight per key, shared, each + caller's `ctx` ending only its own wait; a load installs its tracker only while the key is still absent (`adopt` + always wins). Why: a load outside the line could install a value from before a transaction that committed meanwhile, + and the next delta would be computed on it. **General rule (owner): Chord and Durable align as much as possible** + (the same rules and mechanisms where they solve the same problem: commits, lines, loads, errors, watches). +- **The migration API: the reviewers' consensus (owner, 2026-10-27; slice 6a spec review blocker 8):** the owner + delegates the shape to the spec reviewers ("whatever the reviewers all like and find consistent and easy to implement + correctly"); only a disagreement among them comes back. Proposal in the spec (Opus's shape, Sol's protocol): adds and + drops derived from the declared versus the recorded columns (a new column `opt(...)` or with a `default`; a dropped + column loses its indexes first); `migrations: { N: { rename?, rows? | update? } }` with renames explicit; a version + bump without an entry is a schema-only change; columns or key differing at the same version fail `broken`; key and + column-type changes (`rebuild`) cut from 6a; `rows` synchronous and pure, chunks of 100, writing changed columns plus + `$v` through an internal write form (`tx.update` never writes `$v`); `update` one `updateWhere` stamping `$v`; one + version at a time (a DDL commit recording `version N, migrating`, the data step, `ready`), indexes diffed after all + versions; resume continues `migrating` skipping rows with `$v >= N`; a `rows` function gets the stored row with + `json(...)` columns migrated on access. +- **Nullable columns and the `set` form (owner, 2026-10-27; slice 6a spec review blocker 6 and the DSL):** columns + are non-null by default; a nullable column is L1's `opt(...)` (`note: opt(str())`): in a row an optional field, + absent when `NULL`, never an explicit `null`. A `put` carries every non-`opt` column (a missing or unknown field + fails `invalid/args`). Queries follow SQL's rules for `NULL` (a comparison with `NULL` never matches, `ne`/`not` + included; ask with `isNull` or `or(isNull(...), ne(...))`), on every backend. Keys, index columns and `orderBy` + columns must be non-null (types and `defineTable`), so cursors are total. `alter` adding a column: `opt(...)` or a + declared `default`. The `set` of `update`/`updateWhere` is a plain object of values (`{ note: "fix" }`) or a callback + getting typed column references (`(e) => ({ hits: add(e.hits, 1) })`), like `where`; `col()` and `value()` are gone; + `undefined` clears an `opt` column. The semantics the reviewers listed (simultaneous evaluation of a multi-column + `set`, integer arithmetic and overflow, `replacePrefix` without the prefix, empty and U+10FFFF prefixes, `inArray([])` + and long lists bound as one JSON parameter, `when`'s condition, default order and key tie-breaks, one evaluator for + `where` and `set`) are specified in the 6a spec; no separate DSL review (owner: the spec review is enough). +- **Restricted names instead of assigned table names (owner, 2026-10-27; slice 6a spec review blocker 5):** names + that become SQL names are restricted at declaration (and checked on import) instead of being mapped. Host kinds, + object types (every contract, not only stores) and table names use npm package-name characters `[a-z0-9@/._~-]`, + lowercase only (SQLite compares identifiers case-insensitively), so plugin-prefixed type names (`@pi/file-history.store`) + stay valid; the separator is `:`, which they cannot contain, so the rendering `"session:@pi/file-history.store:edits"` + is injective (`a/b` + `c` and `a` + `b/c` differ) and readable in `sqlite3`. Index names `[a-z0-9_]+`, rendered + `:`. Column names `[A-Za-z_][A-Za-z0-9_]*` (camelCase allowed), two columns equal ignoring ASCII case + refused at `defineTable`, no `$` (the hidden `$v` and `$v` cannot collide). Import is a trust boundary: every + name in an archive is checked against these rules (no `"` can occur: quoting is always safe), every row against its + table's recorded columns, the parts against one compiled `ArchivePart` schema; a failure refuses the whole import. + Rejected: physical names assigned by the store (`t1`, `t2` with a mapping table: unreadable, owner), escaping logical + names into physical ones. +- **`runCounted` in `chord/sqlite` (owner, 2026-10-27; slice 6a spec review blocker 4):** `SqliteExecutor` gains + `runCounted(sql, ...params): Promise`, running one statement like `run` and resolving with the rows it + inserted, updated or deleted (SQLite's `changes()`: index writes and trigger rows do not count); `run` stays + `Promise`. The store uses `runCounted` only where a count matters (bulk writes' `count`, the engine's + conditional `UPDATE`, single `update`/`delete`). Node takes it from `StatementSync.run()` (free); the Durable Object + driver reads `SELECT changes()` after the statement, because the cursor's `rowsWritten` is Cloudflare's billing + count and includes index writes (verified in workerd 2026-10-12: an insert with two indexes 3, a 10-row bulk update + 20, `changes()` 1 and 10). Measured in workerd: `SELECT changes()` costs about 1.5 µs per statement and reads or + writes no billed rows, so making every `run` count would nearly double a small write on Durable Objects for callers + that need no count (Durable's `SqliteStorage`): hence a separate method (option 2, owner). Name `runCounted` + (owner). pi `8025c414d` (upstream first), with a real Durable Object test in Durable's workerd suite. +- **Store errors: Durable's rule (owner, 2026-10-26; slice 6a spec review blocker 3):** any backend error in a commit + step is `storage_failed` (the store stops, every later commit and read rejects `unavailable/storage_failed`, a new + reason in the protocol list; the host closes; `onError` once), as Durable's `GuardedStorage`: the SQLite facade cannot + tell a failed statement from a failed `COMMIT`, and what reaches the step is a real storage problem or memory and disk + disagreeing. Plugin mistakes never reach the step: they are refused when the write is staged (`invalid/args`). A + commit refused by the seal fence is rejected `released` before `BEGIN` and is no error of the backend. A backend + error on a read is `storage_failed` unless the read's `ctx` was aborted (`aborted`); invalid arguments are checked + before the backend (`invalid/args`). An `adopt` that throws is `storage_failed` (owners never throw from it). + Accepted: integer overflow past 2^63 in a bulk `add` is fatal (a `STRICT` integer column refuses SQLite's REAL). + Supersedes the 6a spec's "an error before `COMMIT` returned rejects that commit". +- **A live object can fail (owner, 2026-10-26; slice 6a spec review blocker 7):** a runtime failure of the owner (a + `migrate` that fails on access, a stored version newer than the code) seals its live attempt (the seal fence) and + makes the object `failed` with a `broken` cause: calls on it reject at once with the cause, its state members keep + their last values and get no frames, its natives close, its documents stay on the object (the failed data intact in + the store), `onError` gets one report (the kind is settled in the 6a spec; `object_failed { object, cause }` + proposed); recovery only by a reload or `ref.retry()` (a new attempt; only the failed load is redone). Nothing + cascades: dependents that are live stay live (§6), only their calls into it fail. +- **A native restart drains first (owner, 2026-10-26; general-model discussion):** when a reload restarts an object + with natives, new calls park, the running calls are awaited (the drain, bounded by `closeTimeout`), then the natives + close (reason `reload`), the new factory runs and the natives reopen; only a call that outlives the drain is cut off + by the seal fence. Supersedes §21's "running work ends `aborted`" for calls (running job runs: slice 7 decides with + the job ownership change). The general-model discussion is closed with this entry (owner: "model discussion done"). +- **Keyed state members, `local` contracts only (owner, 2026-10-26; reviews + `/tmp/keyed-members-review-{astra,fable,opus,sol}.md`, all four for A):** the gap: across objects live state travels + only as state members, one value per object, so a view with params (`NoteView { session, id }`) could not follow one + keyed document of a persisted store (P3 keeps persisted data in `local` objects with empty params). A contract member + may be `keyed(schema)`, a third member kind (part of the fingerprint), only in `local` contracts (`remote()` refuses + it): a provider returns a keyed document handle for it (in memory or persisted) or re-exposes another reference's + keyed member; a consumer on the same host takes `ref.member.at(key)` in its factory (only there: a counted pin on the + provider's per-key source while the consumer lives, loaded before the consumer goes live and never evicted while + pinned) and `ref.member.watch(key, ctx)` / `get(key, ctx)` anywhere (`watch` on an absent key gives `undefined`). An + absent key reads `initial()`; a retirement arrives as the `retired` flag; a key whose `migrate` fails breaks the store + (6a's keyed rule). Nothing on the wire changes: tabs and other hosts hold the view, an ordinary remote object with + singleton members. A derive over a retired source carries the `retired` flag on (a reader's redacted member learns of + the deletion). Versions of keyed members are mapped per key in slice 7 (`"refuse"` only for removed ones). Rejected: + B (persisted `local` objects with params: one object per entity, nothing lists or retires across them). Kept for a + real case: keyed members across links (Fable's design: keyed entries inside a hold's `bound`/`state`, a `keys { hold, + add, drop }` message, re-holds carrying their keys; or Sol's selector holds), as a later slice. Option C (a singleton + map plus a lookup derive, p2's `DraftStore`, `Status`) stays for small collections (§12 cookbook). Built in slice 6a + (about 300-350 lines), which already builds the per-key source behind `handle.at(key)`. +- **The seal fence (owner, 2026-10-26; the fence review `/tmp/fence-review-{astra,fable,opus,sol}.md`):** leftovers of + an ended or replaced attempt (handlers past the drain, queued commits, routine calls, watch deliveries) are cut off + by one token per attempt, and one per job run invocation for its state changes and outcome. (1) Sealed handles: + after the seal, new work through the attempt's handles is refused at use with `released` (`self.tx`, `ctx.get`, + which checks the abort signal first, a job start, a routine start); the attempt's references stay usable for its + own natives' closes, then are released. (2) Commits are checked at two points: when they enter the store's commit + step (sealed: rejected `released`, nothing written; not a storage failure) and at adopt (admitted while current, + sealed during the SQL: durable, adopted into the object's documents, staged job changes of the sealed attempt + abandoned); in-memory commits are checked inside the batch. (3) At the seal, a transaction of the attempt that has + not entered the step yet is aborted and the object's line released; a step already started finishes before the + attempt's teardown completes. (4) Cleanup still runs after the seal (releasing what stuck calls hold, `ctx.get` + references, presence entries, `host.activity` counts), so a stuck handler cannot keep a worker busy forever. The + existing start guards stay (an old activation finishing after the object ended, `Job.#invoke`, a `ctx.get` resolving + after its call settled). Unit: the attempt (every way an attempt ends already goes through `Attempt.seal`); a pure + swap is not a replacement (the same attempt continues), so a stuck old routine call past the handover is fenced per + routine invocation (slice 7). Natives' effects outside Chord are the plugin's; their `close` is honoured, and their + writes to Chord state go through transactions and are fenced. Cost: one boolean read per commit at one (in memory) or + two (persisted) points. Replaces slice 6a's per-path rules ("`adopt` of a dropped handle is a no-op", race 4, staged + job runs adopted after `Job.end()`); `Documents.seal`/`assertWritable` become the token's check. +- **Job runs belong to the object; singletons stay resident (owner, 2026-10-26; the four fence reviewers' + follow-ups):** (1) A job's records, keys, active count, `running`, run state documents and pending actions belong to + the object (per job name), with its documents; the attempt keeps the job's implementation (`run`, `finish`) and the + running invocations (its drain, its seal). A run's state changes and its outcome are fenced by the run's own + invocation, not the attempt; `finish` runs on the object's current live attempt (a finishing run waits for one); a + run whose job the new code no longer declares stays `blocked`. Why: with runs per attempt a native restart breaks key + dedupe (a run can start twice), `concurrency` (undercounts), `running` (loses its runs), the state documents (a dead + attempt's cleanup would clear them) and §15's rule that the current code finishes a run aborted by a restart. Open + for the spec: whether a run waiting for a live attempt keeps a `broken` object alive (proposal: it counts only while + an attempt is live or activating). (2) Persisted singletons and `at(key)` documents stay in memory while their object + lives; only keyed persisted documents are evicted. Why: `handle.value` is a synchronous read (methods, derives, + snapshots for new holds), so the value must be in memory when read; an evicted singleton would need an async read, + as keyed documents have. Memory is bounded by the number of declared singletons, one value each; data that grows + belongs in keyed documents or tables. +- **Documents and the line belong to the object, not the attempt (owner, 2026-10-26; general-model discussion after + the slice 6a spec reviews and the fence review `/tmp/fence-review-{astra,fable,opus,sol}.md`):** an object (one + identity) owns its documents (in memory and persisted, their trackers) and its line for its whole life, across + retries, failures and native restarts; an attempt (one try at activating the object: factory, references, natives, + members, derives, routines) keeps only what belongs to one try: its natives, routines, derives and its seal token. + A new attempt's factory gets handles to the same documents (`self.state(def)` by name). Persisted documents load + once per object, and again only after a load failed (a `migrate` that needs fixed code). In-memory state survives a + failure and retry too; ending the object clears it. Why: per-attempt documents let a late commit of a dead attempt + leave the next one on a stale base (the review's race), contradict §9 ("in-memory documents survive native restarts + of a live object"), and were an accident of slice 2 (each activation a fresh instance, documents created inside it). + Built with slice 6a (carry-over), which needs it; replaces slice 7's carry-over "run records and state documents + belong to the object across attempts (documents belong to an attempt today)" for documents. Decided the same + day: the seal token's rule ("The seal fence") and job runs ("Job runs belong to the object"). +- **Writes carry their versions; archives are lossless (owner, 2026-10-26; spec review blocker 1, all four reviewers):** + every `put` write carries the row's `$v` and each `json(...)` column's version explicitly, in the outbox and in + archives alike (`{ kind: "put", table, row, v: 3, columns: { meta: 1 } }`); the applier never infers a version from + the table definition (an archive's `table` write has the current version while cells may be stored at older ones; an + inferred stamp would label an old shape current, and an export runs no plugin code, so it cannot migrate). A `table` + write carries the table's `state` (`migrating` | `ready`), so an export between migration chunks imports as + `migrating` and resumes. A `document.create` carries the record's stored version and fingerprint; for + `history: "latest"` its base is the current value (base plus deltas, as a load gives it); for `history: "full"` + (Durable, 6b) the archive carries every revision with its seq, retired `full` records included (format 1 defines the + fields now, 6b builds them); retired `latest` records stay out. Supersedes in part "Export archives are operations" + ("a `document.create` per live document record (with its base)", "Retired documents … are not in it"). +- **Slice 6a's three code questions (owner, 2026-10-26):** (1) keyed reads take a context: `handle.get(key, ctx)` and + `handle.keys(ctx)` for every keyed handle, in memory too (one signature; persisted, they read the store; "Contexts + are required"). (2) The activation attempt's `readyTimeout` deadline does not run while an eager migration step runs + (a `rows` step over 10M rows takes longer than 60 s; the scan resumes after a crash anyway); getters still time out + by their own `readyTimeout`. (3) A host without an address uses `client` as its host kind in store addresses (owner + strings, table names), so a browser tab with sqlite-wasm can have a store later. +- **The commit-step benchmark (owner, 2026-10-26; S3's condition "SQLite must not get slower"):** in 6a + `test/next/bench/store.ts` measures the store without outbox against Durable's `SqliteStorage` per commit on (a) a + commit of one document delta, (b) a commit with one table `put`, (c) 1,000 commits back to back, in bursts (median + and p95 over several runs) with memory after GC; each must be no slower. The outbox's extra cost per commit is + reported separately. A failed gate goes to the owner with numbers and options before the build continues. 6b repeats + the gate with `StoreStorage` on Durable's own storage benchmark. +- **The outbox is optional, internal and versioned (owner, 2026-10-26):** supersedes in part "The store" and S1/S5 where + every commit writes the outbox. (1) A store has an outbox only when opened with one, `openStore(backend, ctx, + { outbox: true })`; the setting is persisted in the store's metadata. Without it the commit step writes data and + `lastSeq` only (tests, web tabs, a server's instance store, a session before sync). Turning it on records "outbox from + seq N+1" and the sync layer starts from an archive at N; turning it off drops its commits (a later start needs a + snapshot again). Opening with the other setting is an explicit switch with these rules, never a silent gap. (2) The + outbox is a table in the store's own backend, written in the commit step's backend transaction: SQLite commits data + and outbox atomically even with `synchronous=NORMAL`, for one `INSERT`. An external outbox would work only as Durable's + JSONL storage commits across files (outbox entry first, SQLite `COMMIT` as the commit point, recovery cutting entries + above `lastSeq`), only for a local, truncatable outbox (never Radius: an uploaded entry cannot be taken back), and only + with an `fsync` of the outbox before every `COMMIT` (otherwise a power loss keeps the commit and loses its entry, which + no pull-back recovers); can be retrofitted later if needed (owner). (3) No pluggable outbox encoding: the store keeps + commits as it likes and exposes them typed, `store.outbox.read(afterSeq, limit, ctx)` -> `{ seq, writes }[]` and + `store.outbox.trim(throughSeq, ctx)`; the sync layer encodes for the wire. (4) Versioned on two levels: the outbox + table's layout is part of the store's schema (the store's own migrations at `openStore`); the write vocabulary has a + `format` number, the archive manifest's `format`, stored per outbox commit (`outbox(seq, format, writes)`); + `outbox.read` hands out the current format (older commits through the store's pure step functions), a newer format + than the code is refused with the data intact, and batches and archives carry it. (5) The benchmark gate (S3) is + measured without the outbox; its cost per commit is reported separately. Replaces the sync-layer carry-over "outbox + growth without sync". +- **Writes, commits, batches, archives (owner, 2026-10-26):** the names of what the outbox, sync and export carry. A + **write** (`StoreWrite`, tagged by `kind`, Chord's rule for tagged unions) is one change of the store: `put`, + `update`, `delete`, `updateWhere`, `deleteWhere` (with `count`), `table`, `alter`, `rebuild`, `index`, `dropIndex`, + `document.create`, `document.change`, `document.retire` (as Durable's `StorageWrite`, "one record or document + mutation in an atomic storage commit"; S17's "five writes"). A **commit** is the writes of one seq as the outbox keeps + them (`{ seq, writes }`); a **batch** the commits `from..to` that sync uploads and a replica applies; an **archive** a + manifest, the writes from 0 to `lastSeq` and an `end` (its parts are `ArchivePart`). Not "record" (document records, + run records, Durable's records), "op" (Delta ops inside document changes), "entry" (Durable's conversation entries) + or "change" (`WatchChange`, Delta's `Change`). Earlier entries' "outbox record" reads "the outbox's commit" and + "operation" reads "write"; the book's `op:` tags read `kind:`. `StoreStorage` (6b) turns Durable's `StorageWrite`s + into `StoreWrite`s. +- **The `json(...)` column helper (owner, 2026-10-26; slice 6a point 8):** `json(schema, { version, migrate? })`. + The schema is required (reads are typed from it; a migration's result is checked against it once, R8; written values + are not walked, R9); `version` is required, as for `defineState`; `migrate(old, from)` is optional and pure: without + it, or when it throws or returns an invalid value, reading an older value is refused, the data stays intact and the + owning object fails `broken` (R8, R15). New: each `json(...)` column's version and schema fingerprint are recorded in + the table definition (the `table` write); at activation, the same version with another fingerprint fails the object + `broken` (D9; one comparison per column per activation, none per row). A read does not write back: an old row pays + the migration and the check on every read until its column is written; an eager `rows` step rewrites a table at once. +- **Facets only declare (owner, 2026-10-26: "remove facet routines, everything done through objects seems sensible"):** + facet routines (`self.on`, `self.onLive` in `FacetSelf`, added in slice 5 because §16 listed routines) are removed + with the documents. A facet's setup declares only: `host.serve`, `self.contribute` (slice 8), `host.references`; + no natives, documents, routines or jobs, so a facet has no runtime behaviour and cannot fail after `load`. Without + documents a facet routine owned nothing it could write and acted only through other objects' methods; standing + behaviour belongs to the object whose state it reconciles. p2 already works so: all 63 facets are + `defineFacet((host) => …)` with no `self`; its routines and loops live in host-lifetime objects. Built in slice 6a + (carry-over): the routines in `FacetSelf`, their start in `host.ts#setUp`, their drain at close, tests, the facet + part of example 18. +- **Relays do not check answers (owner, 2026-10-24; second 4b review):** supersedes the third fix-round choice. A + relay forwards answers from upstream (`bound`, `state`, results) unchecked and unparsed; the end that parses a part + checks it against its own link's limits. Reason: the relay's check had to release the hold upstream and hold it + again with the same id on `retry`, and an old `state` or `gone` still in flight then looked like the new hold's + (reproduced by three reviewers). The relay still checks everything arriving from downstream. A reference failed + `too_large/part` leaves its path (a reconnect does not hold it again) until `ref.retry()`; this lives only in the + consumer's memory. +- **Slice 4b fix round (owner, 2026-10-24):** the builder's five choices accepted and folded into design.md: call + arguments sent up a parent link are never compressed (a host cannot tell whether its parent sees it as a child or a + route); `revoke` goes on route links before revoked child links close (worker objects end `forbidden/revoked`, not + `released`); ~~a relayed answer over the route link's wire limit fails its call, or releases its hold upstream and + sends `loading` and a broken `failed` downstream (`retry` re-holds with the same id)~~ (**superseded** by "Relays do not check answers"); compressed hold params are a + protocol error on every link; the asynchronous inflate runs one part at a time (its queue is bounded by the parent's + or route peer's window; a misbehaving one of the app's own hosts is accepted). No extra performance regression tests + (cost guards, GC tests and manual benchmarks suffice). A second, non-pedantic review (correctness and performance) + before the commit. +- **Slice 4b review decisions (owner, 2026-10-24; reviews `/tmp/s4b-review-{astra,fable,opus,sol}.md`):** (1) a + dependency failing with `invalid/cycle` makes its dependent broken (exception to R18). (2) Cycles that enter an + already activating object (simultaneous starts) are a documented limitation: they time out transient and retry; no + probe message. (3) Child links refuse compressed parts (protocol error): clients never send them, and a small part + could inflate to 64 MiB on the server; asynchronous inflate then happens only on a host receiving from its parent. + (4) The text codec tags every part, so no JSON value is reserved for compression. (5) A departure discovered while a + batch is sending lands in the next batch (one commit stays one message per link). (6) Presence `origin` is one rule: + the caller's incarnation, also for `ctx.get`. (7) Revocation's error is `forbidden` with a new reason `revoked` (a UI + can say "access revoked"; the protocol table changes, version stays 1 until p3 ships). (8) Compression details: a + tagged adapter result (too large or corrupt), compression also on IPC, `./next/node` export later, hold params never + compressed. (9) `host.references` keeps a type's list from its first ask (entries come and go with references). + (10) Instance ids on restart are slice 7's (carry-over). Revocation propagates over route links; the decoded limit is + stamped on holds only (design §4, §22 aligned). +- **Slice 4b to building (owner, 2026-10-24):** `slice-4b.md` revised for flow control per origin goes to a builder + directly; fresh reviewers (Astra, Fable, Opus, Sol) review the build; the owner then decides the open points the + spec marks as proposals (a cycle error makes the dependent broken; compression as built; `host.references` lazily + per type). Compression is built as option (b) (owner leaning, decided after the build): hosts with a synchronous + compressor compress parts above 16 KiB at their origin; browsers send uncompressed and inflate asynchronously with + `DecompressionStream`, holding later messages of the link in order (bounded by the window and N; acks after apply). +- **Slice 4a approved (owner, 2026-10-24):** `packages/chord/docs/next/slice-4a.md` as revised after the reviews. Build + order: the pending queue in Chord's shared code first (pi, upstream), then the slice by a fresh Opus builder (`s4`). +- **Slice 4a review decisions (owner, 2026-10-23; Astra's and Fable's spec reviews, `/tmp/s4a-review-*.md`):** + supersedes in part "Slice 4 split and first decisions" (resets as `r` ops). A `hold` carries the address the caller + computed; relays route by address and never need the contract or parse params. A `hold` carries `principal?` and + `incarnation?`, honoured only on the parent link (calls inherit their hold's, so calls carry none; refines R1/Q13 + "hold and call"); a reference made on a host sends its maker's principal over route links (`ctx.get` in a guest's + handler acts as the guest); upward, a server ignores a child's fields (a worker acts as itself; documented + boundary). Resets deliver full bindings: `bound` carries a list of holds (`bound { holds: [...] }`, one batch), a + `bound` on a bound hold replaces its replicas (another variant allowed; moved from slice 7 to 4a); overflow stays an + `r` op. A provider answers `reset` through its pending path (never exceeding K). A relay recovers per (downstream + link, upstream link): once something from U cannot reach D, everything from U to D is dropped, and the next free slot + requests one reset of every hold D has on U; an answer that finds no slot is requested again. Heartbeat: one + `lastReceived` timestamp per link and one host timer; the 30 s silence covers the handshake. Transports: a connecting + WebSocket buffers; IPC `send() === false` is backpressure. A hold from the parent is never routed back to it. +- **Slice 4a spec decisions (owner, 2026-10-23; `packages/chord/docs/next/slice-4a.md`):** a heartbeat on every link + kind, in the protocol layer (`ping`/`pong`; a side that received nothing for 15 s pings, a link silent for 30 s + closes; host options with these defaults); supersedes design §3's "heartbeats are the adapter's policy". The text + codec may carry relayed links (one parse per part per hop, accepted for a debugging codec). `resolve(address, + context)` takes a Context. Upstream reconnects at a relay are `loading` downstream, then the new `bound` (3b's + re-hold path, another variant allowed); flow-control resets stay `r` ops. +- **Compression at the origin (owner, 2026-10-19):** a JSON part above a threshold (e.g. 16 KiB) is compressed once + at its origin, whatever the link; byte segments stay raw; relays forward it compressed; IPC carries compressed parts + too. No per-link compression. +- **R6 and contract compatibility (owner, 2026-10-18), strict, like documents:** a hold carries only the consumer's + contract version and fingerprint, never a description. The provider binds when they equal its current version and + fingerprint, or an accepted old version's fingerprint, computed from the provider's own old schemas. Every contract + change, additive ones included, bumps `version`. (**Superseded in part** by Q10: forward evolution in code and + `checkAccepts`; old versions come from the materialized history, not from copies in `accepts`.) `accepts[v]` carries the provider's own copy of version v's member + schemas plus value mappings: old arguments are validated against the provider's old schema, then mapped (state: + new to old; methods: old arguments to new, optional result mapping back); an additive change needs the old schemas + and no mappings. Anything else is `contract_mismatch` (or `updating` during a rollout). Consequences: no + descriptions on the wire (the `describe` message kind of R1 and the per-link description cache go away), R3 is moot, + the coverage check across fingerprints of one version is gone. + +## Portability boundaries in the kernel + +From `research/chord-on-durable-objects.md`, adopted for the Chord v2 kernel because they cost little and help tests: + +- **A host is a logical identity plus an incarnation**, not an OS process. +- **The loader is a host adapter:** the kernel also accepts facets imported ahead of time (static composition, as on a + Durable Object or in tests); loader D is one adapter. +- **Values are JSON plus bytes, defined independently of the transport;** (**superseded in part** by Q4 (IPC carries the same binary message, V8-serialized bytes)) IPC structured clone and WebSocket binary + frames are carriers, not the definition. +- **Durable plugin state:** whatever must survive a host teardown (a DO eviction, a crash, a restart) is a persisted + document written in transactions (`DocumentStorage`). Everything else is rebuilt on activation: in-memory documents, + natives, derives (`previous` is a cache), routines, presence and references (re-held by clients); job runs end + `lost` unless their records persist. + +## Object kinds + +- **`local` stays** next to `remote` (owner, after the plugin scouts judged "one kind only"; all four would have + accepted it with conditions). +- **`local` means:** (**superseded in part** by R7 (validation for every caller; `local`'s exception is Q15, open) and R9 (no dev freeze)) same host, no validation, no copy, no principals; plain data only; **no `opaque`, no callbacks** + as arguments. Behaviour goes through registries, multi-step work through jobs, input through flows (or Questions), + runtime services through host values (**superseded:** host values were removed; runtime services are core objects, + "Hosts and placement" and "Review round 1"). Without a copy caller and object share values: neither may mutate arguments + or results (the playground freezes them in dev mode to check). +- **Kernel semantics decided while building step 2:** + - `local` factories are `(input, self)`; a local method's `ctx` has no principal; references a local object makes + run as the host principal; attribution is passed as data. + - ~~Calls on `remote` types validate and copy arguments and results also on the same host.~~ **Superseded** by "The + state path": same-host calls pass values by reference; arguments are validated only for callers that are not + host principals, at the provider. + - (**superseded in part** by R9 (no dev freeze)) `Uint8Array` leaves are never copied in process (shared, immutable by convention); the dev freeze of `local` + values freezes objects and arrays only. Transports carry bytes raw (IPC structured clone, WebSocket binary + segments), never base64. + - (**Superseded** by Q12: unknown fields are rejected.) Undeclared argument fields are stripped as in today's Chord: the consumer strips to its schema, the provider to + its own, then validates; a handler receives exactly its declared shape (additive rule; no mass assignment). + - A failed reference stays alive: `ref.state` is a tagged value (`loading`, `live`, `failed` with `cause` and + `retryAt`, `released`); while held, Chord retries per the per-address backoff; `ref.retry()` tries at once; calls + on a failed reference reject at once. Calls park on link drops, loading or restarting objects and rollout + `updating`. +- **Pattern: shared core, per-principal views** (confirmed; renamed "shared store" in "The state path"). Identity stays (type, input, principal key): the + factory receives the principal and builds its variant and principal-specific state (e.g. a read-only guest's + filtered view). Heavy or shared parts (a Durable watch, a prompt watcher, a sandbox, a store) live once in a + `local` core object (no principals: one per (type, input) on the host; no copy); per-principal remote objects + `self.get` it and derive their view from it. Same for stores that serve different views per principal. +- From the scouts' review of "one kind only", still valid for `remote`: same-host state as immutable snapshot plus + deltas (never a full clone per change); bytes transferred, not cloned; real schemas instead of `Type.Unsafe`; double + copies through facade layers are fixed by merging owners. + +## Reload + +- **Per host:** load (bundle; a load error rejects) → staging (run the facet setup and every live object's new factory + with a recording `self`, declarations only, nothing activated; a throw rejects) → commit (atomic) → after. +- **Rollback is all-or-nothing per host:** any load or staging error leaves the host entirely on its previous code. + A host is completely at revision R or R−1, so status per host suffices. +- **Pure objects** (**superseded in part** by R15 (old code touching a document migrated past its version throws)) (no `self.open`, old or new) are swapped in place: new methods and derives; documents, references + and jobs stay; routines restart serialized; new references become live; then the object's gate opens. Running calls + finish with their old code. +- **Native objects of the plugins in the reload restart:** running work in them is aborted (outcome `aborted`), + natives close, the factory runs, natives reopen. ~~No per-module digest: if a native is expensive to restart, the + author splits it into its own plugin.~~ **Superseded** by R16 (restart patterns: drain in the background, swap + through a registry). An open that fails makes that object `failed` with backoff; the rest of the + host runs on the new code. +- **Remote consumers** of a restarting object see what they see for any object that briefly goes away: `loading`, + values readable, calls park, re-hold; calls running on the old instance end with an error. +- Rollback covers code errors (load, setup, factories); activation errors (opens, new references, migrations) stay per + object; a bug in a method fails that call. + +## API additions + +- `self.derive(name, sources, (values, previous) => …)`: `previous` is the derive's last value, kept across factory + re-runs by name. +- `job.start(args, ctx, opts)` returns a run with `outcome(ctx)`: the caller can await the outcome; the caller's abort + ends only its wait, never the run. +- Jobs stay as designed (intent / run / finish). Jobs as objects with an "until done" lifetime are an option for later. +- **Bytes are a schema type** (**superseded in part** by Q4 (IPC carries the same binary message)) (`Uint8Array`), transported natively: structured clone on IPC (`fork` with + `serialization: "advanced"`), binary frames on WebSockets. No base64 in JSON. (Today an IPC payload is JSON, deflated + and base64-encoded; a 1 MiB chunk of compressed data grows by a third or more.) +- **Large transfers are chunked by the owner, not by Chord:** the owning object offers `begin` / `append({ upload, + offset, bytes })` / `finish` and `read({ id, offset, length })`, chunks of about 1 MiB. Each side holds one chunk; + the awaited `append` is the backpressure; after a link drop the writer asks for the stored offset and resumes; an + unfinished upload is removed after a timeout by the owner. A small helper wraps the loop. Chord gets no stream type + (it would need its own rules for link drops, cancellation, relays and restarts); reconsider only if many owners + repeat the protocol. + +## Registries and HTTP + +- **Registries work like Durable's** (option c): (**superseded in part** by R21 (only facets and `lifetime: "host"` objects contribute) and "Changing contributions, `self.writable`") each contributor (a facet or an object, by its identity) contributes + at most one value per registry; the table is an immutable, ordered list of `(contributor, value)`; Chord has no + generic keys and no `conflict` flag. The owner composes the table (in the registry's `compose`, below) and decides + conflicts and order (**superseded:** composition runs at runtime and reports conflicts, D1 in "Review round 1"). Chord ships helpers for the common compositions + (`keyed(...)` with a conflict rule, `ordered(...)`). Several contributors (model providers, environment kinds) are + the owner's composition, not a special case; selection (which contributions apply where) stays possible. +- ~~The composition runs in staging and a conflict rolls the host back.~~ **Superseded** by D1 ("Review round 1"): + `compose(table, { report })` runs at runtime, always delivers a value and reports conflicts. + +- **Registries may carry natives.** An entry lives exactly as long as the object or facet that registered it; the + table changes when it goes away. Consumers look up at each use or follow the table and keep nothing from an entry + beyond a use (a pinned Durable phase holding an old table is fine). Natives reach other plugins only this way: no + `opaque` members, no natives in method results. What a native's old value does after its owner restarts is the + owner's `close` (a model runtime keeps working until collected; a closed SSH connection fails its in-flight `exec`). +- **HTTP is an optional Chord component** (`@earendil-works/chord/next/http`): a route gets a plain request summary + (method, path params, query, principal, selected headers; parsed and capped JSON bodies; uploads handed over in + chunks) and returns a response description (`json`, `page`, `bytes` with `size`, `mime` and `read(offset, length)`, + `redirect`, `error`). The component does ranges (206/416), HEAD, download headers, streaming, abort on disconnect and + the CSP nonce once, instead of every plugin. Request resolvers (who sent a request) are a hook taking the summary. + A **channel component** provides WebSocket byte channels (`send`, `onData`, `close`) for terminals. +- **HTTP routes and wraps (middleware):** a contribution holds `routes` (pattern to route) and `wraps`. Routes are + composed with `keyed` (two contributors with one pattern: a reported conflict, D1). Wraps run in **phases the owner defines** (e.g. + `observe`, `headers`, `transform`; outermost first), within a phase in contributor order; no numeric priorities + (plugins have no single registration order, unlike Express or Hono; Fastify's lifecycle phases are the model). A wrap + applies by default only to its own contributor's routes (`scope: "own"`, Fastify's encapsulation); `scope: "all"` + is explicit; `route` narrows it to one route, and a `route` that does not exist is a reported conflict (D1). Wraps + may add a security boundary on top of the owner, never instead of the owner's own checks. A request uses the router + as it was when the request started. + +## Durable harness inputs + +- **`HarnessOptions.models` and `settings` accept a function** (`() => Models`, `() => HarnessSettings`), like + `env`; a plain object is still accepted. Internally both are stored as functions; generation, hooks and tools get + the function's current return value. Replaces p2's `Proxy` around the registry entry. Done in pi main first, then + merged into `private`. +- Models and environments reach the harness through registry entries (functions) registered by the objects that own + their natives; the `Harness` object opens Durable itself (no host value) and looks the entries up at each use. + +## Input into running work + +- **Questions (`questions.md`) is one possible pattern, not the required one.** Another: the owner of a flow keeps it + as state keyed by a uuidv7. Example login: `start({ provider })` returns an id; the flow's live state holds the + current step (`{ id, kind, ... }`), notices and status; `answer({ id, step, value })` resolves the waiting step (a + late or second answer to a step that is gone gets "no longer active"); `cancel`. The client watches the state, so no + `next()` is needed. pi-ai's promise-based `prompt` callback stays inside the owner (the job keeps the resolver); + nothing crosses an object boundary. Open: the flow as a keyed document of `Login`, or an object `LoginFlow {id}` that + lives until done ("jobs as objects"). +- **Imperative consumers get a client helper over the same flow state,** (**superseded in part** by design section 20 (flow state is tagged by `kind`, not `status`)) not a second server protocol. Tests, + scripts, a CLI or an agent tool loop over the steps; the web UI watches the state; both see the same steps: + + ```ts + async function* steps(flow, ctx) { // generic: any flow with { status, step?: { id, ... } } + let last = 0; + for (;;) { + const s = await until(flow.state, (v) => v.status !== "running" || (v.step && v.step.id > last), ctx); + if (s.status !== "running") return s.status; // done | failed | cancelled + last = s.step.id; + yield s.step; + } + } + + const { id } = await login.start({ provider }, ctx); + for await (const step of steps(await ctx.get(LoginFlow, { id }), ctx)) + await login.answer({ id, step: step.id, value: answerFor(step) }, ctx); + ``` + + Why not a long-held server `step()` call: a link drop fails it and the consumer cannot tell whether it missed a + step; the helper reads state that re-holds by itself, and `last` keeps steps from being lost or repeated. The + same shape fits login, tool approval and a question from the agent. + +## Ownership principle + +- **Every thing has exactly one owning object: the smallest object it belongs to.** The owner holds its data + (documents), enforces its rules (in its methods) and offers the operations others need. Nobody reaches past the + owner into its data, writes its files or documents, or attaches rules to it from outside. +- Ownership nests: the session owns the set of its conversations and its location (creation, relocation); each + conversation owns its content and rules (read-only, follows the session cwd) and enforces them in its own methods. + Rules set at creation are written in the creating commit (Durable `createConversation` with `init`). +- Methods return data, never objects; an object is handed out as its address, and the receiver gets it with + `ctx.get(Type, address)` (identity, lifecycle and principal stay with Chord). +- Registries are extension points that add behaviour (tools, routes, renderers, commands, send routers, input gates); + they never guard owned data. +- Core owners are always present; a plugin owner may be missing (optional dependencies are registries owned by the + dependent, "Cuts and closed deferred points"). +- Risk to avoid: a core god object. Each core owner owns only what belongs to it. + +## Porting guidelines + +- **What core enforces on data belongs in a format core owns,** written by the plugin when it creates the data; a + plugin's private document is read only by that plugin. Example: subagents/agent mark one-shot and cwd-following + child conversations in a core per-conversation policy document (`readOnly`, `followsSessionCwd`) in the creating + commit; chat and relocation read it directly; `ReadOnlyConversations` and `SessionCwdFollowers` go away. (This was + plugin design, not a missing mechanism: today the rule is lost when subagents is not loaded.) + +## Still valid from `target-design.md` + +Documents and the store (async `tx.draft`, keys, tags, eviction, Durable's transaction rule), job sandwich, routines +(`self.routines.on/every/loop/custom`), registries (versioned, immutable tables, no late binding), the +Questions service (`questions.md`), retry triggers and `retryOn`, revisions over the complete code set, mismatch rules +during a rollout, start barrier. (**Superseded in part** by R25 (contributions may change; rejected are registries +created at runtime and mutable indirection), R17 (the harness starts after the barrier) and R6 (strict compatibility).) + +## Open + +Everything else from earlier lists is decided (see the entries above and `packages/chord/docs/next/design.md` §26). + +- **Future goal: deploy Chord apps to Cloudflare Durable Objects and similar platforms.** Hot reload there is not + needed; composition through plugins is (plugin set fixed at deploy). Research in `research/chord-on-durable-objects.md`. +- Deferred until a concrete case: jobs as objects and where a flow lives (when login is ported); project plugins. +- p3 domain points (decided per domain when p3 is designed): `SessionFiles {session}` (proposed), session delete, + connection pools, config files with a single writer, file formats as versioned interfaces, the terminal host + process as a revision artifact. diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md new file mode 100644 index 000000000..af87a6489 --- /dev/null +++ b/packages/chord/docs/next/design.md @@ -0,0 +1,2199 @@ +# Chord v2: target design + +The design of Chord v2 (`packages/chord/src/next`, export `@earendil-works/chord/next`) and of how apps (the +playground, p3) use it: **the one current statement of every rule** (owner, 2026-10-24). Every decision is recorded in +the decision log `packages/chord/docs/next/decisions.md` (the history and the reasons, kept complete) and folded into this +document in the same commit. Where a rule is built, the slice is named (`slices.md`); the code follows this document. +It replaces `apps/p2/docs/target-design.md`. + +Marks: everything is decided unless marked **Decision needed** (problem with an example, then options; nothing in such +a block is settled; section 26 lists them all). **(carried over)** marks behaviour of today's Chord that v2 keeps +without a new decision. API names are working names. + +Contents: 1 goals · 2 concepts · 3 hosts and links · 4 principals · 5 object types · 6 objects · 7 references · 8 ctx · +9 documents · 10 store and storage · 11 kinds of state · 12 derives · 13 natives · 14 routines · 15 jobs · 16 the facet +· 17 registries · 18 start data and platform bindings · 19 presence · 20 input into running work · 21 reload and rollout +· 22 wire · 23 optional components · 24 Durable integration · 25 apps: ownership and porting · 26 decisions needed · 27 +versions and adapters · 28 building: slices · 29 telemetry · 30 cost model · 31 rejected + +--- + +## 1. Goals and principles + +1. **Reload without restarts.** A plugin's new code takes over inside the running app: hosts keep running; data, open + connections and running work survive where they can. Reloads are explicit (a button or a command), never on save. + Base code (Chord, the app runtime) restarts the hosts. +2. **Ownership.** Every thing has exactly one owning object, the smallest it belongs to. The owner holds the data, + enforces its rules in its methods and offers the operations others need. Nobody reaches past it. +3. **Factories declare, Chord activates.** A factory is synchronous and only declares; Chord performs activation. + Handles a factory gets are valid for the object's whole life. +4. **Natives only in objects.** OS-level values are opened and closed by the object that owns them, and reach other + plugins only through that owner's methods, state or registry entries. +5. **Errors look like disconnects.** No request-id framework; consumers handle a failed call as they handle a dropped + link. References re-hold by themselves. +6. **Tagged unions, always.** States, results and outcomes are discriminated by `kind`; each state carries exactly its + own fields. +7. **Portable kernel.** A host is a logical identity plus an incarnation; facets may come pre-imported; values are + JSON plus bytes independent of the transport. Node first; Cloudflare Durable Objects and similar stay possible. +8. **Context last.** Parameters are the required arguments, then the context (`ctx` or a Durable `Context`), then + optional arguments: `self.tx(fn, ctx)`, `handle.change(key, fn, ctx)`, `chord(fn, context)`, `job.start(args, ctx, + { key })` (owner, 2026-10-21). +9. **Few concepts, no ceremony.** No special directories, no `defineNative`, no lints standing in for design. +10. **Aligned with Durable** (owner, 2026-10-27): where Chord and Durable solve the same problem (commits, lines, + loads, storage errors, watches), they use the same rules and mechanisms. + +--- + +## 2. Concepts + +| Concept | What it is | +| --- | --- | +| host | one participant (a process, a tab, a DO): a logical identity plus an incarnation | +| link | a connection between two hosts | +| host kind / address | typed description of addressable hosts / one host of a kind (`Session({ id })`) | +| facet | a plugin's declarations for one host kind: the types it serves, its contributions; no state or behaviour of its own | +| object type | a contract: name, params (which instance), host placement, members, version | +| object | one instance per (type, params, principal key); `local`: per (type, params) | +| principal | who calls; stamped by links, typed by the app | +| reference | what `get` returns; keeps its object alive; live and hydrated when awaited | +| document | a named JSON value an object owns, written in transactions | +| derive | a value computed from watchables | +| native | an OS-level value opened with `self.open`; only in objects | +| routine | standing behaviour of an object: `self.on(source, fn)`, `self.onLive(fn)` | +| job | owned work a call starts that outlives it: intent, run, finish; optional state document per run | +| registry | an owner's extension point: one contribution per contributor, composed by the owner | +| shared store | a `local` object holding shared data once, behind per-principal `remote` views | +| `ctx` | per call: cancellation, principal, call-scoped references | +| revision | (server incarnation, sequence): the complete code set a host runs | + +--- + +## 3. Hosts and links + +### Roles + +A Node app has one shape: + +``` +client client client browsers, TUIs: not addressable + └──────┼──────┘ + server exactly one, addressable + ┌──────┼──────┐ +worker worker worker addressable by a typed address, started on demand +``` + +| Role | Addressable | Principal comes from | Lifecycle | +| --- | --- | --- | --- | +| client | no | authentication of its link | connects by itself | +| server | yes, one | its own declaration | runs | +| worker | yes, many | stamped by the parent | started on demand by the app | + +Clients serve only `local` types (they are not addressable): a host without a principal refuses to serve a remote type +(`load` throws; a remote object is owned by a principal of the host serving it; owner, 2026-10-24). A client's local +calls have no principal; what a client calls on the server is stamped with the client's authenticated principal. +Clients reach workers only through the server, which relays; workers reach each other through the server too. A +detached process (e.g. a terminal host) would be another child kind of the server. Things that are not hosts: a code +mode sandbox (a worker thread inside a worker), remote environment daemons (their own protocol, owned by an object), +the TUI sandbox behind its launcher (the launcher only passes frames). A Durable Object deployment is a mesh routed by +the edge (section 24 and the research note). + +### Host kinds, addresses and placement + +```ts +export const Server = hostKind("server"); +export const Session = hostKind("session", { id: str() }); // params like a contract's + +export const Notes = remote("notes", { + version: 1, + params: { session: str() }, // flat record of required scalars + placement: ({ session }) => Session({ id: session }), // a typed function of the params + variants: { … }, +}); +export const Catalog = remote("catalog", { placement: () => Server(), … }); +``` + +- `hostKind(name, params?)` declares a kind of addressable host; its params follow the params policy of contracts + (section 5). `Session({ id: "S1" })` is an **address**: `{ kind, params, key }`, the key built like an identity key + (`session|S1`); two addresses are equal when their keys are. No host-name strings appear in app code. +- **`placement`** (owner, 2026-10-21) is a typed function of the params returning an address; `local` contracts have + none. It is not part of the contract fingerprint; a changed rule surfaces as a refusal (below). +- **`params`** (owner, 2026-10-21; was `input`) select the instance among objects of its type and are part of its + identity (section 5). +- **The caller computes the address** and sends it in the `hold`; a host that relays routes by that address alone and + never needs the contract or parses params. **The provider computes it again** from the params and refuses an object + that is not its own: broken `invalid/placement` (misrouted holds are never forwarded). During a rollout the refusal + is handled like a mismatch (provider on another revision: park; same revision: fail; slice 7). +- **Where a reference made on a host, or a hold arriving on one of its links, goes** (first match; slice 4a): + + | Case | Goes | + | --- | --- | + | the type is served here and its placement (computed here from the params) is this host's address, or the host has no address, or the type no placement | here | + | a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) to a served type placed on another host | on, as if not served here (next rows): worker S1 reaches `Notes { session: "S2" }` through the parent | + | an arriving hold for a served type whose params place it on another host | broken `invalid/placement`; the address the hold names only routes it | + | the hold names an address of a kind this host has a route for | over that route's link (the address's params checked against the kind, then `authorize`) | + | otherwise (a remote type) | over the parent link; a hold that arrived *on* the parent link is never sent back to it: `unavailable` | + | no parent link | `unavailable` | + + A `placement` or `authorize` that throws fails the hold broken with that error; an address whose params fail the + kind's check is broken `invalid/placement`. + +### The low-level host API + +```ts +// server.ts +const server = createHost

({ address: Server(), principal: { kind: "server" }, onError }); +server.route(Session, { + authorize: (p, address) => p.kind === "owner" || p.kind === "server" || (p.kind === "guest" && p.session === address.params.id), + resolve: async (address, context) => { + const child = fork("worker.ts", [], { serialization: "advanced" }); // no `signal`: see below + child.send({ id: address.params.id, init: await initFor(address.params.id) }); // the app's own start message + return { link: ipcLink(child), principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, +}); +await server.load([{ name: "@app/core", facet: coreServerFacet(config) }, ...serverPlugins], ctx); + +wss.on("connection", (ws, req) => { // an entry point: app or adapter code + const principal = authenticate(req); + if (!principal) return ws.close(4401); + server.accept(webSocketLink(ws), { principal }, ctx); +}); +radius.onChannel((ws) => server.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } }, ctx)); +``` + +```ts +// worker.ts: connect first (W6), then load; start data through the core facet (section 18) +const { id, init } = await startMessage(process); +const worker = createHost

({ address: Session({ id }), principal: { kind: "worker", session: id }, onError }); +worker.connect(ipcLink(process), { principal: { kind: "server" }, trusted: true }, ctx); // stores may reference server types +await worker.load([{ name: "@app/core", facet: coreWorkerFacet(init) }, ...workerPlugins], ctx); // serves SessionInfo +await (await worker.get(Harness, {}, ctx)).resume(ctx); // tasks run only after every plugin loaded + +// a browser tab +const tab = createHost

({ onError }); +await tab.load(webPlugins, ctx); +tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx); +const notes = await tab.get(Notes, { session: "S1" }, ctx); // tab → server → route(Session) → worker +``` + +```ts +// a test: everything in one process +server.route(Session, { + authorize: () => true, + resolve: async (address) => { + const [up, down] = memoryLinkPair(); + const principal = { kind: "worker", session: address.params.id } as const; + const worker = createHost

({ address, principal, onError }); + worker.connect(up, { principal: { kind: "server" } }, ctx); // W6: connect, load, resume + void worker.load([{ name: "@app/core", facet: coreWorkerFacet({ id: address.params.id }) }, ...workerPlugins], ctx); + return { link: down, principal }; + }, +}); +``` + +Do not pass `context.abortSignal` to `fork`: `host.close` aborts it, which kills the worker and makes Node emit an +`error` event that crashes an app without a handler. Chord closes the link at `host.close`; the worker exits when its +parent link closes. `resolve` returns the link before the worker has loaded (the worker connects first, W6): a +`resolve` waiting for the worker's `load` while the worker's facets hold server types would deadlock. + +| Call | Meaning | +| --- | --- | +| `createHost

({ address?, principal?, ...options })` | with an address: addressable, serves only objects placed there, acts as `principal` on its own host; without: a client. `P` is the app's principal type | +| `load([{ name, facet }], context)` | facets with their plugin names (the plugin's `package.json` name, npm style; duplicates and invalid names are refused); runs every facet's setup, then waits for the start barrier (section 21); a throwing setup rejects and closes the host; runs once. Loaders are adapters; base code passes its core facet the same way | +| `connect(link, { principal, trusted?, codec? }, context)` | the link to the parent, at most one open; once it closed (a drop), `connect` takes the next, and every remote reference over the parent is held again on it. `principal` = who the parent is | +| `accept(link, { principal, trusted?, codec? }, context)` | a link from a child; holds and calls on it act as `principal` | +| `route(Kind, { authorize, resolve })` | how to reach hosts of a kind (below); called before `load`, once per kind | +| `get(Type, params, context, { readyTimeout? })` | a reference once its object is live (section 7) | +| `hold(Type, params, context, { readyTimeout? })` | a reference at once that follows its object until released (section 7) | +| `close(context)` | closes the links (children, routes, then the parent), then ends every object at once and tears them down, the last to go live first (reason `close`); rejects with an `AggregateError` of the native closes that threw | +| `revoke(r, context)` | revocation (section 4; slice 4b) | +| `root(fn, …)` | a call scope for code outside objects (a Durable hook, a CLI); built in slice 8 (**decision needed** on its signature, section 26) | + +**Options of `createHost`:** `address`, `principal`, `principalKey` (section 4; the same function on every host), +`telemetry` (section 29), `onError`, `closeTimeout`, `readyTimeout`, `heartbeat: { ping?, close? }`; later `revokedBy` +(section 4, slice 4b) and `store` (section 10, slice 6a: the store, opened by the app with `openStore(backend, ctx)` +and closed by the app with `store.close(ctx)` after `host.close`; the host never closes it). **`onError`** receives everything that fails in the +background, as a tagged `HostError`: `listener_error` (a watch listener threw), `delivery_failed` (a frame delivery +threw), `activation_failed { object, failure }`, `close_failed { object, native, error }`, `close_timeout { object, +native }`, `drain_timeout { object }`, `protocol_error { error }`, `resolve_failed { address, error }`; later routine +errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), `object_failed { object, failure }` (a live object failing, slice 6a; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27) and `storage_failed { error }` +(slice 6a), which every app must handle: Chord never ends the process, +the app exits with its own code for its supervisor or reopens. + +**Timeouts** are host options; no fixed numbers in the code. Two host timeouts (C13, owner, 2026-10-22): + +| Timeout | Bounds | Host default | Override | +| --- | --- | --- | --- | +| `closeTimeout` | a native's `close` (then reported as a possible leak; the rest continues); the drain of an ending object's running handlers (callers are rejected at once) | 30 s | per native: `self.open(open, close, { closeTimeout })` | +| `readyTimeout` | an activation attempt (its wait for the previous instance's teardown included); a getter's whole wait; how long a call made while its reference is loading waits (the park deadline) | 60 s | per `get`/`hold`: `{ readyTimeout }` | + +The park deadline's per-type override is not built (only if a real case needs it). Timeouts of later slices +(routine handover 10 s, job `finish` retry backoff 1 s to 60 s, rollout install 60 s, provider-side `linger` per type) +belong to their mechanisms (**decision needed** on which stay host options, section 26). A route's `resolve` is +bounded by the app. + +- **Incarnations** are minted by Chord (uuidv7, code copied from pi-ai, no dependency), never passed in, and carried + in the link handshake (`hello`): the other side sees a restart as a new incarnation of the same identity; + references to it re-hold. +- **Entry points vary** (loopback or tunnel WebSocket servers, Radius dialing out and delivering channels, a DO edge, + memory), so the kernel has no `listen`; each entry point authenticates and calls `accept`. Radius no longer needs to + pose as a loopback request with a secret header. +- **App code, not Chord:** starting processes, init data, authentication, keep-alive decisions. Helpers come only when + apps repeat the same code. +- Compared with today's Chord: same shape (`createHost`, `connect`, `accept`, `route`, `load`); gone are string host + names and prefix parsing, app-managed generations, the duplicate `upstream` option, `opaque` runtime objects and the + loader inside the kernel. +- **Worker keep-alive** works as in p2: the worker counts, the app's supervisor decides. **`host.activity`** (slice 4b; + owner, 2026-10-24) is a JSON `Watchable` of `{ references, calls, jobs, lingering }`: holds arriving over links, + method calls running on this host, running jobs, provider-side lingering objects. Counters are kept where the work + happens; a change marks it dirty and it is published at most once per microtask turn with changes. Core code + reports it plus Durable tasks (the core `Harness`); the server's supervisor stops the worker when all are zero for + the idle time (p2: 5 minutes, checked every second). Chord provides the counts, no idle logic. + +### Routes + +```ts +host.route(Session, { + authorize(principal: P, address: Address): boolean; + resolve(address: Address, context: Context): + Promise<{ link: Link; principal: P; trusted?: boolean; codec?: Codec }>; +}); +``` + +- **`authorize`** runs for every hold routed over the route, with the hold's principal (section 4), before anything + starts; `false` fails the hold broken `forbidden`. +- **`resolve`** runs at most once at a time per address: the first hold starts it, later holds wait for the same + link. The returned principal is stamped on that link; `trusted` and `codec` as for `accept`. Its `context` is + aborted only when the host closes; a link it returns after that is closed at once. A `resolve` whose holds were all + released meanwhile finishes and its link is kept (a worker start that produced a link finishes even if its first + requester left). A `resolve` that never settles is the app's (it bounds its own `fork`); getters still time out. +- **Failure and re-resolve:** a rejected `resolve` (reported `resolve_failed`), a returned link already closed, or the + address's link closing while holds over it exist, starts the **per-address backoff** (1 s doubling to 60 s; a map + keyed by address, separate from objects' identity backoff). Meanwhile the holds are `loading`. Only a link that + completed its handshake resets the backoff. Without holds for the address nothing is resolved again, so passive + viewers recover without a user action while they hold. +- Chord never closes a route link itself except at `host.close`; stopping idle workers is the app's (`host.activity`). +- Route links have no app context yet: work arriving on them runs under the background context without telemetry + (slice 4b gives routes a context). + +### Links + +- **A link is delivery only:** an ordered, bidirectional channel of `Uint8Array` messages; adapters only move bytes + and close both ends. Session recovery is the adapter's policy. Implementations (slices 3a, 4a): + - `memoryLinkPair()`: delivers with `queueMicrotask` (no Node APIs); closing one end closes the peer a microtask + later. + - `ipcLink(port)` over a Node IPC channel (`fork` with `serialization: "advanced"`, or `process` in the child), + typed structurally (no Node import in Chord). A `Uint8Array` crosses as one message; other messages on the + channel (the app's start message) are skipped. It listens from the moment it is made and queues until its + listener is set. `send` returning `false` is backpressure, not failure; a thrown `send`, an error callback or + `disconnect` closes the link once; listeners are removed on close. + - `webSocketLink(socket)` over a WebSocket (`ws` in Node, the browser's), typed structurally: one binary frame per + message, `binaryType = "arraybuffer"`; messages sent while the socket connects are buffered until `open`; a socket + closing or closed already gives a closed link; `close`, `error` and a text frame close it. The transport's + `maxPayload` stays its hard cap (code 1009 closes the connection). +- **One link object per link carries both directions** (slice 4a): the channel (handshake, codec, checks, + heartbeat), a **provider side per origin** (holds and calls the peer sent, its own or those of a relay's downstream + link, by the peer's ids) and a **consumer side** (holds and calls this host's own references sent, by its own + counters). Message kinds are directional, so each goes to its side. A link carries nothing until a hold or call + (W6). +- **Roles:** `parent` (made by `connect`), `child` (`accept`), `route` (`resolve`). The role decides whose principal a + hold acts as (section 4). **Trusted links** are an explicit option (`trusted: true` on `accept`, `connect` and in + `resolve`'s result; default untrusted; never inferred from the link kind; owner, 2026-10-23): they skip only the + `MESSAGE` schema check; framing, the codec's decode, `checkParams`, `checkArgs` and Delta decoding still run. Trust + never grants the principal fields (they are read only on the parent link). +- **Remote objects belong to a path:** the parent, or a route address. Each new link of a path holds every remote + object of that path again (with new hold ids); a closed link drops its tables and the objects go `loading`. +- **Heartbeat on every link kind, in the protocol layer** (owner, 2026-10-23): each link keeps a `lastReceived` + timestamp (every received message; sends do not count); the host runs one interval timer for all its links (none per + message). A link silent for 15 s sends `ping` (after its handshake); the peer answers `pong` at once, never queued + behind the window. A link silent for 30 s, the handshake included, is closed (its references go `loading`). Both + numbers are host options (`heartbeat`). The heartbeat detects a silent transport (TCP does not notice a sleeping + laptop), not a peer that answers pings but withholds acks (bounded by the window and the pending cap instead). A + handler that blocks the event loop longer than 30 s gets its links closed (documented). +- **Flow control: an ack-clocked window per origin** (owner, 2026-10-23; per origin 2026-10-24). Per (link, origin), at + most **K = 2** of the provider's own `bound` and `state` messages are unacknowledged; only these two kinds count + (everything else is control, never queued behind them). The consumer sends `ack` (with the message's origin) after + applying each, after its batch; an `ack` crossing nothing in flight is ignored. While the window is full: + - each touched hold member keeps its ops in its **pending queue** (section 11: raw, never mutated, folded through + the leaf index, at most 10,000 ops, then only "replace with the current value"); when an `ack` frees a slot, one + `state` message carries every waiting hold's entry, its members' pending ops encoded once (or `[["r", value]]` + after an overflow), all reflecting the latest commit, still atomic for the consumer. So a slow consumer gets + coarser updates, never stale ones (measured: a `bufferedAmount` cap alone left state 9.4 s old behind kernel and + proxy buffers; acks kept it 0.0 s old); + - a hold that binds waits for a slot; its snapshot and subscription are taken in one synchronous step when its + `bound` is sent; bindings and waiting state take turns when both wait, so neither starves; + - a hold whose object leaves `live` while its `bound` waits must go back to pending; only a reload makes a live + object leave `live`, so slice 7 builds it (D1, owner, 2026-10-24). + Not built, by decision: a byte window, credit messages, snapshot admission or debt, an adapter `backlog()` API, + `reset` from anyone. +- **Calls: a call window N** (default 256, the same on every host) announced in `hello`: how many concurrent calls a + side accepts **per origin**. A side sends no call before the peer's `hello`. A host sends at most N of its own calls + per link in flight; further calls wait in the link's **call queue**, one FIFO, sent in order as replies free slots; + nothing is refused for being "too many" (K16). A slot is freed once per call: by its `result` or `error`, its + `abort` (when sent, at the caller; when received, at the provider), its hold's `release`, `gone`, its origin's + `closed` or the link closing. A call still queued when its hold rebinds (`loading`) is rejected `unavailable` (it may + name a method of the old variant). A provider seeing more than N running calls of one origin closes the link + (protocol error); a call naming a hold it does not know is ignored before it is counted (Q1). A per-link N, if ever + needed, must bound a relay's forwarding too. +- Holds are not limited by count (authorization and params checks protect them); size limits are a separate check + (section 22). Byte-heavy traffic (terminals) uses separate channels (section 23), so a large state message never + holds up keystrokes. + +### Relays (slice 4a; decision "Relay flow control per origin", owner, 2026-10-24) + +A host that receives a hold it cannot serve (the placement table above: a route or the parent) **relays** it: it holds +it again on that link (upstream) with the arriving hold's principal and incarnation, and forwards between the two +without parsing payloads. The provider keeps the window, the pending queues and the calls per origin; the relay never +drops, folds or resets. + +- **Origins:** the relay gives each downstream origin (a downstream link, and the origin its messages carry when that + link is itself a relay's) a number, one counter per host, never reused; 0 is a host's own work. Every message + upstream for that origin carries `origin`; the provider keys holds, calls, windows and acks by (link, origin, id), and + its answers carry the origin back. `origin` is honoured on every link (a worker may relay a child's hold to its + parent); principals and incarnations stay parent-only. +- **Ids are not renumbered:** an upstream hold has the downstream hold's id under the origin's number; calls the same. + Two tabs (two holds) of one principal are two upstream holds, each with its own encoders (R1). +- **Nothing is parsed:** the relay routes by the hold's address and authorizes it; params, arguments, results, + snapshots and ops travel as opaque parts. Cost per message: decode the envelope, swap the origin, encode the + envelope; parts copied once into the outgoing buffer. +- **What the relay keeps:** per relayed hold, its upstream link and its original `hold` message (params copied once), + which routes `call`, `abort`, `retry` and `release` upstream at once and holds again on the next link; per + downstream origin, the FIFO of the sender of each `bound`/`state` message in flight and the set of upstream links it + relayed to. +- **Forwarding up:** `hold`, `call`, `abort`, `retry`, `release` with the origin's number; a hold waiting for its + route is sent when the link exists; a `release` before that drops it. A call is forwarded at once; with no upstream + link it is answered `unavailable` (it crossed the `loading` notice). Each downstream consumer is bounded by its own + N, so the upstream sees at most N per origin. +- **Forwarding down:** a message from upstream with an origin goes to that origin's provider side, which forwards it + with its own origin (none for a tab), ids and parts unchanged. **A link answers only for what was relayed over it** + (owner, 2026-10-24): every entry of a `bound` or `state`, every `failed`, `loading` and `gone` by its hold, every + `result` and `error` by its call is checked against the relay's table; an answer for a hold or call not relayed over + the link it came from is dropped (Q1: also one crossing a release). A `bound` or `state` left with some entries is + forwarded with those; left with none, it is acknowledged to that link at once. A message for an origin from a link + the origin relayed nothing to is ignored whole. +- **Acks:** the relay forwards each upstream `bound`/`state` at once and returns its `ack` to the upstream link that + sent it when the downstream acknowledges it (the senders FIFO). A tab served by W workers may have 2W messages in + flight (plus the relay's own 2), each worker's window bounding its own. A slow tab stalls only its origin at each + worker; other tabs are other origins. Measured: a slow tab through the relay catches up like a direct one (a 60 s + stall: 0.38 s, 23.7 KiB at 64 KiB/s; before the rework, 24 s and three full snapshots). +- **Upstream drop:** the relay sends `loading { hold }` downstream for every relayed hold on the dropped link, fails + their relayed calls in flight `unavailable` (never resent), re-resolves and holds again upstream from the kept + message (same id, same origin); the new `bound` is forwarded. Acks for messages of the dropped link still in flight + downstream go nowhere. +- **Downstream drop:** the origin's provider side ends: its served holds are released, its relayed holds forgotten, + and every upstream link it relayed to gets one `closed { origin }`, which releases the origin's holds and aborts its + calls there; the origin's number is never used again; messages crossing it are dropped. +- **`gone`** from upstream is forwarded; the relayed hold is forgotten. Relays never make a `gone`. + +### Recovery + +- After a drop the consumer re-holds every hold of the path over the new link and receives snapshots (a `bound` on a + bound hold replaces its replicas in one batch, another variant allowed); calls in flight fail like on a disconnect; + calls made meanwhile park (section 7). +- A slow connection never ends an object: nothing releases a hold for falling behind (the reference count is unchanged). +- A link drop is a departure (section 19). + +### Values + +- **JSON plus `Uint8Array` leaves**, with a bytes schema type, defined independently of the transport. Every link + carries the same `Message` through its codec (section 22); parts are encoded at their origin; byte segments travel + raw, never base64. Same-host calls (no link) pass values by reference. +- Schemas are TypeBox and must describe data (no callbacks, no native types, no runtime codecs). Method arguments are + validated at the provider for every caller against closed schemas (section 5, R7, Q12, Q15); results and state are + never validated (section 30). Values are immutable by convention once handed over (Delta's rule): nobody mutates a + value it got, and nothing is copied or frozen to enforce it. There is no development mode (R9): one behavior + everywhere. + +--- + +## 4. Principals + +- The app defines its principal type `P`: JSON, every variant with a `kind`, a type alias (not an interface), + unversioned (owner, 2026-10-22): Chord never persists principals, base code changes every host at once, links stamp + them. Objects persist a projection with its own schema, never the principal (p2: chat's `Author { kind, id, name }`). + p2/p3: `owner` (`access: "local" | "relay" | "device"`), `guest` (`session`, `mode`, grant revision), `server`, + `worker` (`session`). +- **Links stamp principals.** A caller's own data never sets its principal. +- **The principal key** maps a principal to its object identity (`principalKey`, default canonical JSON; cached per + principal object; the same function on every host). It should include connection provenance and grant revision, so + a revoked grant's objects are distinct from a new grant's **(carried over)**. +- **Principals on holds** (slice 4a; owner, 2026-10-23): a `hold` carries `principal?` and `incarnation?`. The receiver + uses them **only on its parent link**; everywhere else they are ignored and the link's principal and peer + incarnation apply (a tab cannot claim to be the owner; a relay overwrites what a client sent). Calls carry neither: a + call runs as its hold's principal and with its hold's incarnation (refines R1/Q13, which stamped calls too). What a + host sends in those fields when it holds over a route link (towards a child that honours them): + + | The reference was made by | `principal` | `incarnation` | + | --- | --- | --- | + | `host.get` / `host.hold` | the host's own | this host's | + | `ctx.get` in a handler | the handler's `ctx.principal` (the caller's) | the handler's `ctx.incarnation` | + | `self.get` in a remote object | that object's principal | this host's | + | `self.get` in a local object or a facet | the host's own | this host's | + | a relayed hold | the arriving hold's principal | the arriving hold's incarnation | + + So a guest's call on the server whose handler `ctx.get`s a worker type reaches the worker as the guest and gets the + guest's variant; two references with different principals are two remote objects. +- **Upward is a boundary:** a worker's hold of a server type goes over its parent link, and the server ignores + principal fields from a child, so it acts as the worker, whoever the worker's handler serves. A worker that acts for + a user towards the server passes the user as data and the server object decides. +- **The caller's incarnation** (R20, Q13): `ctx.incarnation` is the incarnation of the host where the call started + (from the handshake, or from the hold's field on a parent link), next to `ctx.principal`, not part of the principal + or its key, never sent as data by apps; it is this host's own for local calls (section 25 uses it as a fence). +- **Revocation** (slice 4b) is a typed app function, not field matching: + + ```ts + type Revocation = { grant: string } | { device: string } | { session: string }; + const server = createHost({ + address: Server(), principal: { kind: "server" }, onError, + revokedBy: (p): Revocation[] => + p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] + : p.kind === "owner" && p.access === "device" ? [{ device: p.device }] + : [], + }); + server.revoke({ grant: "g1" }, ctx); + ``` + + `host.revoke(r)` ends every object and closes every link whose principal `p` has `r` in `revokedBy(p)` (compared by + canonical JSON of each value `revokedBy` returns, computed once per principal; never the host's own principal), and propagates `r` to the children as a `revoke` message on its route links, + honoured only on a child's parent link, where each applies its own `revokedBy` (the same app code; nothing is + replayed to a child connected later). Holders of a revoked object get `gone` with `forbidden/revoked` (owner, + 2026-10-24: a reason of its own, so a UI can say "access revoked"); holds still waiting for `load` are ended too. + Order: the `revoke` message goes out on the route links before the revoked child links close, so a worker's objects + end `forbidden/revoked`, not `released` by the tab's `closed`. Revocation ends a principal's objects at + once, whatever references or jobs remain (R11); their jobs get the abort signal and the normal `closeTimeout`. + There is no re-stamping: a changed grant (mode `write` → `read`) is a revocation of the old one; the client + reconnects and authenticates anew, and its references re-hold. Reconnecting never restores revoked authority. Work + started on behalf of a revoked principal in a shared store stops when its reconciling routine runs (R10's window, + accepted; section 25); there is no revocation hook. +- `local` objects have no principal in their identity; their handlers and references act as the host (section 5). + +--- + +## 5. Object types + +```ts +export const Notes = remote("notes", { version, placement, params, variants }); +export const NoteStore = local("notes.store", { version, params, members }); +``` + +| | `remote` | `local` | +| --- | --- | --- | +| reachable from | any host, across links | its own host only | +| identity | (type, params, principal key) | (type, params) | +| principals, variants | yes | no | +| factory | `(params, principal, self)` returning `{ variant, members }` | `(params, self)` returning the members | +| arguments | validated once at the provider for every caller (compiled closed schema, R7, Q12); unknown fields rejected; passed by reference on the same host | validated the same way (Q15), never copied | +| results | not validated; passed by reference on the same host | not validated, not copied | +| members | state (JSON) and methods (JSON plus bytes) | plain-data state and methods; no `opaque`, no callbacks | +| `ctx.principal` in methods | the caller | the host's principal (none on a client); references a local object makes act as the host | + +```ts +variants: { + editor: { state: { notes: NotesSchema }, methods: { add: method(AddArgs, NoteId), remove: method(RemoveArgs) } }, + reader: { state: { notes: PublicNotesSchema } }, +}, +// the factory chooses one variant per object (slice 3a, C15): +host.serve(Notes, ({ session }, principal, self) => + principal.kind === "owner" ? { variant: "editor", members: { notes, add, remove } } : { variant: "reader", members: { notes: publicNotes } }); +``` + +- **Names** (owner, 2026-10-27): object type names, host kind names, document (`defineState`) and job (`defineJob`) + names use `[a-z0-9@/._~-]`, lowercase, checked at declaration (they become SQL table names in the store, joined with `:`). +- **Keyed state members** (owner, 2026-10-26; `local` contracts only): `members: { notes: keyed(NoteSchema) }`; the + provider returns a keyed document handle; a consumer on the same host takes `ref.notes.at(key)` in its factory (a + counted pin: the key stays loaded while the consumer lives) or `ref.notes.watch(key, ctx)`/`get(key, ctx)` anywhere; + an absent key reads `initial()`, a retirement arrives as the `retired` flag (a derive over it carries the flag on). + Across links, a view per entity (`NoteView { session, id }`) is held as an ordinary remote object. +- **Methods:** `method()` (no arguments, no result), `method(args)`, `method(args, result)`, `method(undefined, + result)`. A method without arguments omits them in the API (`ref.bump(ctx)`, implemented as `bump: async (ctx) => + …`); `null` is their internal and wire form (owner, 2026-10-22). +- **Implementations** (owner, 2026-10-22): a state member is a JSON `Watchable` the factory returns (a document handle, + a derive, a reference member, `constant(...)`), never a getter; a method is a function `(args, ctx)` or `(ctx)`. + Activation checks what the factory returned (plugin code): exactly the variant's members, methods as functions, + state members as JSON watchables (a `NativeWatchable` or anything else fails the object `broken` naming the member, + Q14). +- **Reserved member names:** `then`, `release`, `state`, `retry` and `variant` are refused at declaration (a reference + has them; `then` would make an awaited reference thenable). +- **Compatibility (strict, like documents; R6):** a hold carries only the consumer's contract `version` and + fingerprint, never a description. The provider binds when both equal its current ones, or (slice 7) when the + version is one it accepts and the fingerprint equals the one computed from its own history (section 27). Every + contract change, additive ones included, bumps `version`. Anything else is `contract_mismatch` (or `updating` during + a rollout, section 21). The fingerprint covers params, variants and members, not placement. A consumer whose contract + does not fit gets a mismatch on its own binding; the provider object and its other bindings are not affected. +- **Variants:** a member name has the same kind (state or method) in every variant that has it, and a method the same + signature (arguments and result) in every variant; state schemas may differ per variant (redaction). Enforced at + the type level in `remote(...)` and at declaration at runtime. A remote factory returns one variant, `{ variant, + members }`; `ref.variant` is the discriminant (section 7). Calls are checked against the binding's variant + (`invalid/member` outside it). +- **Params** (owner, 2026-10-21/22): a flat record of required fields, each `str()`, `int()`, `bool()` or `oneOf(...)` + of string literals, constraints allowed (`str({ pattern })`, `int({ min: 0 })`); no defaults, optional, nested or + unknown fields. Checked for every caller with one compiled closed-object TypeBox check (a missing or extra field + fails `invalid/params`). The identity key is the field values in declared order, each escaped, joined with `|`. + Params are never copied or frozen: the object keeps the caller's object, immutable by convention. +- **Arguments (R7, Q12, Q15):** always validated with the provider's schema, whoever the caller is (trusted code is + not type-checked code: plugins are bundled without type checks), never with a schema a peer sent; object schemas are + closed (`obj()` sets `additionalProperties: false`), so an unknown field fails `invalid/args` instead of being + stripped; nothing is copied or mutated; arguments have no defaults (optional fields are `opt()`, handlers write + `args.limit ?? 50`). Same-host arguments are not checked for plain data (no prototype or getter checks): callers pass + plain data by convention. Provider schemas compile lazily at first use, cached in `WeakMap`s keyed by the schema + owner; accepted versions' validators also on first use. Bytes are leaves (`instanceof Uint8Array`, optional length + bound), never walked. Results and state are not validated. +- **Providers** (`bindProvider(Type, accepts)`, L1): runs `checkAccepts` once per load and reload and keeps the + validated mappings; per call `checkArgs(provider, method, raw, boundVersion, context)` returns a tagged value, + `{ kind: "method"; args; result? }` (current-shape arguments and, for an old-version caller, the result mapping back) + or `{ kind: "call"; args; call }` (the provider's own replacement of a removed method, invoked by Chord, slice 7). The + binding is held only by the served type's registration and by calls in flight (GC-tested; no global strong cache). +- **Errors** (L1, slice 3a): `ChordError` with a code and, for some codes, a reason, a closed union that is part of the + link protocol (adding one bumps the protocol version; a receiver rejects unknown reasons; a test pins the list): + `invalid` (`args`, `params`, `member`, `placement`, `reentrant`, `read_after_bulk`, `cycle`), `contract_mismatch` + (`updating`), `forbidden` (`revoked`), `gone`, `not_ready`, `released` (`ended`, `closed`, `settled`, `version`, `discarded`; none for a holder's own release), `too_large` (`part`), `unavailable` (`locked`, + `park_timeout`, `busy`, `storage_failed`), `aborted`. A failed check names TypeBox's instance path ("unknown field", "missing", else TypeBox's + message; keys containing `/` or `~` make the path ambiguous, documented). Plugins throw `PluginError(code, message, + details?)` for codes of their own; anything else crosses a link as the plugin code `internal` with its message, never + a stack or a cause; plugin errors never use Chord's codes. An error rebuilt from the wire is not copied and only its + envelope is checked; a malformed envelope is a protocol error. +- **Methods return data,** never objects; an object is handed out as its address and the receiver does `ctx.get`. +- **No callbacks as arguments,** on either kind: behaviour goes through registries, multi-step work through jobs, + input through flows, runtime services through core objects. Real schemas instead of `Type.Unsafe`. + +--- + +## 6. Objects + +### Lifecycle + +``` +(get) → activating → live → ending → ended + └→ failed (backoff) → activating … + live → failed (a runtime failure of the owner, slice 6a: broken, by reload or retry only) +``` + +- **One object per identity.** The first `get` runs the factory; later gets share the object. +- **Lifetime follows from kind and params (Q-A, owner, 2026-10-21):** there is no `lifetime` option and no lazy flag. + A `local` type with empty params lives as long as the host: started eagerly at host start, never ended by reference + count (it may hold persisted data, P3, and contribute to registries, R21). Every other `local` type and every + `remote` type lives while referenced (`linger` keeps it briefly after the last reference). No remote object has host + lifetime: its identity includes the principal, and long-lived parts belong in a local shared store. Below, + "host-lifetime object" means a `local` object with empty params. Running and finishing job runs keep their + object alive as a count, not as references; when the count reaches zero with no holders, the object ends (§15). +- **Activation** (one fresh attempt each time, bounded by `readyTimeout`, its wait for the previous instance's + teardown included): the factory runs; singleton documents take `initial()` and, on the object's first activation, + persisted documents load and its tables' definitions are checked and migrated (slice 6a; the deadline paused during + eager migration work); references are bound (before the + previous attempt's are released, so a dependency held only by it never ends in between) and waited for until live + and hydrated; natives open one after another; the returned members are checked (section 5); derives compute; + routines start (slice 5). The object goes live when all of that succeeded. Calls wait behind its **gate** until + then. +- **The seal fence** (owner, 2026-10-26): every attempt has a seal token (a job run invocation its own, for its state + changes and outcome). After the seal, new work through the attempt's handles is refused at use (`self.tx`, `ctx.get`, + a job or routine start); commits are checked when they enter the store's commit step (sealed: rejected `released`, + nothing written) and at adopt (sealed during the SQL: durable, adopted into the object's documents, staged job + changes abandoned); at the seal a transaction not yet in the step is aborted and the line released, a started step + finishes before teardown completes; cleanup (references, presence, activity counts) still runs. A pure swap is not a + replacement. +- **Ending** (slice 1, 2): new calls are rejected; every caller of a running call gets `released` at once; the + handler keeps running in the background under an aborted context and its `self.tx` throws `released` (a running + method can change nothing after the end). Each call runs under a context aborted by its caller and by the object's + end (one controller per attempt, linked until the call settles; a caller's cancellation aborts only its own call); a + call whose caller context is already aborted is rejected before its handler runs. Then the **drain** waits for the + handlers' own promises, bounded by `closeTimeout` (`drain_timeout` reported), natives close in reverse order, + documents are dropped, derives detached, and references released last (closes may use them). Routines stop and jobs + follow their rules (slice 5). +- **A live object can fail** (owner, 2026-10-26): a runtime failure of the owner (a `migrate` failing on access, a + stored version newer than the code) seals the live attempt and makes the object `failed` (`broken`): calls reject + with the cause, state members keep their last values, natives close, documents stay on the object; recovery by + reload or `ref.retry()`; live dependents stay live. The members freeze only after the commits admitted before the + seal adopted and published once, and a retry's factory waits for that (2026-10-27, slice 6a build reviews). +- **Waiting for dependencies (R18):** an object whose references are not live yet stays `loading` and goes live as + soon as they are (bounded by `readyTimeout`). A dependency that fails (`broken` or `transient`), or does not become + live in time, fails the dependent **`transient`** with the dependency's cause inside, never `broken`: the cause is + the dependency's. The dependent keeps its references while failed, and Chord installs their state as an automatic + retry trigger: the moment the dependency goes live, the dependent is retried at once (the remaining backoff is + cancelled; the backoff stays as a safety net). Factory authors write `self.get` and know nothing of this. When a + dependent's last reference goes, it ends and releases its dependencies. After `live`, a dependency leaving `live` + does not change the dependent: it stays live, and its calls that use the dependency get its error (or park while it + is loading). Only R18 failures keep references; any other failure releases them. +- **Dependency cycles are rejected (Q5):** every object instance has a uuidv7 instance id, minted at its first + attempt; an object that restarts gets a new one, so an old chain never matches (slice 4b, the first user; D4, + owner, 2026-10-24). On one host, at + `self.get(B)` during A's activation Chord follows the waiting edges of activating objects from B; reaching A is a + cycle. Across hosts, a hold made during an activation carries the activation chain of instance ids in its header + (bounded; ignored from untrusted links): the provider rejects the hold when the instance being activated is in the + chain (slice 4b; on one host slice 1; at most 32 entries; locally created objects inherit the activating chain). A + cycle fails the activation `broken invalid/cycle` with the cycle as cause ("A → B → A"), and every other activating + object on it the same way; no references are kept. A dependency failing with `invalid/cycle` fails its dependent + **broken** with the same error, an exception to R18 (owner, 2026-10-24: otherwise every backoff retry hits the same + cycle); dependents that are not on the cycle break too, and a reload or `ref.retry()` recovers them. Checked again + after a reload. Cost: O(chain) per declared reference, only during activation. + **Known limitation** (owner, 2026-10-24): a chain is fixed when its object is created, so a cycle entering an object + that is already activating is not detected. Example: A on the server and B on a worker start at the same moment, A + declares B and B declares A; the holds carry `[A]` and `[B]`, neither finds the other, both fail `transient` at + `readyTimeout` and retry with backoff (later attempts usually come in order and detect it). A probe along waiting + holds was not built. +- **Failed:** a broken part fails the whole object: a throwing factory, document hydration or `migrate`, a failed + open, a derive that throws on its first computation during activation, or a dependency that did not become live. + After the object is live, a throwing derive fails only its own member (`broken`, W1). The cause is tagged and decided by Chord from where it + failed: + + ```ts + cause: { kind: "transient"; error: ChordError } // an open threw, a reference was unreachable: automatic backoff + | { kind: "broken"; error: ChordError } // factory, derive, hydration or migrate threw: no automatic retries + ``` + + Only an `open` may choose `broken` itself (`throw broken(error)`, e.g. invalid configuration). **Transient** causes + get a **backoff per identity** (1 s doubling to 60 s; a map holding no object, so a failed object is collected after + its last reference while its window runs; concurrent gets share one attempt; success resets). **Broken** ones are + retried only by a trigger. Triggers (for both): `ref.retry()`, a plugin reload (slice 7) and wake signals the factory + declares with `self.retryOn(...readables)` (slice 8; anywhere in the factory: the factory specifies, Chord installs + the watches before natives open, so a failed open still has them). Every trigger works like `ref.retry()`: the + backoff window is skipped, an attempt starts now, holders are told `loading`. `retryOn` sources must live independently of the + attempt (registry values); a reference's member is a type error. A facet cannot fail + after `load`: its setup only declares (a throwing setup rejects `load`, slice 2; owner, 2026-10-26). + +### The factory principle + +```ts +host.serve(NoteStore, ({ session }, self) => { + const notes = self.state(NoteDoc); // section 9 + const index = self.state(Index); + const count = self.derive([index], ([i]) => i.ids.length); // section 12 + const watcher = self.open(async (context) => watch(dir, { signal: context.abortSignal }), (w) => w.close()); // section 13 + self.onLive(async (ctx) => { for (;;) { await sleep(HOUR, ctx); await cleanup(ctx); } }); // section 14 + const imports = self.job(ImportJob, { run, finish }); // section 15 + self.contribute(Http, { routes: { … } }); // section 17 + const info = self.get(SessionInfo, {}); // start data, section 18 + const catalog = self.get(Catalog, {}); // section 7 + return { index, count, add: …, startImport: … }; +}); +``` + +- Synchronous and declarative: no awaits, no side effects; async work happens in activation (opens), in calls, in + routines and jobs. +- Handles stay valid for their attempt's life (owner, 2026-10-26: documents and the line belong to the object; each + attempt's handles carry its seal token and definition version); nothing is "read at use" or re-fetched. +- On a reload the factory runs again for pure objects (new methods and derives, same documents); native + objects restart (section 21). + +### Pattern: shared store, per-principal views + +``` +NoteStore {S1} local shared store: the stored notes, once + ▲ self.get ▲ +Notes {S1} as owner Notes {S1} as read-only guest + variant "editor" variant "reader": derives a filtered view +``` + +Heavy or shared parts (a Durable watch, a prompt watcher, a sandbox, a store) live once in a `local` shared store; +per-principal `remote` views get it and expose what their principal may see. Authority is the view's; the store is +trusted and acts for the host. Views of the same variant read one value from the store (a redaction is computed once, +not per guest; section 12). Any plugin uses this pattern with the same primitives; base code is not special. + +**Worker-wide native owners:** the facet cannot `self.get` a type it serves, so an owner of a per-session native (an +inbox watcher, a sandbox) is a `local` object with empty params (host lifetime) that reads its session from the core's start +data (`self.get(SessionInfo, {})`, read in its opener). Paths are derived by the plugin from generic start data (the +data directory), never provided per plugin. +Such an object cannot `self.get` a session-scoped object in its factory (the session is known only after activation): +it gets one with `ctx.get` in a routine or method (R22). A `get` with params computed after activation is added only +if a real case needs it. + +**Natives that follow a relocation:** the native is a stable manager (`self.open` once); a routine on the session +location switches its internal resource (the watched directory) and bumps a generation, so late events from the old +location are dropped. + +--- + +## 7. References + +| Where | API | Released | +| --- | --- | --- | +| a factory | `self.get(Type, params)` | when the object ends | +| a call | `await ctx.get(Type, params, { readyTimeout? })` | when the handler settles | +| outside | `await host.get(Type, params, context, { readyTimeout? })` / `host.hold(Type, params, context, { readyTimeout? })` | `release()` | + +- **`host.get`** (and `ctx.get`) resolves with a reference once its object is live and **hydrated** (every state member + of its variant has a value). A broken failure rejects at once; through transient ones it waits up to its + `readyTimeout`, then rejects with the last failure's error; a getter that gives up releases its reference. + **`host.hold`** returns a reference at once that follows its object until released (loading, live, failed and + retried as its object is, loading again while its link is down) and never rejects; invalid params and a local type + not served here throw at once (before `load` resolved, these fail the reference broken once it did); its `variant` is + `undefined` until its first binding (slice 3b). There is no `state: false` option (C8) and no `optional` flag. +- **Narrowing by variant:** `ref.variant` is the discriminant (C15: no `inVariant`/`expectVariant`); `RefIn` + narrows to some variants (strict unions, never weakened). +- **Members across attempts** (2026-10-27, slice 6a build reviews): on the same host a reference's state member + follows its object's member across attempts, as a remote reference re-holds: one member source per object and + member name; when a new attempt goes live every member re-binds by name (watches get one diff frame from the last + value, consumers' derives recompute, also one that failed on the member's earlier error, which heals; a member the + new variant lacks fails `invalid/member`); while the object is + failed or activating again after having been live, reads and new watches return the last value and get no frames. + A re-exposed member freezes at the re-exposing object; the provider's own source is untouched. +- **Members:** each state member of a reference is a watchable of the provider's member (a replica over a link); + watches taken through a reference belong to it and end when it is released. Methods are functions (`ref.add(args, + ctx)`, `ref.bump(ctx)`). +- **`ref.state`** is a `NativeWatchable` (slice 3b) of a tagged value; watches get each new state: + + ```ts + | { kind: "loading" } | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // retryAt undefined: broken, no automatic retry + | { kind: "ended"; error: ChordError } // released by its holder, or its object ended + ``` + +| Situation | Calls | Values | +| --- | --- | --- | +| link drop | park until reconnect | readable (`loading`) | +| object loading or restarting | park until live | readable (`loading`) | +| rollout: provider ahead or installing | park until installed, with a deadline (slice 7) | readable | +| object `failed` | reject at once with the cause | `failed` with cause and `retryAt` | + +- A failed reference stays alive: while held, Chord retries per the backoff; `ref.retry()` tries at once (a Retry + button; over a link the `retry { hold }` message); success goes `loading`, then `live`. A provider tells a pending + hold's consumer each failure (`failed`, with the rest of its backoff window) and the next attempt (`loading`). +- **Parked calls** (slice 3b): calls made while the reference is loading are queued in order, each waiting at most the + reference's `readyTimeout` from the moment it was made (never renewed), then fail `unavailable/park_timeout`; they + are sent in order when it goes live (a new call never overtakes them) and rejected with the cause when it fails. + Otherwise a call ends only through its caller's signal; there is no per-call deadline. +- Calls already sent when a link drops fail `unavailable` and are not resent; calls running on an instance that ends + fail and are not resent. **Releasing a reference rejects the calls made through it** (`released`), locally and over + links (slice 3a). A consumer receiving `gone(h)` rejects its own pending calls on `h` (Q1). +- References survive link drops: one owner of a host's remote references holds them again on every new link of their + path; a re-hold may bind another variant (slice 3b). +- **Consumers** handle a failed call as they handle a disconnect (error shown, user retries); Chat's own submission + `requestId` stays an app concern. + +- **Optional dependencies** are registries owned by the dependent, not optional references: ports contributes its + forwards to `radius.forwards`; without ports the table is empty and radius works. + +- **Authority checks** live in the per-principal view's methods (and in `route.authorize` before a host starts); + there is no async activation guard. An object for an address that carries authority (`LoginFlow { id }`) may exist; + what a caller can see and do is decided by its view. + +--- + +## 8. `ctx` + +Every handler gets a `ctx` (a `Context`): `telemetry` (the current span, section 29), `abortSignal` (the caller +cancelled or the object ended; there is no per-call deadline), `principal` (the caller as its link stamped it; on a +local object's handler the host's principal, none on a client), `incarnation` (the calling host's, section 4), and +`get(Type, params, { readyTimeout? })` (references as the caller sees them, released when the handler settles; a +`get` after that throws `released`). Later: `ctx.trace(name, attributes, fn)` gives `fn` a child `ctx` under a child +span (slice 9, cut C11 to decide); in a job's `run` also `jobId` and `state` (slice 5). Pass `ctx` to everything you wait +on. Code outside objects gets a `ctx` from `host.root` (slice 8). Durable's `Context` is the same module +(`../context`); bridging is explicit. No code versions, no request ids. + +**Contexts are required** (owner, 2026-10-22): every function that takes a context requires it, as the parameter after +the required ones (context last); optional only on error-reporting paths. A getter that cannot take one (a lazy derive +read) uses its owner's. + +**Call-scoped natives** (a native that lives exactly as long as one call: a temporary SSH session for a directory +listing, a file handle for a download) are opened and closed inside the method with `try/finally`; there is no +`ctx.open`: + +```ts +list: async ({ path }, ctx) => { + const session = await ssh.open(config, ctx.abortSignal); + try { return await session.readDir(path, ctx.abortSignal); } + finally { session.close(); } +}, +``` + +--- + +## 9. Documents + +### Definition and handles + +```ts +const Settings = defineState("settings", { + schema: SettingsSchema, + version: 1, + initial: () => ({ theme: "dark" }), + persist: true, // default: in memory + checkpointWhen: () => true, // required for persisted documents; small: a base per write +}); +const Draft = defineState("draft", { + schema: DraftSchema, + version: 2, + migrate: (old, from) => ({ body: old.text }), // pure + initial: () => ({ body: "" }), + keyed: true, // many per object + persist: true, + checkpointWhen: (value, ops, info) => info.deltasSinceBase > 50, +}); + +host.serve(DraftStore, (_, self) => { // local, empty params: may persist (P3) + const settings = self.state(Settings); // THE settings document + const drafts = self.state(Draft); // all Draft documents of this object + return { + settings, // a singleton handle is an exposable Watchable + get: ({ conversation }, ctx) => drafts.get(conversation, ctx), + save: ({ conversation, body }, ctx) => + self.tx(async (tx) => { + const d = await tx.draft(drafts, conversation); + d.body = body; + return d.body.length; + }, ctx), + drop: ({ conversation }, ctx) => self.tx(async (tx) => tx.retire(drafts, conversation), ctx), + }; +}); +``` + +- **Ownership:** a document belongs to exactly one object identity, and so does its line (owner, 2026-10-26: across the + object's attempts, retries, failures and native restarts; an attempt keeps only its natives, routines, derives and + seal token) (persisted ones across restarts; in-memory ones for + the object's life, across reloads and native restarts). Facets have no documents (owner, 2026-10-26: C16). + Others see it only through exposed members. References to other objects are stored as data (addresses). +- **Singleton** (no `keyed`): loaded at activation, kept in memory while the object lives; `handle.value` is + synchronous; the handle is an exposable `Watchable`. +- **Keyed:** addressed by a short key local to the object, loaded on demand; `await handle.get(key, ctx)` reads (creates + nothing); `for await (const key of handle.keys(ctx))` pages keys (both take a context, owner, 2026-10-26) (indexed per owner and definition, also in memory). No + live key list: a collection shown live is a singleton (an object holding a list or map); large collections are + methods returning pages. +- **Values are immutable by convention** (Chord Delta's rule, as in Durable): `handle.value` and published values + are shared, never mutated by their readers, and never copied or frozen; a draft belongs only to its transaction. + +### Writes and transactions + +- **Only `tx.draft`** (as Durable's `tx.doc`): `await tx.draft(handle, key?)` inside `self.tx(fn, ctx)` + returns a draft of the current document, or a draft over `initial()` if it is absent. Reading is drafting (Delta's + overlay: no copy). An unchanged draft (empty ops) commits nothing, so an absent document is never stored by being + read. `tx.retire(handle, key?)` removes a document. `keyed` makes the key mandatory at the type level. There is no + `tx.get` for documents (tables have one, section 10). +- Drafts record ops (assignment, `push`, `Object.assign`); the same document returns the same draft within one + transaction. **Drafts are revoked when the transaction settles (R12, as in Durable):** a transaction writes and + returns at most plain values (an id, a count), never anything taken from a draft. To return data, read after the + commit from the document (`handle.value`, `handle.get`), which then holds the committed revision; a later commit + may slip in between, and the reader gets that newer value. +- **No existence check inside a transaction (R13, as in Durable):** `tx.retire` of an absent document is a no-op (it + never creates); `tx.draft(handle, key)` creates a missing document when the draft changes; + whether a keyed document exists is read outside the transaction (`handle.get(key, ctx)` gives `undefined` for an absent + one). **A singleton always has a value (Q11):** `handle.value` is the written value or `initial()`, never + `undefined`, so an exposed singleton is typed `T`; singleton existence has no API (unlike Durable's `snapshot()`). + + Patterns: + + ```ts + // Return data: write in the transaction, read after the commit. + await self.tx(async (tx) => { (await tx.draft(notes)).items.push(note); }, ctx); + return notes.value.items; // the committed revision (or a newer one) + + // Remove if present: retire is a no-op for an absent document. + await self.tx((tx) => tx.retire(drafts, conversation), ctx); + + // Create if missing: the draft creates it (only if it changes). + await self.tx(async (tx) => { + (await tx.draft(drafts, conversation)).text = text; + }, ctx); + + // Branch on existence (keyed documents): read first, outside the transaction. + if ((await drafts.get(conversation, ctx)) === undefined) { … } + ``` +- Document roots are `JsonObject` (as in Durable; lists live in a field); documents hold JSON only (bytes are a value + and wire type). A singleton that does not exist yet reads `initial()` through its handle; retiring resets it to + `initial()`. Handles may be used only in transactions of their own object. +- **Commit sequence** (Durable's): wait for the object's line (per object, never across objects; reads outside a + transaction do not take it exclusively); run the callback; an error or an abort before the storage write discards + everything; prepare every draft (Delta copies and checks only the placed values); write persisted documents in one + storage batch through the store's commit step, `await store.commit(writes, adopt, ctx)` without the caller's abort + signal (deltas, or a base when `checkpointWhen` says so; the line held until the step returned, as Durable's + `#runCommit`, owner, 2026-10-26); adopt by pointer swap and mark dependents in the step, persisted and in-memory + documents and staged changes in one `host.batch.commit` (any backend error in the step is `storage_failed`, + Durable's rule, owner, 2026-10-26: plugin mistakes are refused when a write is staged; a seal-fenced commit is + rejected `released` before `BEGIN`); then the owner's synchronous flush recomputes dirty derives, sends one message + per link and queues watch callbacks (W1); drafts then throw if touched. In-memory-only commits take no store seq and never + enter the commit queue (W12). No + revalidation of the whole document, no copy, no freeze. +- **Nothing leaves a transaction before it commits (S2):** ids and data reach other owners only after the commit; + reference what is committed (a reference to a later commit dangles on every replica that stops in between). +- **Durable's rule:** inside the callback only `await tx.*`: no other objects, models, processes or network, no second + `self.tx`, no reads past `tx` (otherwise a hang, not an error). Fetch first; decide and write inside. +- A transaction covers only its own object's documents and tables: one line, no transactions across owners (facet + documents were removed, owner, 2026-10-26; shared plugin state is a `local` store object's, reached by methods). +- **Shorthands:** `handle.change(fn, ctx)` (keyed: `handle.change(key, fn, ctx)`) (one document, loaded before the synchronous `fn`), + `handle.retire(ctx)` (keyed: `handle.retire(key, ctx)`); several steps atomically: `self.tx`. + +### Versions + +- A shape or meaning change raises `version` with a pure `migrate(old, from)`. A document migrates on access by the + new definition (in memory too); the migrated base is persisted with the next commit. +- **Schema fingerprints instead of value validation:** `defineState` computes the fingerprint of its structural schema + once (descriptions and titles ignored) and keeps it on the token; every document record stores the fingerprint it was + written with (at creation, and with the new base after a `migrate`). On access Chord compares the token's fingerprint + with the record's: the same version with a different fingerprint fails the object `broken` ("schema changed without + a version bump"). Always on, O(1). Every structural schema change needs a version bump (an additive change gets a + trivial `migrate`). Values are not walked against the schema in production, with two exceptions (R8): + - the result of `migrate` is validated against the new schema once, when it is produced (per document per version + change, off the commit path); a failure refuses the access, keeps the old data and fails the object (slice 6); + - `initial()` is checked once per definition, at its first value (D3, owner, 2026-10-24: `initial()` takes no input, + so a wrong shape is wrong at its first call; checking every call would cost one validation per created document + and catch nothing more). A failing singleton `initial()` fails its object's activation (`broken`); a failing + keyed one fails only the transaction that drafts the absent document, naming the definition (slice 1). +- Old code whose definition is older than the document throws on access. Without `migrate` the access is refused and + the data kept. A changed meaning without a bump cannot be detected. +- Singleton documents migrate during activation and after reload (hydration counts as access). A hydration or + migration failure keeps stored data and fails the whole owning object, not only the exposed member. Keyed + documents migrate on their next access; a migration failure there also fails their owner. A migrated base is written + back by the next commit that changes the document (owner, 2026-10-27); until then a load migrates again (once per + object, since documents stay on the object). + +### Lifetime, persistence + +- **No tags (owner, 2026-10-21):** deleting a session deletes its store and directory; instance data on the server that + refers to a session is a table with a `session` column, cleaned up by its owner (`deleteWhere`, reconciling from the + catalog). +- **In-memory documents are instance state (P2, owner, 2026-10-21):** attached to the live object, with no storage, no + address, no seq and never in the outbox; their commit is prepare and adopt (inside the store's commit step when the + transaction also writes persisted data). +- **Lifetime:** in-memory documents end with their object (last reference released, linger expired, a params change on + reload) and survive reloads and native restarts of a live object; long-lived objects creating many keyed in-memory documents retire + them themselves. Persisted documents stay until retired. +- **Persisted data only in `local` objects with empty params (P3, owner, 2026-10-21):** persisted documents and tables + live in `local` objects with empty params (one per host and type); their address is (host kind, object type, name, + key), never a host instance such as the session id (W7), + so a params change never orphans data. Per-entity data is keyed documents or table rows there, with an app-chosen + key; objects with params keep only in-memory state and read persisted data by key. Renaming a persisted store's type or + definition, or changing how keys are formed, is an explicit one-off migration in the store. + Why not elsewhere: a `remote` object's identity contains the principal key (provenance, grant revision), so its + persisted documents would scatter across access paths and be orphaned after a reconnect; a shared storage key + (`storeKey`) would put two objects with two lines on one document, so it does not exist; params would orphan data + on a params change. Enforced at the type level (`self.state(…, persist)` and `self.table` only in factories of + `local` types with empty params) and as a declaration error at runtime ("persist needs a stable identity: put the + documents into a local shared store"). +- **Durable plugin state:** whatever must survive a host teardown (crash, restart, a DO eviction) is a persisted + document; everything else is rebuilt on activation (in-memory documents, natives, derives, routines, presence, + references re-held by clients; job runs end `lost` unless their records persist). + +### Rules across owners + +- **No plugin reads or writes another plugin's documents** (Chord or Durable). Data crosses plugins only through + state and methods. Data core enforces lives in a format core owns and is handed to core through its API when the data is + created (`createConversation` with `init`), never written into core's documents directly. +- **Invariants across owners** are enforced by the owner of the dependent data, which reconciles from the other + owner's state with an idempotent routine. No cross-owner transactions, no "notify the other owner" calls that can + be lost: + + ```ts + // ports' ForwardStore: "a forward of an archived session is closed" + const catalog = self.get(Catalog, {}); + self.on(catalog.sessions, async ({ value: sessions }, ctx) => { + for (const f of forwards.value.list) if (sessions[f.session]?.archived) await close(f.id, ctx); // idempotent + }); + ``` + + After a crash the routine runs again with the current value; sessions does not know ports exists. + +--- + +## 10. Store and storage + +> **Superseded (owner, 2026-10-21):** persisted data lives in one transactional store per session, designed in +> `packages/chord/docs/next/store/book.md` and decided in the log ("The store", S1-S17, W1-W14, P2-P4, "Flat columns", +> "Query runner", "Index definitions are recorded", "Blobs and derived data"). In short: backends SQLite (Node, Durable +> Objects, sqlite-wasm in the browser; tests on SQLite `:memory:`, no hand-written memory backend, owner, 2026-10-27); real tables owned by objects with empty params (P3), flat +> columns (scalars, every `str()` as plain UTF-8 `TEXT` in code point order with lone surrogates replaced and NUL refused (owner, 2026-10-26, superseding W8), `json(...)` with its own version and on-access migration), a small +> query language, writes `put`/`update`/`delete`/`updateWhere`/`deleteWhere`; one documents engine for Durable +> (`StoreStorage`) and Chord; one commit queue per store that writes, adopts and marks (S1, W1), with `Storage.commit( +> writes, adopt, context)` in Durable; an outbox of ordered operations in the same transaction (no compaction at first); +> eager migrations by scan with row `v` for eager steps only. In-memory documents are instance state with no storage +> (P2). The text below describes the per-object lines, Delta trackers and commit sequence, which stay; its storage +> interface is replaced by the store. + +- **What stays from Durable's store design** (`durable/src/session/transaction.ts`, `session.ts`): one Delta tracker + per document held by its object, prepare and adopt by pointer swap, ops per commit, lines per object, migrate on + access, `checkpointWhen` (required for persisted documents, R14), retire, and the documents engine's invariants (create before change, no + change after retire, one live incarnation per address, `deltasSinceBase`). +- **Where persisted documents live:** rows of the store's documents engine (S13) with address `(namespace: "chord", + owner: ":", kind: , key)` (key `""` for a singleton), never a host instance (W7); ids from the store's + counter (S10). Chord ships no storage of its own: no `DocumentStorage` interface, no Chord SQLite or memory document + storage and no Chord storage conformance suite (the store's conformance suite covers the engine); the frozen `next/` + storages are not ported. The store is passed to the session's host by base code (owner, 2026-10-26): + + ```ts + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(dir, "session.sqlite"))), ctx); + const worker = createHost

({ address: Session({ id }), principal, store, onError }); // tests: SQLite :memory: + … + await worker.close(ctx); + await store.close(ctx); // the app owns the store; in 6b the Durable Session shares it + ``` + + `openStore` runs the store's own migrations, checks the file is UTF-8 and reads `lastSeq`, the definitions and the + id counters once; `sqliteBackend(db)` takes any `SqliteDatabase` (Node, Durable Objects, sqlite-wasm later); the + backend interface is internal; `importStore(backend, archive, ctx)` takes the same backends. No `openNodeStore`. +- **Loading at activation (owner, 2026-10-26):** persisted singletons load right after the factory, sequentially, + all singletons of the object with plain reads (owner, 2026-10-27: the object holds its line, so no read transaction + is needed), once per object, before its references are bound: find, load (base plus + deltas), fingerprint check, `migrate` with its result validated once (R8). A storage error is `storage_failed`; a + fingerprint mismatch, a throwing `migrate` or an invalid migration result fails the object `broken`. +- **Statements per commit (owner, 2026-10-26):** the store keeps `lastSeq` (the seq of its last commit) in memory (read at open, + persisted with each commit, S9) and folds the documents engine's checks into the writes (unique keys, a partial + unique index on the live address, conditional `UPDATE … WHERE id = ? AND version = ?`): a delta commit is about three + statements (revision, the record's counted conditional `UPDATE`, `lastSeq`), plus one `INSERT` with an outbox. Durable's ids in the store: `StoreStorage` keeps a `record_ids` table (one + namespace across Durable's five tables), its counter at open `coalesce(max(id), 1) + 1`. +- **Tables in a transaction (owner, 2026-10-26):** the table handle first, keys as objects with exactly the key + columns, reads async, writes synchronous and buffered like a draft's ops (checked and staged, no I/O; written in the + commit step or dropped on failure); no write shorthands on the handle: + + ```ts + self.tx(async (tx) => { + const old = await tx.get(edits, { id }); // overlays own put/update/delete (W10 after bulk) + const { items, next } = await tx.page(edits, q, 50, { after }); // committed rows, stable for the transaction + tx.put(edits, row); + tx.update(edits, { id }, { at: now }); + tx.delete(edits, { id }); + tx.updateWhere(edits, (e) => prefix(e.path, from), (e) => ({ path: replacePrefix(e.path, from, to) })); + tx.deleteWhere(edits, (e) => lt(e.at, cutoff)); + }, ctx); + await edits.get({ id }, ctx); // outside: committed state, ctx last + ``` +- **The outbox (owner, 2026-10-26):** optional, `openStore(backend, ctx, { outbox: true })` (an omitted option keeps the + persisted setting; switching needs an explicit `true` or `false`), persisted in the store's + metadata (turning it on starts sync from an archive at the current `lastSeq`); a table in the store's backend, + written in the commit step's transaction (atomic for one `INSERT`); exposed as `store.outbox.read(afterSeq, limit, + ctx)` -> `{ seq, writes }[]` and `store.outbox.trim(throughSeq, ctx)`, no pluggable encoding; versioned by the store's + schema migrations (its layout) and a `format` per outbox commit (the write vocabulary, the archive's `format`; older + formats stepped forward on read, newer ones refused). +- **Writes, commits, batches, archives (owner, 2026-10-26):** a *write* (`StoreWrite`, tagged by `kind`) is one change + of the store (`put`, `update`, `updateWhere`, `table`, `index`, `document.create`, …); a *commit* the writes of one + seq in the outbox; a *batch* the commits `from..to` that sync uploads and a replica applies; an *archive* a manifest, + the writes from 0 to `lastSeq` and an `end`. `lastSeq` is the seq of the store's last commit. +- **Export and import (owner, 2026-10-26):** `store.export(ctx)` yields the archive's parts, the state after its + `lastSeq` as writes (manifest, `table`, `index`, `put`, `document.create`, `end`), holding the commit queue + until the iteration ends or `ctx` aborts; `importStore(backend, parts, ctx)` applies them into an empty backend in + one transaction with the replica's applier. Lossless (owner, 2026-10-26): every `put` write carries its versions (`$v`, + each `json(...)` column's) in the outbox and in archives, a `table` write its state, a `latest` document its current + value; the applier never infers a version. The encoding is pluggable; plain JSONL ships first. Sync is not export: + it ships outbox batches as commits happen. +- **`json(schema, { version, migrate? })` columns (owner, 2026-10-26):** schema and version required, `migrate` + optional and pure, on access (W11), its result checked once (R8); without it (or failing) the read is refused and + the owner fails `broken`; each column's version and schema fingerprint are recorded in the table definition and + compared at activation (same version, other fingerprint: `broken`, D9); a read never writes back. +- **Table versions: `evolveTable` (2026-10-27, the second spec review's majority):** a table's history is a base + (`defineTable`) and one `evolveTable(prev, (t) => t.rename(…).add(…).rows(…).drop(…).index(…))` per version, each + materialized and fingerprinted at declaration, as contracts' `evolve`; the store replays the recorded version's + successors in order (DDL and `migrating`, the data step, `ready`); key columns are immutable in `set`. +- **Eager migrations and addresses (owner, 2026-10-26):** the activation attempt's `readyTimeout` deadline does not + run while an eager migration step runs (getters still time out by their own); a host without an address uses `client` + as its host kind in store addresses. +- **Eager `rows` migrations** commit in chunks of 100 rows (owner, 2026-10-26; a fixed constant: about 1 ms of commit + step per chunk, the total time is the same for larger chunks, and every other owner's commit waits for the step). + Between chunks the migration yields one macrotask without `setTimeout`'s 1 ms clamp: `setImmediate` where the + runtime has it, otherwise a `MessageChannel` message (2026-10-27, slice 6a build reviews). +- **Eviction as in Durable,** for keyed persisted documents only (persisted singletons and `at(key)` documents stay + in memory while their object lives: `handle.value` is a synchronous read; owner, 2026-10-26): one not read, written or watched for the retention + period (default 10 minutes) leaves memory and reloads on the next access. A storage error ends the store and closes + the host; `onError` receives `{ kind: "storage_failed" }` and the app decides (section 3). Any backend error in a + commit step, or on a read whose `ctx` was not aborted, is a storage error (Durable's rule); later calls reject + `unavailable/storage_failed`. + +--- + +## 11. Kinds of state + +| Kind | For | API | +| --- | --- | --- | +| document | data the object owns | `self.state`, `self.tx` | +| derive | computed values | `self.derive(sources, fn)` | +| Durable's views | the conversation view, task graph and Durable documents | `Watchable`s through Durable's watches; feed derives and state members | +| paged method | large or queryable collections | a method returning pages (chat history) | +| table | a plugin's own records with queries (owner, 2026-10-21) | `self.table`, the store's query DSL | +| plugin database | derived data only: full-text search, caches (rebuilt, never synced) | a native opened by its owner | + +App state lives in Chord documents; only transcript-bound data (entries, documents that fork and rewind with a +conversation) lives in Durable. + +### Watchables (owner, 2026-10-21; log "Watchable and one watch API", W1-W4) + +There is no `replicatedState` in v2. Everything observable is one of two kinds, sharing one lifecycle: + +```ts +interface Watchable { // JSON only (no bytes); documents, derives, reference members, constant(v), Durable's views + readonly value: T; + watch(ctx: Context): Promise>; // listener (call, ctx), call: WatchCall (below) +} +interface NativeWatchable { // self.writable(initial) with set(), composed registry values + readonly value: T; + watch(ctx: Context): Promise>; // listener (call, ctx), call: NativeWatchCall; a change is a new value (!==) +} +type WatchCall = // owner, 2026-10-27: a recoverable failure is a call, the watch stays open + | { kind: "value"; value: T; ops: Op[]; retired: boolean } + | { kind: "failed"; error: unknown }; +type NativeWatchCall = { kind: "value"; value: T } | { kind: "failed"; error: unknown }; +// const w = await notes.watch(ctx); send(w.value); +// w.start(async (call, ctx) => { if (call.kind === "failed") return showError(call.error); render(call.value); }); +// … await w.stop(); +``` + +- **Failures are calls, watches end only for good (owner, 2026-10-27):** a source that fails and can recover (a + derive whose compute throws, a replica member a hold lists as failed, a member a new variant lacks, a registry whose + `compose` throws, slice 8) gives its watches one `{ kind: "failed", error }` call and keeps them open; its next good + value arrives as `[["r", value]]` (folded with the failure for a busy listener). A failed derive recomputes on every + later change of its sources and on a re-bind (self-healing). A watch ends only when nothing can come back: its owner + ended (`ended`), `stop()` or a cancelled context, a throwing listener (`listener_error`), `storage_failed`. `self.on` + handlers are not called for failures: a routine waits for the next good value (the failure is reported where it + happens: the derive's owner reports `derive_failed { object, member?, error }` once per failure episode, the first + failure after a value or after creation, not for a derive failing only because a source failed). `ref.state` stays + a separate watchable of the object's reachability (`loading | live | failed | ended`), with the same words. A + replica member that vanished with a variant is kept and comes back with a later binding; while failed it accepts + only `[["r", value]]`. +- **Watch handles are Durable's, moved into Chord** (Durable imports them): `watch()` atomically takes the current + value and buffers later commits; `start(listener)` installs the one listener; commits are delivered in order, async + listeners awaited; a listener still busy with an earlier frame gets the commits made meanwhile as one call, from + the **pending queue** (owner, 2026-10-23): ops stay raw and are folded through a leaf index (a map from path to + the last pending append or primitive set; a new one on that path merges into it; any structural op clears the + map; `r` drops everything before it); past 10,000 pending ops only "replace with the current value" is kept (no + size, text or time caps; no folding of list appends or trims; ops come only from Chord's tracker and + `diffRevisions`, so there are no defences against hand-written ops). `PendingOps` is Chord's shared Delta code. The + same queue holds a link hold's ops while its window is full (section 3). Natives: the newest value. Durable's + event stream keeps its own queue (events are not state); `stop()` is idempotent, drops pending frames, never aborts or awaits a running call; `stop()` and + `closed` return the end, tagged by `kind` (owner, 2026-10-27): `stopped`, `aborted` (its context aborted), `ended`, + `listener_error { error }`, `failed { error }` (a final failure; a recoverable one is a call). Durable's synchronous + observers take the same shapes: `advance(call, ctx)`, `end(end)`. + **Retirement (owner, 2026-10-26, option A; replaces W2a's `[["r", null]]` frame and the end `retired`):** one + mechanism for Chord and Durable documents, singletons and keyed alike. A watch follows its document's address; a + retirement is one last call with a `retired` flag (a Chord document: `initial()` with `[["r", initial()]]`, which a + wire replica applies; a Durable document: the last good value, no ops); the watch goes on, and a later + write delivers the new incarnation as `[["r", value]]`. A watch ends `ended` with its owner (the object of a Chord + document; for Durable the task of a task document, right after its retirement call, or the Session). Listeners take + one object before the context (owner, 2026-10-26; tagged since 2026-10-27, above): `(call, ctx)`, a `WatchCall` or + `NativeWatchCall`. + A retirement call carries the address's value after the retirement: `initial()` for a Chord document, the last value + (no ops) for a Durable one, which has none while absent. Watching an absent address gives `undefined`. On the wire a + `state` entry lists retired members like failed ones. A per-key source in a factory, `handle.at(key)`, is a singleton per key: loaded at activation, `initial()` while absent, usable as a state member, derive source and `on` source (owner, 2026-10-26). A + stopped watch releases every reference it held (W3). +- **Only `Watchable` can be a contract state member or go on the wire**, by type and by a kind check when the + object is served (Q14); a `NativeWatchable` never can, even holding JSON. Mutating a native value in place notifies + nobody. +- **Publication (W1):** the commit step writes, adopts (pointer swaps, Durable's touched views dispatched by id) and + marks (dirty derives; ops appended by reference to each touched hold's batch and each watch's queue). Right after + the step, synchronously, the owner's flush recomputes the dirty derives someone holds or watches (dependency order), + encodes one entry per touched hold into one `state` message per link and origin, and queues watch callbacks. A + throwing derive fails only its member; reentrant `self.tx`/`watch`/`release`/`get`/`hold`/`load`/`close` throw + `invalid/reentrant`; a delivery or flush that throws is reported (`delivery_failed`) and the others still run. The + same boundary runs without a store seq for in-memory commits, `writable.set`, registry changes and inbound wire + batches (a `bound` or `state` message is applied as one batch before listeners run). There is **one batch per host**, + passed to its sources (no module-level singleton); a dirty derive nobody holds or watches recomputes lazily on its + next read. +- **Consumers:** derives and the wire use the internal flush; app code uses watches; `self.on` is a watch owned by + the object (first call with the current value, then every commit; Chord awaits a running call on reload); clients + get ordered ops over the link, folded by the provider's pending queue when they fall behind (section 3); a re-hold + gets a fresh snapshot. + +--- + +## 12. Derives + +`self.derive(sources, (values) => value)`: a synchronous pure function, recomputed when a source changes and +diffed into ops for the wire. Sources are `Watchable`s and `NativeWatchable`s (documents, other derives, reference +members, Durable's views, registry values); the result is always JSON (state is JSON only; bytes only in method arguments and results). A derive is marked dirty in the +commit step and recomputed in the owner's flush right after it, so its change travels in the same message as its +source (W1); a dirty derive nobody holds recomputes on its next read. A throwing derive fails only its member +(`broken`). There is no `previous` (W14): `mapShared` keeps unchanged elements identical. + +- **Built on what exists:** derives sit on Chord's internal publication primitive (not today's `cell.ts` or + `replicatedState`); the diff is + Delta's `diffRevisions` called directly (identity short-circuit: unchanged shared subtrees cost one `===`; bytes are + atomic leaves compared by identity; equality never by stringify). +- **Container width (R24):** structural sharing is O(change) along a path, not across a container's width: changing + one item of a flat array copies the array's slots, and `mapShared` visits every element (one identity comparison + each). Large collections are keyed documents or bounded windows; a benchmark of long arrays (allocations, element + visits per change) sets the documented limit. +- **The footgun:** a derive that builds new objects for unchanged data makes every commit cost O(size of the value), + per view, because the diff must compare contents. Unchanged parts must be the same objects as last time. +- **`mapShared(f)`** (from `@earendil-works/chord/next` state; it moves into Delta when v2 goes upstream): an identity cache per source element: `f` runs only for new or + changed source elements, unchanged ones return their previous result object. With it a list derive costs pointer + comparisons, not content comparisons: + + ```ts + const strip = mapShared((n: Note) => ({ text: n.text, at: n.at })); + self.derive([notes], ([n]) => ({ list: n.list.map(strip) })); + // one appended note: one new object, 2,000 identical references, diffRevisions emits one insert + ``` + +- **Cookbook** (documentation with examples per case): select a field, look up by key, small aggregates (just compute; + the value is small); filter or map a list (`mapShared`); redaction (preferably separate documents per visibility, + otherwise `mapShared`; computed once in the shared store, not per view); joins across two sources (`mapShared` plus + the looked-up source's revision); a window over a large list (take a section of the source elements without + rebuilding them, cache per-element measurements by identity, keep the window's start stable so changes arrive as + appends). Large combined views from several sources re-path the sources' ops (as Durable's `ConversationView`); + application code never interprets ops. + +--- + +## 13. Natives + +```ts +const watcher = self.open(async (context) => watchDir(path, { signal: context.abortSignal }), (w) => w.close()); +self.onLive(async (ctx) => { for await (const f of watcher.value.files(ctx.abortSignal)) await importFile(f, ctx); }); +``` + +- `self.open(open, close, { closeTimeout? })` (slice 2): `open(context)` gets a context aborted when the attempt is + abandoned (a timeout, a cycle, its end); `close(value, reason, context)` gets one aborted at the close timeout, after + which a possible leak is reported (`close_timeout`) and the rest continues. It returns a handle with `.value`, not a + proxy: reading it before its open resolved throws `not_ready`, after its close started `released`. A value that + arrives after its attempt was abandoned is closed at once. + Natives open during activation (the object goes live after all opens) and close in reverse order when it ends. + The opened value never changes during that attempt: no `self.resource`, versions, `replace`, `deps` or stale + handles. To change a live native, the object restarts. A partly failed activation closes already-open natives + with reason `end`; a later attempt opens them all anew. `retryOn` may appear anywhere in the synchronous factory. + Type names: `Native` (the handle), `NativeCloseReason`. +- **Order:** natives open one after another in declaration order (a later opener may use an earlier handle's + `.value`) and close in reverse. A throwing `close` does not stop the others: on `host.close()` all errors are + collected and it rejects with them; an object ending in the background reports them to `onError`. +- Behaviour around a native goes into routines; listeners on natives outside routines are forbidden. +- The facet may not open natives. +- What an old native does after its owner restarts is its owner's `close` (a model runtime keeps working until + collected; a closed SSH connection fails its in-flight `exec`). +- An open that fails makes the object `failed` (`transient` with backoff, or `broken` if it says so). + +- **`close` gets a reason:** `reason: { kind: "reload" } | { kind: "close" } | + { kind: "end" }` (the plugin is reloaded and the object is native / the host shuts down / the last reference ended or + the type was removed). Chord aborts only the object's own calls and jobs; other work using the native (Durable tool + executions on a code mode sandbox) is the owner's: `close` may drain instead of tearing down, e.g. seal the sandbox + (no new scripts) and end it when its running scripts finish or hit their deadline. +- **Restart stays sequential (R16):** close the old native, then open the new one. Two patterns for natives whose + resource has work in flight: + - **Drain in the background:** `close` detaches the old resource, returns at once, and disposes of it when its work + has finished or after `closeTimeout` (30 s at most, Q7: a longer script is aborted); Chord then opens the new one + immediately. Code mode's sandbox does + this; its tool entry looks up the current sandbox at `execute()`. + - **Swap through a registry:** the resource is a registry contribution; readers always find the current one, and a + reader that must wait for one uses a cancellable "wait until present" with the park deadline. + +--- + +## 14. Routines + +Two kinds, declared in the factory (owner, 2026-10-24; C4 decided; slice 5): + +```ts +self.on(catalog.sessions, async ({ value: sessions }, ctx) => { // reacts to a value: first the current one, then every change + for (const f of forwards.value.list) if (sessions[f.session]?.archived) await close(f.id, ctx); // idempotent +}); +self.onLive(async (ctx) => { // called once when the object goes live + for await (const file of watcher.value.files(ctx.abortSignal)) await importFile(file, ctx); +}); +self.onLive(async (ctx) => { // periodic: a loop with sleep + for (;;) { await sleep(HOUR, ctx); await dropOld(ctx); } +}); +self.onLive(async (ctx) => { // set up, wait for the end, clean up + const timer = setInterval(tick, 1000); + try { await untilAborted(ctx); } finally { clearInterval(timer); } +}); +``` + +- **Lifecycle (both kinds):** declared in objects, never in facets (owner, 2026-10-26: everything is done through + objects); they start when the object goes live (in the same turn) + and stop when it ends (their `ctx` is aborted, then + the drain waits for running calls, bounded by `closeTimeout`). Each call gets its own `ctx` (the object's principal + for a remote type, the host's for a local one; the host's incarnation; `ctx.get` released when the call settles). +- **On reload** the old routines' `ctx` is aborted; the new routines start once all old routine calls of the object + have finished (bounded; a stuck call is reported and the new routines start anyway): a handover per object, so + routines need no names. Example it prevents: an old handler midway through `restart(W1)` and a new one seeing W1 + still pending and restarting it again. The bound is `closeTimeout` (owner, 2026-10-24); calls do not wait for the + handover, only the new routines do (slice 7). +- **`on(source, fn)`:** any watchable (a document, a derive, a reference member, a registry value, `ref.state`). The + first call gets the current value (`[["r", value]]`, W2b), then one call per change; calls never overlap (S8); + changes made during a call arrive folded in the next (newest value, folded ops); a running call is not aborted by a + change (native sources too: a busy call is followed by one call with the newest value). A throwing call is + reported and called again with the current value after a backoff (1 s doubling to 60 s; + a change during the backoff calls at once; success resets it). The source ending (a released reference, its object's + end) ends the routine without a report; a retired document does not end it (the call has `retired: true`). +- **`onLive(fn)`:** called once when the object goes live. Returning ends it until the next reload or activation; + throwing is reported and restarts it after the same backoff (reset after a run of at least a minute); an abort is + no error. +- **Idempotent reconcilers:** a routine reconciles from the current value, writes no "in progress" flags and starts a + job for long work. Chord keeps no state for routines across reloads or restarts; what must survive is a document of + the object (e.g. a `lastCleanup` time, which also survives a crash). +- **Helpers:** `sleep(ms, ctx)` (rejects when `ctx` is aborted, leaves no timer or listener) and `untilAborted(ctx)`. + Routines and handlers wait only through calls that take `ctx`; otherwise Chord cannot stop them, and the handover + and the drain report them stuck. +- **Errors:** `onError` gets `routine_failed { object, routine, error }`, `routine` a `Position` (`{ kind: "on" | + "onLive", position }`; natives use the same type); never uncaught. +- **Keep-alive:** routines do not count in `host.activity` (an `onLive` runs forever). +- No bare timers or callbacks that touch state outside routines, jobs or calls. Gone (C4): `every` (an `onLive` with + `sleep`), `custom` and its `run(fn)` (an `onLive` with `try`/`finally`; callbacks that must run one at a time go + through a small queue or the source's async iterator), names, a due time kept across reloads. + +--- + +## 15. Jobs + +Owned work a call starts that outlives the call (a shell command, a session clone, a peer send, a login's work), in +Durable's sandwich: **intent** (a transaction), **run** (any effect), **finish** (a transaction on the current code). +Decided with the owner on 2026-10-24 (decision log "Routines and jobs open points", "Jobs keep accepts", "Job crash +handling"; C3 settled this way); slice 5 builds jobs in memory, slice 6 persistence, slice 7 their reload rules. + +```ts +const ShellState = defineState("chat.shell.state", { + schema: obj({ output: str() }), version: 1, initial: () => ({ output: "" }), +}); +const ShellJob = defineJob("chat.shell", { + version: 1, + args: obj({ command: str(), author: Author }), + result: obj({ exit: int(), output: str() }), + state: ShellState, // optional: one document per run + concurrency: 1, // optional: at most one active run per object +}); + +host.serve(ChatStore, ({ conversation }, self) => { // local: no principals; views filter (below) + const entries = self.state(ChatEntries); + const shell = self.job(ShellJob, { + run: async ({ command }, ctx) => { + const out = new OutputBuffer({ maxLines: 2000, retain: "tail" }); + const progress = throttle(100, () => ctx.state.change((d) => { d.output = out.snapshot().text; })); + try { + const r = await runProcess(command, ctx.abortSignal, (chunk) => { out.push(chunk); progress.schedule(); }); + return { exit: r.code, output: out.snapshot().text }; + } finally { + await progress.flush(); // the last output reaches outcome.state, also after an abort + } + }, + finish: async (outcome, tx) => { + const list = (await tx.draft(entries)).list; + list.push(outcome.kind === "done" + ? shellEntry(outcome.args, outcome.result) + : shellEntry(outcome.args, { interrupted: outcome.error.kind, output: outcome.state.output })); + }, // the run's state document is retired in this commit + }); + return { + shells: shell.running, + runShell: async ({ command, author, requestId }, ctx) => + ({ id: (await shell.start({ command, author }, ctx, { key: requestId })).id }), + abortShell: async ({ id }) => shell.abort(id), + }; +}); +``` + +- **Definition:** `defineJob(name, { version, args, result, state?, concurrency?, accepts?, persist? })`. `args` and + `result` are schemas; `state` is a document definition (a singleton, one document per run); `persist` arrives with + slice 6. `self.job(Job, { run, finish? })` is declared in an object's factory (not in a facet) and returns the job + handle: `start(args, ctx, { key? })` (a `Run { id, outcome(ctx) }` after the commit), `tx.start(job, args, { key? })` + inside `self.tx` (a `Run` handle at once, whose `outcome` rejects if the transaction is discarded), `outcome(id, ctx)` + (unsettled runs only), `abort(id)` (async and stored with persisted jobs, slice 6b) and `running`. +- **1. Intent:** one transaction on the owner's line. Arguments are checked against `args` (closed schema, like + method arguments). Dedupe first: a run with the same `key` that is not settled is returned as it is (the run whose + `finish` this transaction is never counts, so `finish` can start the same job again with the same key). Then the + `concurrency` check: at the limit `start` rejects with `unavailable/busy` (a queue is the owner's, built from a + document and a routine). Otherwise a run record `{ id: uuidv7, version, args, key }` and its state document + (`initial()`) are written in the commit, with whatever else the transaction writes (`tx.start`: a request entry and + the run, both or neither). `run` starts after the commit. +- **2. Run:** `run(args, ctx)` may do anything, outside transactions. `ctx` is a `Ctx` (the object's principal for a + remote type, the host's for a local one) plus `jobId` and `state`: `ctx.state.value` (synchronous) and + `ctx.state.change(fn)` (a transaction on the owner's line on the run's document, without the run's abort signal). + State changes are accepted until `run` settles, also after an abort (so output written while stopping still reaches `outcome.state`), and are + refused (`released`) afterwards, never recreating a retired document. Each change is a commit and, while someone + watches, a message: authors throttle frequent changes themselves (Chord does not coalesce them); awaiting or + catching each change is the author's (an unhandled rejection is the author's bug). External writes carry keys made + from `ctx.jobId`. A `run` that throws after its signal fired ends `aborted`, otherwise `threw` (reported as + `job_failed`). +- **3. Finish:** when `run` settles, Chord records the outcome on the run (`phase: "finishing"`), then calls the + current code's `finish(outcome, tx)` in a transaction (Durable's rule: only `await tx.*`). In the same commit Chord + deletes the run record and its key (Q9: dedupe covers unsettled runs only) and retires the run's state document. A + `finish` that throws changes nothing: it is retried with backoff (1 s doubling to 60 s), each failure reported + (`job_finish_failed`); after five consecutive failures the run is `blocked { finish_failed }`, its outcome, state + and key kept, retried on the next reload. `finish` is optional (then the commit only cleans up). + + ```ts + type Outcome = { readonly id: string; readonly args: Args; readonly state: State } & ( + | { readonly kind: "done"; readonly result: Result } + | { readonly kind: "failed"; readonly error: JobError } + ); + type JobError = + | { readonly kind: "threw"; readonly error: ChordError | PluginError } // run threw (reported) + | { readonly kind: "aborted"; readonly reason: "abort" | "reload" | "restart" }; + // abort(id); a native restart on reload (slice 7); "restart": a persisted run abandoned after a crash (slice 6b), + // or a run cut off by its attempt's seal while the object continues (slice 6a) + ``` +- **Abort is a request:** `abort(id)` fires `run`'s `ctx.abortSignal`; `run` decides what it returns (a result that + already arrived stays a success); a run ignoring the signal keeps its slot. Abort of a finishing run does nothing; + abort of a blocked run drops it (no `finish`, its record, key and state document go, reported; owner, 2026-10-24). +- **Waiting:** `run.outcome(ctx)` resolves with the outcome once `finish` committed; the caller's abort ends only its + wait. Across a link nobody waits on a long call: UIs watch state. +- **`running`:** a JSON watchable, one entry per unsettled run, tagged by phase: + `{ id, version, key?, args, state, phase: "running" | "finishing" } | { id, version, phase: "blocked", reason }`. A + blocked run carries no `args` or `state` (it may be of another version than the code); views filter it as they + filter anything. Per-principal views derive what each principal may see (§6, the shared store): a reader without + output, a guest only their own runs. +- **Concurrency and keep-alive:** `concurrency` counts runs that are running or finishing; blocked runs do not count. + **Runs belong to the object** (owner, 2026-10-26): records, keys, the active count, `running`, run state documents + and pending actions are the object's, across attempts and native restarts; an attempt has the implementation and + its running invocations; a run's state changes and outcome are fenced by the run's own invocation; `finish` runs + on the current live attempt. **Jobs keep their object alive** (D4): running and finishing runs are a count on the object, not references (they + never appear in `host.references`); an object whose last reference goes stays while the count is above zero, and + ends (after its type's `linger`) when it reaches zero, dropping blocked runs with a report. `host.activity.jobs` + counts them. +- **Object end:** revocation (R11), `host.close` and a reload that removes the type end the object at once: runs get + the abort signal, the drain is bounded by `closeTimeout`; an in-memory run's outcome and state go with the object + (no `finish`). Jobs in `local` stores are never revoked (no principal): a store that runs work for a principal keeps + the grant as data and reconciles with a routine (§25). +- **Versions (`accepts`, slice 7):** a job is versioned like everything else; `accepts` maps an older version's args + forwards and its result forwards (`{ 1: { args: (old) => current, result: (old) => current } }`; a producer's + direction, unlike a contract's). A run started on an older version finishes on its old code across a pure reload; + `finish` and `running` see its args and result mapped (mapped once per run, never per publication); its state + document migrates through its own `migrate` (when it may migrate while the old code still writes it is slice 7's + rule). Without an adapter, or for a run of a newer version than the code (a rollback), the run is + `blocked { version }` until a reload brings matching code. A version change aborts nothing and discards nothing. +- **Persistence (slice 6):** `persist: { resume: "rerun" | "abandon" }` keeps the run record and its state document + in the owner's store (owner: a `local` object with empty params, P3). How a run resumes after a crash is required, no + default: rerun (the current + `run`, args mapped forwards, state migrated; only work since the last state change repeats, so it must be + idempotent) or abandon (`finish` gets `failed { aborted, reason: "restart" }`); wording aligned with Durable + (replay-safe, abandon on restart). A recorded outcome resumes only `finish`, never `run`; an abort request is + persisted. A restart limit as Durable's: five crash interruptions, counted and persisted, clean closes not counted; + then `blocked { restart_limit }`, cleared by `abort` or a reload. A persisted state document may use the deferred + persistence option (stored within about a second, flushed before the outcome is recorded). +- **Chaining:** `finish` may `tx.start` the next job, atomically with the outcome (separate steps, branches). +- **Not jobs:** request/response work (a method), standing behaviour (a routine), input into running work (a flow, + §20: the owner's document plus methods), multi-step durable work with sleeps, waits or ownership trees on a host + with a harness (a Durable task). Jobs as objects (#9) are rejected: a third lifetime, two objects per run for two + principals, a `finish` across owners. + +--- + +## 16. The facet + +```ts +import { defineFacet } from "@app/core"; // base code's defineFacet, bound to the app's P + +export default defineFacet((host, self) => { // `self` (contributions) arrives with slice 8 + self.contribute(Http, { routes: { "GET /notes/export": exportNotes } }); + host.serve(Notes, (params, principal, self) => …); + host.serve(NoteStore, (params, self) => …); +}); +``` + +- **Facets are typed by the app's principal (owner, 2026-10-21):** base code exports `defineFacet` already bound to its + `P` (`export const defineFacet = bindFacet

()`, next to the principal union), and every facet (server, worker, + web, TUI) imports it from there; `host.load` accepts only facets typed for the host's `P`, so a factory's `principal` + narrows by variant. Nothing is checked at runtime: the principal a factory receives is the one the link stamped. +- A plugin's per-host setup, **declarations only** (owner, 2026-10-26): `host.serve`, `self.contribute` (slice 8), + `host.references`; no natives, no documents (C16 accepted), no routines, no jobs. Everything a plugin does at runtime + is done through objects: shared or persisted state and standing behaviour live in a `local` store object with empty + params (host lifetime, its own failure domain, R18 recovers its dependents), as p2 does (its 63 facets take only + `host`). A facet cannot fail after `load`. +- `host.serve(Type, factory, { linger? })`: `linger` (milliseconds, 0 by default; slice 3b) keeps a live object with + params alive that long after its last reference, so a new reference within that time reuses it with its documents + and dependencies; it never applies to objects that are not live. +- `host.serve` and contributions are valid only while the setup runs. The facet never `self.get`s a type it serves + itself. +- Between plugins only: object types and registries. No natives across plugins except through registry entries. + +--- + +## 17. Registries + +```ts +export const Http = defineRegistry("chord.http", { + version: 2, + accepts: { 1: (old: HttpContributionV1): HttpContribution => upgrade(old) }, + compose: (table, { report }) => buildRouter(table, report), // pure; always delivers a value; reports conflicts +}); +self.contribute(Http, { routes: { … }, wraps: [ … ] }); +const router = host.registry(Http).value; // live +``` + +- Each contributor (a facet or an object) contributes **at most one value per registry**; the table is an immutable, + ordered list of `(contributor, value)`. Contributor order is sorted by plugin name (never load order): deterministic + on every host and across reloads, and deliberately meaningless; meaningful order comes from the owner's composition + (e.g. HTTP phases). A reload replaces a contribution as a + whole. +- **`compose` is a runtime composition, not a staging check:** contributions can be dynamic (known only once a native + is open, or changing at runtime; contributions may be readables), so no conflict check can be complete in staging. + `compose(table, { report })` is pure and runs whenever the table changes (after a commit, when a contributor goes + live or ends, when a dynamic contribution changes). It always delivers a value with its fallback built in (e.g. the + first by plugin name wins) and reports conflicts through `report` (owner notices and telemetry events); readers get + a plain value and never handle conflicts. Only if `compose` throws is the registry in an error state, and readers + get an error when they use it. Work that started earlier keeps the value it read (a pinned Durable phase keeps its + old table). Helpers: `keyed(table, entriesOf)`, `ordered(table, itemsOf, orderOf?)`. +- **Who contributes (R21):** only facets and host-lifetime objects (`local`, empty params); their factory's `self` has `contribute`, a + referenced object's `self` does not (a type error). Reading a registry's composed value stays open to every object. +- **A contribution is in the table exactly while its contributor is live:** a facet's while its plugin is loaded; an + object's from `live` until it ends, fails or restarts. +- **The owner decides** conflicts, order and selection (which contributions apply where). +- **One authoritative declaration (Q8):** only the facet of the plugin that defines a registry declares it to the host + (`compose`, version, `accepts`); other plugins' tokens only identify it by name and types (contribute and read, + never bring their own `compose`). A reload of the defining plugin replaces the declaration atomically and the table + recomposes. Without a declaration (the defining plugin is not loaded) the registry has no composed value: readers + get an error like a failed reference; contributions wait in the table. +- **Versions (Q8, the owner-adapter pattern):** a contribution carries the registry version its plugin was built + against; older ones are adapted with the defining facet's pure `accepts` when the table is built. A contribution + without a matching version or adapter (older without adapter, or newer) is left out of the table and reported (an + owner notice naming plugin, registry and versions); its plugin keeps working. Contributions are in-process values + (often functions) and are not schema-checked. +- **Use sites guard contributions:** a contribution is untrusted code (third-party plugins are a goal of p3). Every + use site catches synchronous throws and rejections when it calls an entry, reports naming the contributor (the + composed table keeps each entry's plugin name), and continues with the other entries. Durable already does this for + extensions (`extension_error`; a throwing tool becomes an error result); `compose` reports through `report`. +- **Registries may carry natives.** An entry lives exactly as long as its contributor; consumers look up at each use + or follow the composed value and keep nothing from an entry beyond a use. +- **Contributions that change (owner, 2026-10-19; W4):** registries are host-local, so contributions are usually + in-process values (functions, runtimes, handles). A contribution is fixed for its contributor's life (dynamic parts + behind handles read at use; a change means a restart or reload), a `Watchable`, or a `NativeWatchable`; Chord follows + the latter two through the internal flush: on every change it replaces that entry (a new immutable table) and runs + `compose` again, in the owner's flush, never inside another owner's commit step. The composed value is itself a + `NativeWatchable`; use sites watch it or read its current value. For in-process values that change, + `self.writable(initial)` is a `NativeWatchable` with `set` (identity compared, never ops, never a state member): + + ```ts + const ext = self.writable(mcpExtension([])); // e.g. MCP: tools known only at runtime + self.contribute(HarnessExtensions, ext); + self.on(mcp.tools, ({ value: tools }) => ext.set(mcpExtension(tools))); + ``` + + Tools, sections, hooks, wraps and tasks reach the harness as one Durable `Extension` per plugin contributed to + p3's core registry; `compose` builds Durable's registry snapshot (Durable applies selection and wraps, and pins a + snapshot per task phase). Skills and other lists work the same way. +- **Removing a contribution:** there is no remove call (`contribute` is declarative, in the factory). An entry leaves + the table when its contributor ends, or after a reload whose new code no longer contributes. A live contributor that + has nothing to offer for now sets its readable to the registry's empty value (an extension without tools, empty + routes); the table keeps exactly one entry per live contributor. +- Registries add behaviour (tools, routes, renderers, commands, input gates); they never replace an owner's checks. + No lifecycle in entries: anything needing cleanup is a native of the contributing object. + +--- + +## 18. Start data and platform bindings + +There are no host values. Start data (session id, init, data directory) is a `local` object with empty params (host lifetime) +and constant state (`constant(value)`: a readable that never changes, R23), served by the base code's core facet from the start message: + +```ts +export const SessionInfo = local("app.session-info", { version: 1, params: {}, members: { init: InitSchema } }); + +// base code (apps//src): the core facet, passed to load() like any facet +export const coreWorkerFacet = (init: Init) => defineFacet((host) => { // base code's own bound defineFacet + host.serve(SessionInfo, () => ({ init: constant(init) })); // local, empty params: host lifetime +}); + +// any plugin +host.serve(NoteInbox, (_, self) => { + const info = self.get(SessionInfo, {}); + const watcher = self.open(async (context) => + watchDir(join(info.init.value.dataDir, info.init.value.id, "inbox"), { signal: context.abortSignal }), (w) => w.close()); + return {}; +}); // NoteInbox: local, empty params → host lifetime +``` + +Plugins read start data through a reference, after activation (in openers, derives, routines and methods); nothing +reads it synchronously in a factory. Platform bindings the host owns (a Durable Object's `state` and `env`) are core +objects whose `self.open` returns the binding and whose `close` does nothing. A long-lived native from the host is a +host-lifetime object in a core facet (the Durable harness, section 24). + +**Core vs. plugin is decided by where the code lives:** base code (`src/`, imported statically by the start code under +a fixed name such as `@app/core`, not switchable, a change restarts the host) or a plugin directory (found by listing, +loaded through loader D, named by its `package.json`, switchable, a change reloads it). Core is what the app always +needs and everything builds on (harness, catalog, access, session location); optional features are plugins (code +mode is a plugin). Both use exactly the same primitives. + +--- + +## 19. Presence + +```ts +const viewers = self.derive([host.references(Notes)], ([refs]) => refs.length); +``` + +`host.references(Type)` (slice 4b) is a JSON `Watchable` of the references to objects of a type on this host: +`{ params, principal?, incarnation }[]` (named `origin` until 2026-10-27). It is one rule (owner, 2026-10-24): the incarnation of the host that made +the reference as its caller sees it, so two tabs are two origins, and a guest's `ctx.get` shows the guest's tab. An +entry is added when a reference binds and removed when it ends, in that batch; a departure discovered while a batch +is sending lands in the next batch (so one commit stays one message per link). The host keeps a type's list from its +first `host.references(Type)` on (one small object per asked type; entries come and go with references). **A link drop is a departure** +(accurate and simple); smoothing (a phone switching networks) is the UI's job, e.g. a debounced viewer count. Counts +per person or origin are derived by the app from the principals and origins. There is no consumer-side linger +(a closed tab closes its link); provider-side `linger` keeps a just-closed object warm and keeps the worker alive +(section 3). Replaces the presence, viewer and login-abandon leases. "Leaving" is never derived from an unmount +(section 21). + +--- + +## 20. Input into running work + +### Flows kept by their owner + +```ts +start({ provider }, ctx) → { id } // uuidv7 +LoginFlow {id}: state | { kind: "running"; step?: { id, … }; notices } | { kind: "done" | "failed" | "cancelled"; notices } +answer({ id, step, value }, ctx) → ok | "step no longer active" +cancel({ id }, ctx) +``` + +The UI watches the state; a late or second answer to a step that is gone is refused; a promise-based callback (pi-ai's +`prompt`) stays inside the owner (its job keeps the resolver). Imperative consumers (tests, scripts, a CLI, an agent +tool) use a generic client helper over the same state, not a second protocol: + +```ts +async function* steps(flow, ctx) { + let last = 0; + for (;;) { + const s = await until(flow.state, (v) => v.kind !== "running" || (v.step && v.step.id > last), ctx); + if (s.kind !== "running") return s.kind; + last = s.step.id; + yield s.step; + } +} +for await (const step of steps(await ctx.get(LoginFlow, { id }), ctx)) + await login.answer({ id, step: step.id, value: answerFor(step) }, ctx); +``` + +A long-held server `step()` call is not used: a link drop fails it and the consumer cannot tell whether it missed a +step; state re-holds by itself and `last` keeps steps from being lost or repeated. + +### The Questions service + +Another pattern (`apps/p2/docs/questions.md`): a shared store with per-principal views (asker: `ask`; viewer: `open`, +`answer`), typed questions (`text`, `select`, `confirm`), policies over a live audience (`first`, `quorum`, `all`), +reattachment by `key`, deadlines, authorization per question, secrets redacted. Fits tool approval and a question the +agent asks everyone present. Its store must live in an object (it predates "natives only in objects"). + +> **Decision needed: where a flow lives.** Options: (a) a keyed document of its owner; (b) an object `LoginFlow {id}` +> living until done. (Jobs as objects were rejected, §15; a flow kept as its owner's document plus a job is the likely +> answer.) + +--- + +## 21. Reload and rollout + +### Per host + +``` +1. load new bundles; an import or syntax error rejects +2. staging facet setups and every live object's new factory run with a recording `self` (declarations only, + nothing activated); any throw rejects +3. commit the host switches to the new code atomically +4. after per object (below) +``` + +- **Rollback is all-or-nothing per host:** any load or staging error (code errors: load errors, throwing factories) + leaves the whole host on its previous code; a host is completely at revision R or R−1, so status per host suffices. + Runtime states (registry conflicts, failed opens, migrations) are not rollback reasons. +- **Pure objects** (no `self.open`, old or new) are swapped in place: new methods and derives; documents, references + and jobs stay; routines restart (serialized); new references become live; then the gate opens. Running calls finish + on old code, and keep their old handles valid until they settle (their transactions keep working), except for + documents whose version rose (R15, as in Durable): old code touching a document migrated past its version throws, + the document stays intact, and the call fails once (not resent). Only new calls get the new handles. +- **Native objects of the reloaded plugins restart:** new calls park, running calls are drained first (bounded by + `closeTimeout`; owner, 2026-10-26: a call outliving the drain is cut off by the seal fence), then natives close (reason `reload`), + the factory runs, natives reopen; documents stay. In practice this costs little: the Durable harness is core (base + code), terminal PTYs live in the separate terminal host process (the server's native is only the connection), model + runtimes, sandboxes and the search index rebuild in milliseconds; only reloading the plugin that owns SSH connections + (running remote commands are killed) or the relay socket (relayed connections drop) hurts. Natives are not kept + across reloads by name: natives have no line, no transactions and no schema, so old and new code would interleave + unchecked. +- **Derives on reload:** derives have no names; exposed derives are matched by their member name, internal ones are + rebuilt (matching only if a real need appears). +- **Holds across reloads:** a hold's indexes belong to its bound version. A reload that still accepts that version + keeps the hold: a pure object re-subscribes it through the version's mappings and sends `[["r", value]]` for each + member whose value changed; a restarted object sends a fresh `bound` for the same hold (a `bound` on a bound hold + replaces its replicas). A served hold whose object leaves `live` while its `bound` waits for a slot goes back to + pending (slice 7 builds it: nothing in 4a leaves `live`). A reload that drops the version ends its holds with `gone` (`contract_mismatch`) before + the new code serves anything. A consumer whose own code moves to a new version re-holds (a new hold id) and releases + the old. +- **Old-version calls:** `checkArgs`'s `{ kind: "call" }` result runs the provider's replacement; `{ kind: "method"; + result }` maps the result back for an old-version caller (section 5). +- **Retries:** a reload retries a broken object as `ref.retry()` does (section 6). +- **After the commit,** errors stay per object: a failed open or reference makes that object `failed` (backoff); a + migration that throws on a lazily loaded document refuses that access; a method bug fails that call. +- **Removing a type:** its objects end (routines stop, natives close, references end with an error); in-memory + documents go with them; persisted ones stay until retired. +- **Consumers elsewhere** see a restarting object like any object that briefly goes away. +- No factory trial, no migrate dry run, no consumer contract check during staging: incompatible consumers fail at bind + with `contract_mismatch` and a notice naming plugin and type. +- **Start barrier** (host start and first load): load facets (the core facet with its start data among them), wait + until every facet and host-lifetime object is live or failed (an object that must + contribute at start, like the model runtime, is a `local` object with empty params). Then the registries hold the contributions of every + plugin that loaded, and dependent work + resumes: base code calls `harness.resume()` after `load` (the harness is an ordinary host-lifetime object that + opened Durable paused during the barrier; owner, 2026-10-21, refines R17). Nothing + waits for specific contributions: what is missing shows where it is used (section 24). Failed plugins are reported + with their error. Plugin reloads need no barrier. + +### Rollout across hosts + +- A **revision** is (server incarnation, sequence) and names the complete desired code set (every facet with its + digest), not a batch. Every install reconciles everything missing; a host reports a revision only when it runs that + complete set. +- **Order:** server, workers, clients. Each host reports `installed`, `installed_failed` (named failed objects) or + `rejected` (the reason; the host stays on its previous code). A server that rejects stops the rollout. +- A host that starts or reconnects installs the current revision before anything is routed to it. "Installed" means + the code is published; objects activate on demand. +- **Mismatches:** a mismatch response carries the provider's revision and status. Provider ahead, or behind and + installing → `updating` (values readable, calls park, with a deadline); provider rejected → fail at once (retry on + the next revision); same revision → `contract_mismatch`, final. Calls parked on a binding that is released end with + an error and are not resent. No cross-host transaction, no votes. + +- **A host checks its own install status before waiting on a provider:** a rejected own install fails at once with + the reason. **Installs are serialized per host.** **A revision also carries the code-set digest** (the hash over the + facet bundles' content hashes, which loader D computes anyway; no module analysis); hosts compare the digest on + reconnect, so a server restart with unchanged code triggers no reinstall. + +### Loading and the dev loop + +- Plugins come from built-in, global (`/plugins`), package (npm/git, installed with `--ignore-scripts`) and + project roots, found by directory listing (no source parsing); the server selects; dependencies resolve by contract + name at runtime; names are prefixed (`.`; tools `_`). +- **Loader D:** one base bundle per surface (host packages and the app runtime) plus one script per facet (host + packages passed in, no module map); old generations are reclaimed by the GC; stable filename per (facet, digest) and + V8's compilation cache off on hosts that reload in place. The loader is an adapter: static imports (tests, Durable + Objects) work the same. +- Project plugins, trust, snapshots and scopes: `apps/p2/docs/plugin-loading.md` (still the reference for those). +- **Dev loop:** a source watcher marks plugins dirty (never applies changes); a reload control (playground web shell, + CLI, agent tool) reloads the changed plugins or restarts all hosts for base code. + +- **UI reload vs. leaving:** "leaving" is never derived from an unmount, only from explicit navigation or presence (a + dropped reference); a component unmounted by a reload has not left. + +> **Decision needed: project plugins.** Per-directory plugins with trust and snapshots (`plugin-loading.md`) need their +> own pass over this design. + +--- + +## 22. Wire + +Exposed documents, derives and readables are state members: a snapshot in `bound` when a hold binds, ops per commit +(a draft's recorded ops, no diff), ops from diffs for derives, ops of job state documents (slice 5), and failed +members in the entry of the message that carries them. **Commits are atomic for consumers:** locally, subscribers are +notified once after every document of the commit is adopted; remotely, one `state` message per commit per link and +origin carries an entry per touched hold, applied in one batch before listeners run (also when one Durable commit +touches several holds), so remote consumers never see a torn commit. Consumers apply ops with `applyImmutable`: no +validation, no clone of snapshots or ops. + +- **The protocol version** travels in `hello`; hosts of different versions refuse each other (a web page reloads for + a newer server). Error reasons and message kinds are part of it. Until p3 ships, changes keep `PROTOCOL_VERSION` 1 + (nothing is deployed); after that every message or reason change bumps it. +- **One typed `Message`** (owner, 2026-10-22, "Wire protocol"): a TypeBox schema `MESSAGE`, a tagged union over + `kind`, from which the protocol types derive; every decoded message is checked against it (compiled once) unless + the link is trusted (section 3). Kinds: + + | Kind | Direction | Fields | + | --- | --- | --- | + | `hello` | both, first | `protocol`, `incarnation`, `calls` (the call window N) | + | `hold` | to provider | `origin?`, `hold`, `type`, `version`, `fingerprint`, `params` (part), `address?`, `principal?`, `incarnation?` | + | `release`, `retry` | to provider | `origin?`, `hold` | + | `call` | to provider | `origin?`, `call`, `hold`, `method` (index), `args?` (part; absent: no arguments), `segments?` | + | `abort` | to provider | `origin?`, `call` (named `cancel` until 2026-10-27) | + | `ack` | to provider | `origin?`; one per applied `bound`/`state` message | + | `closed` | relay to provider | `origin`: the downstream origin ended | + | `bound` | to consumer | `origin?`, `holds: [{ hold, variant, members: [{ member, value }], failed }]`; one batch | + | `state` | to consumer | `origin?`, `holds: [{ hold, members: [{ member, ops }], failed, retired }]`; one batch (`retired`: members retired since the last message, slice 6a) | + | `failed` | to consumer | `origin?`, `hold`, `cause { kind, error }`, `retryIn?` (absent: broken) | + | `loading` | to consumer | `origin?`, `hold`: a new attempt after a failure, or a relay's upstream dropped | + | `result` | to consumer | `origin?`, `call`, `value?` (part; absent: `null`), `segments?` | + | `error` | to consumer | `origin?`, `call`, `error` (a ChordError's JSON or a plugin's code, message, details) | + | `gone` | to consumer | `origin?`, `hold`, `error`: the held object ended (revocation 4b, reload 7; relays never make one) | + | `ping`, `pong` | both | the heartbeat | + + A `bound` on a hold that is bound already replaces its replicas in one batch, another variant allowed (re-holds, + slice 7's restarted objects). +- **Codecs are pluggable per link** (both ends use the same): `binaryCodec` (default: a kind byte with a flag for a + following origin varint, varints, raw incarnations and fingerprints, length-prefixed strings and parts; about 23 B + for a streamed token) and `textCodec` (Delta-style JSON tuples without field names, a message with an origin wrapped + as `["@", origin, tuple]`; for debugging; `describeMessage` prints any message so). Measured: a 2,000-token reply + about 46 KiB instead of 196 KiB. No CBOR (revisit only if benchmarks show encoding as a bottleneck). +- **Parts are opaque** (slice 4a): `params`, `args`, `value`, snapshot values and ops are a `Part` (JSON text at the + origin, its UTF-8 bytes once a binary codec received it). A part is parsed only where it is owned, right when its + message is handled (the provider parses params and arguments, the consumer results, snapshots and ops), so a relay + forwards it unparsed and a malformed part is still a protocol error. A message naming an unknown id is ignored + before its parts are parsed. The text codec re-serializes parts at decode (one parse per part per hop, accepted for + a debugging codec). +- **Names travel as indexes** into the hold's bound version (each variant's state members and methods sorted; variants + sorted). A reload that drops that version ends its holds first (slice 7). +- **Checks by layer:** the codec's structural decode, the `MESSAGE` envelope (skipped on trusted links), the bound + version (ids, indexes, order), then the owners (`checkParams`, `checkArgs`, Delta's decoder). +- **Unknown ids are ignored on both sides (Q1):** they are normal races (a `result` crossing an `abort`, a `call` + crossing `gone`, a state part crossing a `release`); the side that ended a hold or call settles its own pending work + (a consumer receiving `gone(h)` rejects its pending calls on `h`; per-direction order guarantees nothing else for `h` + follows). Protocol errors close the link and are reported (`protocol_error`): a message failing the schema or the + codec, a malformed part, a reused live id, more than N running calls of one origin, a broken handshake (a first + message that is no `hello` of the link's codec, a second `hello`, another protocol version). +- **Byte segments:** a payload (arguments, a result) whose method's schema has a `bytes()` leaf carries each + `Uint8Array` as a raw segment referenced by `{"$bytes": i}` and read back as a view of the received message; `$bytes` + is the one reserved key there (`invalid/args`). Never base64. State is JSON only. +- **Encoders:** each member of a hold has its own Delta encoder and decoder (path interning end to end between + provider and consumer), restarted with every snapshot. Two holds of the same principal (two tabs) encode the same + ops separately (O(change) each). A part is encoded once at its origin; relays never re-encode it. +- **Sending:** one `state` message per commit per link and origin while the window has a slot; otherwise the ops wait + in the members' pending queues and leave as one message when an `ack` frees a slot (section 3). +- **Relays** read and rewrite only envelopes (section 3, "Relays"). +- **Size limits per part (R2, Q2; slice 4b):** limits apply to each part (wire and decoded size), configurable per + link, checked from the part's length before it is parsed or inflated; the decoded limit is enforced in the one + decode function. An oversized part fails only its own hold or call (`too_large/part`); the other parts of the message + apply; a snapshot over the limit fails its hold with no automatic re-hold. The decoded limit travels with relayed + work: the relay checks the admitting link's wire limit itself and stamps its decoded limit on the upstream `hold` + (honoured on the parent link only, never taken from a client; calls inherit their hold's), and the host that + inflates enforces the smaller of its own and the stamped one while inflating, after the part's owner and id are + known (parts for unknown ids are never inflated, Q1). A relay checks only what arrives from downstream; answers + from upstream (the app's own workers) it forwards unchecked and unparsed, and the end that parses them checks them + against its own link's limits (owner, 2026-10-24: each end checks what it parses; a relay check needed a re-hold + with the same id, which old messages in flight could mistake for the new hold). So IPC carries no limit of its own. + A reference failed `too_large/part` is never held again automatically: it leaves its path, so a reconnect does not + hold it; only `ref.retry()` puts it back (a new hold). This lives only in the consumer's memory, as long as the + reference. The + WebSocket's `maxPayload` stays only the transport's hard cap (e.g. 64 MiB; code 1009 closes the connection). + Defaults for client WebSockets: 16 MiB on the wire and 64 MiB decoded per part; none on IPC. +- **Compression at the origin** (owner, 2026-10-19; slice 4b): a JSON part above a threshold (e.g. 16 KiB) is + compressed once by a host with a synchronous compressor when it encodes it (`createHost({ compression })`, Node's + zlib adapter in a Node-only subpath), whatever the link (IPC included), only if it shrinks; hold params are never + compressed; byte segments stay raw; the codec is in the part's envelope (the text codec tags every part, so no JSON + value is reserved); relays forward it compressed; the end that owns it inflates it. Option (b) (owner, 2026-10-24): + hosts without a compressor (browsers) send uncompressed and inflate asynchronously (`DecompressionStream`), holding + later messages of that link in order (bounded by the window and N; acks go out after apply). **Child links refuse + compressed parts** (protocol error): honest clients never send them, and a 65 KB part could inflate to 64 MiB on the + server; so asynchronous inflate happens only on a host receiving from its parent or a route. Because a host cannot + tell whether its parent sees it as a child (a tab) or over a route (a worker), call arguments sent up a parent link + are never compressed; results and state going up are. Compressed hold params are a protocol error on every link. + The asynchronous inflate runs one part at a time; the queue behind it is bounded by the sending peer's window and N, + so only a misbehaving parent or route peer could grow it (accepted: those are the app's own hosts). The adapter reports "too large" + and "corrupt" as a tagged result. Limits and the threshold count bytes. No per-link compression. +- **Trace carriers (R5; slice 9)** are an optional bounded field of `call` and `hold` (key count, string lengths); + relays read them from the envelope. +- **Adapted bindings get diffs** (slice 7): a consumer bound through `accepts` is served like a derive of the + provider's state through the adapter: Chord keeps the last adapted value per binding, applies the adapter to the new + value and sends the diff (`diffRevisions`), so an old tab during a rollout never receives a large state whole on + every commit. + +--- + +## 23. Optional components + +Deferred: the HTTP and channel components and middleware phases are built after the kernel fixes and a first real +proof (harness, one tool with an approval pause, a code mode reload); the playground's web shell serves its pages +with plain app code until then. The design below stays the target. + +### HTTP (`@earendil-works/chord/next/http`) + +A route is `(req, ctx) => Promise`. `req` is a plain request summary (method, path params, query, +principal, provenance such as local or relayed, selected headers; parsed and capped JSON bodies; uploads handed over in +chunks); `ctx.principal` is the request's principal (from the request resolvers), so `ctx.get(Notes, …)` gives that +principal's view. A route returns a description: + +```ts +| { kind: "json"; value } | { kind: "page"; title; body } +| { kind: "bytes"; name; mime; size; read(offset, length) } +| { kind: "redirect"; to } | { kind: "error"; status } +``` + +The component does ranges (206/416), HEAD, download headers (safe filename, `nosniff`, CSP, caching), streaming, +abort on disconnect and the CSP nonce once, instead of every plugin. Contributions hold `routes` (composed with +`keyed`; a duplicate pattern is a reported conflict, the first by plugin name wins) and `wraps`: + +```ts +type Wrap = { + name: string; + phase: "observe" | "headers" | "transform"; // owner-defined phases, outermost first + scope?: "own" | "all"; // default "own": only this contributor's routes + route?: string; // only this route; a missing route is a reported conflict + wrap: (req, next, ctx) => Promise; +}; +``` + +Within a phase, wraps run in contributor order (no numeric priorities: plugins have no single registration order, as +Express or Hono apps do; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary +on top of an owner's checks, never instead of them. A request uses the router as it was when it started. Request +resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary. + +### Channels + +WebSocket byte channels (`send`, `onData`, `close`) for terminals, separate from object links, with their own flow +control (acknowledged windows). + +### Chunked transfers + +Large byte transfers are chunked by the owning object: `begin`, `append({ upload, offset, bytes })`, `finish`, +`read({ id, offset, length })`, about 1 MiB per chunk. The awaited `append` is the backpressure; after a drop the +writer asks for the stored offset and resumes; unfinished uploads expire (the owner's routine). A helper wraps the +loop. Chord has no stream type (it would need its own rules for drops, cancellation, relays and restarts). + +--- + +## 24. Durable integration and portability + +### Durable + +- Durable keeps importing today's Chord's `context`, `delta`, `json`, `output` and `sqlite`; `next/` imports the same + modules, so `Context`, `Draft` and `JsonValue` are one type. Durable's conversation view, task graph and documents + are `Watchable`s through Durable's watches (Durable's `WatchHandle` now lives in Chord); Durable adopts and dispatches + them synchronously inside the storage's commit (S1, W1; with `StoreStorage` that is the store's commit step), so a derive over a Durable view travels with it. This needs Durable to publish a commit when it adopts it (owner, 2026-10-26; a pi change, upstream first; explainer `durable-observers/`): inside `adopt`, its trackers, telemetry events and internal observers (view mounts, task graph, scheduler, submissions) run, and the host `subscribeCommits` listeners are dispatched in one microtask queued before the observers, so they run before every wait the commit resolves and every abort it queues, but after the storage's synchronous step, never inside Chord's batch; host listeners and watch listeners have no order between them (owner, 2026-10-26). Durable's public primitive for adapters is a synchronous observer, `observeConversation`, `observeTaskGraph`, `observeDoc` (its `advance({ value, ops, retired }, ctx)` runs inside `adopt`: record only, no awaits, no Session calls, no throws); `watchXXX` is built on it for app code. Chord's `durableView` (slice 6b) is a source over it. Otherwise a derive over two Durable views computes between their frames of one commit. With `StoreStorage`, Durable's `adopt` runs inside the host's batch; the store knows nothing of Durable (no special cases) and needs only its step and flush to stay synchronous. Durable's + state getters (`documentState`, `viewState`, `taskGraph()`) and the `replicatedState` family are removed (W14). Only + plain data crosses between Durable and v2 objects. +- **The harness is an object in the core facet** (base code): a `Harness` `local` object with empty params, so host lifetime (it runs for the worker's whole life, + whether or not anyone holds it) opens Durable paused with `self.open` during the barrier and gets model + access and execution environments from registry entries, looked up at each use: + + ```ts + const harness = self.open(async () => Durable.open(storage, { + models: () => host.registry(HarnessModels).value, + env: (target) => host.registry(HarnessEnvironments).value(target.cwd), + settings: () => currentSettings(), + registry: durableRegistry(host.registry(HarnessExtensions)), // a RegistryReader over the composed NativeWatchable + }), (h) => h.close()); + ``` + + Durable's `models` and `settings` accept functions (pi `7ec1f648e`). Durable opens with scheduling paused; base code + calls `harness.resume()` after `await host.load(...)`; the worker connects to its parent first (W6), so host-lifetime + stores can reference server types during the barrier; requests arriving during `load` park like any call to a + loading object (refines R17). A missing tool gives its call an error result the model sees; a missing model makes the generation + report "model unavailable" and wait (not fail) until a model contribution appears; a task whose definition is missing + stays blocked (`missing_task`) until the plugin is installed or the task is aborted, as Durable does. +- `HarnessModels` and `HarnessEnvironments` start with one contribution each, from the models and envs plugins' + objects that own the natives; several providers or environment kinds are their `compose`. After a models reload + the next request uses the new runtime; a running stream finishes on the old one. +- **Chord from Durable code (Q6):** tools, hooks and sections get Durable's `ToolExecutionApi` and a `Context`, never a + Chord `ctx`. p3's core opens the harness with a base context holding the host under a typed key + (`withContextValue(ChordHostKey, host, …)`); every phase context inherits it. A core helper opens a call scope from + any Durable context: `chord(async (ctx) => …, context)` = `host.root(fn, { signal: context.abortSignal, telemetry: + context.telemetry })` (host principal; attribution as data; references released when `fn` settles). No wrapping of + tools and no p3 `defineTool`. Clients are never reachable this way (they are not addressable): a tool that needs the + user goes through an owner's state, e.g. the Questions service (`apps/p2/docs/questions.md`): the model-facing + `ask_user` tool and any other tool call `Questions.ask`/`wait`, tabs render the open questions and `answer` them. + Documented with this example. +- **Code mode** (a plugin) keeps owning its `pi-codemode` sandbox; its tool entry looks up the current sandbox when + `execute()` starts, so a call pinned by a phase before a reload (e.g. waiting in an approval hook) starts on the new + sandbox; on `reload` and `close` its `close` detaches the old sandbox, returns at once and disposes of it when its + scripts finish or after `closeTimeout` (30 s at most, R16, Q7); after a worker stop Durable replays as today. +- **Telemetry:** `HarnessOptions.telemetry`; submissions and tasks store an optional carrier (section 29). + +### Portability + +Adopted in the kernel: host = identity plus incarnation; the loader is an adapter; values independent of the +transport; durable state only in persisted documents. A Durable Object deployment +(`apps/p2/docs/research/chord-on-durable-objects.md`) would map a logical host to a DO, compose plugins at build time +(Workers forbid `eval`), replace the relaying tree with edge routing and use alarms for routines. + +> **Decision needed: a Durable Object profile.** Problem: a DO can lose its memory while clients stay connected +> (hibernation); routines need wake-ups (one alarm per DO); links between DOs are RPC stubs. Hot reload is not needed +> there; plugin composition is. For later. + +--- + +## 25. Apps: ownership and porting + +- **Every thing has one owner, the smallest object it belongs to;** ownership nests; rules set at creation are written + in the creating commit (Durable `createConversation` with `init`). +- **Core owners are always present;** a plugin owner may be missing (optional dependencies are registries owned by the + dependent, section 7). Avoid a core god object: each core owner owns only what belongs to it. +- **Revocation of work in shared stores:** a store that runs work on behalf of a principal stores the grant (or device) + as data with the work and reconciles from the grant owner's live grants (a routine aborts jobs, withdraws questions, + drops queued input whose grant no longer exists); no revocation hook. The window between a revocation and the + routine's run is accepted (R10): a step started in it still runs; there is always some window. +- **Reading without starting a worker** is plain pub/sub in an ordinary plugin: a host-lifetime object of its worker facet + records the summary locally and publishes it to the plugin's server object (a method call from a routine on the source); clients read + the server object, which never starts a worker; it keeps the latest value per session (a document if it must + survive a server restart) and ignores reports from an older worker incarnation. **The caller's incarnation is + stamped by Chord (R20):** a call arriving over a link carries the calling host's incarnation from the link handshake + (`ctx.incarnation`, next to `ctx.principal`, not part of the principal or its key, so object identities do not + change with restarts); nobody sends it as data and it cannot be forged. On relayed calls it is the **origin** host's + incarnation (Q13, owner, 2026-10-21): the relay takes it from the link the hold came in on and sends it in + the upstream `hold` (a client's value is ignored, like its principal); calls inherit their hold's (section 4). Session list, status, children, unread and + artifact lists follow this. +- **Porting guideline:** what core enforces on data lives in a format core owns, written by the plugin that creates + the data; a plugin's private document is read only by that plugin. Example: subagents mark one-shot and + cwd-following child conversations in a core per-conversation policy document (`readOnly`, `followsSessionCwd`); + chat and relocation read it; `ReadOnlyConversations` and `SessionCwdFollowers` go away. +- **p3 is built next to p2,** fully isolated (`apps/p3`, own data directory), without migrating p2 data; plugins are + derived from the functionality map (`apps/p2/docs/p3-functions/`), not ported one to one. +- **Session lifecycle:** archive retires nothing (stops the worker, ends guest access, closes forwards); a session + delete removes the whole session (tagged documents, the worker's databases, files, the catalog entry). + +--- + +## 26. Decisions needed + +Open, decided with the slice named or a concrete case (numbers kept stable; everything else is decided and folded into +this document): + +| # | Topic | Section | When | +| --- | --- | --- | --- | +| 11 | where a flow lives | 20 | when login is ported | +| 14 | project plugins | 21 | own pass over `plugin-loading.md` | +| 16 | a Durable Object profile | 24 | when a DO target becomes concrete | +| 24 | the rollout install timeout (60 s) and a per-type drain override: host options or fixed (the routine handover uses `closeTimeout`; a failing job `finish` backs off 1-60 s and blocks after five failures, decided) | 3 | with slice 7 | +| 25 | `host.root` signature (`root(fn, context, { principal? })` proposed) and what an aborted, uncooperative `fn` means for its references (match the handler and `CallContext` rules) | 3, 24 | slice 8 | +| 26 | how a migration commit reaches consumers (`[["r", value]]` per changed member, or a fresh `bound`; adapted by `accepts` for older consumers) | 21, 22 | slice 7 | +| 28 | Astra's circuit breaker (a hard transport-buffer cap plus a stall deadline); the heartbeat is built, the cap was never decided | 3 | if a real case needs it | +| 29 | round-3 cut C11 (`ctx.trace`); C3 and C4 are decided (§14, §15) | 8 | slice 9 | + +Apps (p3), from the functionality map, decided per domain when p3 is designed: + +| # | Topic | Problem and options | +| --- | --- | --- | +| 17 | session files | one owner `SessionFiles {session}` on the server for uploads, sent-file snapshots and artifact versions (caps per kind, index readable without a worker, copy on clone/fork, delete with the session); proposed, sub-points: one preview limit, fork's artifact list, copying attachments into remote environments | +| 18 | session delete | per-session objects plus `SessionTag` plus a session directory, or owner hooks through a registry; search index per session or global plus a hook | +| 20 | connection pools | SSH connections to one environment exist four times: one owner with a pool, or per-use connections | +| 21 | config files with a single writer | settings, models and auth are watched by every process and written by two: one owner on the server, others follow its state | +| 22 | file formats as versioned interfaces | `envs.json`, downloads, uploads have no version: version them like documents, or keep each internal to one owner | +| 23 | the terminal host process | a detached process with its own code: part of the revision (reloaded or restarted with it) or deliberately outside | + +--- + +## 27. Versions and adapters + +| Surface | Version | On change | +| --- | --- | --- | +| object type | `version` plus contract fingerprint | every change bumps; forward evolution and `checkAccepts` (Q10); otherwise `contract_mismatch` | +| registry | `version` | `accepts` adapter on contributions | +| document | `version` plus schema fingerprint | pure `migrate`, on access; a schema change without a bump fails the object `broken` | +| job | `version` | `accepts` maps older args and results forwards (`finish`, `running`); the state document's own `migrate`; `blocked` without an adapter or for a newer version | +| native | none | the object restarts on reload | +| Durable tasks and documents | `version` | Durable's `migrate` | +| link protocol | protocol version | refused; hosts update together | +| UI components, Chord and app APIs | none yet | exposed as they are; versioned with an SDK later | + +**Contract evolution in code (Q10; implemented in `packages/chord/src/next/contracts/`, see its README):** + +```ts +export const NotesV1 = remote("notes", { version: 1, params, variants: { + editor: { state: { notes: arr(NoteV1) }, methods: { add: method(NoteV1, Id) } }, +} }); +export const NotesV2 = evolve(NotesV1, (e) => e.state("editor", "notes", arr(NoteV2)).args("add", NoteV2)); + +const accepted = checkAccepts(NotesV2, { + 1: { state: { editor: { notes: (current) => current.map(toV1) } }, methods: { add: { args: (old) => toV2(old) } } }, +}); +``` + +- A history starts with a base written in full with an explicit version (`remote()` / `local()`; `local` has flat + `members` and no variants). `evolve(prev, (e) => …)` always continues it with structured edits: `state`, + `addState`, `removeState`, `args`, `result`, `method(name, args, result)`, `addMethod`, `removeMethod` (local: no + variant argument). Every version is materialized and fingerprinted at declaration, so an old plugin's own copy of + v1 computes the same fingerprint; no build step, no generated files, no hand-copied fingerprints. +- **A new history base** (a params change or a change of the variant set) is a new `remote()`/`local()` with an + explicit version and no accepts for older versions: old plugins get `contract_mismatch` and must update. +- **Mappings live with the provider,** never in the contract: the pure `checkAccepts(contract, accepts)` (called by + the objects layer) requires, at type level and again at runtime, a mapping for every member that exists in the old + version and changed or was removed since: changed state `(currentMember) => oldMember`, removed state `(current + variant state) => oldMember` or `"refuse"`, `args: (old) => current`, `result: (current) => old`, removed methods + `{ call }` or `"refuse"`; added members need nothing. No-op edits need identity mappings. Untyped JavaScript + providers fail at declaration naming the member; unknown keys and wrong shapes are rejected. +- **The fingerprint** is a hash (`hash64`, portable, not a security boundary) of a schema-aware, data-only normal form + (`NORMAL_FORM_VERSION`, tied to the protocol): annotations dropped only on schema nodes, `required`/`enum` sorted, + union and intersection order kept (it is meaning: defaults), a whitelist of TypeBox keywords, executable features + rejected; bytes are a `{ type: "bytes" }` data marker. Schemas are deep-copied into a private history (never frozen; immutable by convention). +- **Schema helpers:** `str num int bool nul lit oneOf union obj arr map opt bytes unknown` and `type Static`. +- Chord wraps state mappings as a derive and diffs per binding (section 22); members an old consumer does not know do + not appear. +--- + +## 28. Building: slices + +Chord v2 is built in vertical slices; the plan, the feature map (every feature of this document assigned to the slice +that builds it) and the carry-over of deferred obligations are in `slices.md`. Slice specs (`slice-N.md`) stay as the +history of how each slice was specified; this document states the current rules. The first layered build (kernel steps +1-6) was deleted on 2026-10-22 and survives only in git history. Done in pi: `pi-telemetry` (propagation, links, +metrics, the store interface, the renderer), `chord/context` (`Context.telemetry`), pi-ai and Durable telemetry, +`chord/telemetry-sqlite` (section 29), Chord's shared `Watch`/`NativeWatch`/`PendingOps`, Delta's `appendText`. + +**Rules for every slice** (owner, 2026-10-22): the code follows the root `AGENTS.md` section "Chord v2 Code Style"; +one test file per source file, one `describe` per mechanism, the spec's test list case by case; cost guards through the +in-memory telemetry adapter (section 30); GC tests for every cache and binding; one or more runnable examples +(`packages/chord/test/next/examples/NN-topic.ts`); hot-path slices add a light benchmark script +(`test/next/bench/`, by hand) with results in `benchmarks.md`; `npm run check` and the slice's tests before reporting. +Caches are `WeakMap`s keyed by their owner; no module-level strong caches; no hot-path work no decision asked for. The +function doing counted work counts it through its context; call sites never count for others; plain totals without +site labels. + +**Testing the core:** every slice spec contains an explicit test list, one group per mechanism with its edge cases, +tested individually and exhaustively: activation and dependency recovery (R18: broken dependency, trigger, backoff as +safety net, a dependency failing while live, cycles), revocation over running jobs (R11), compatibility (R6), params +and arguments for every caller (R7), migration results and old calls on migrated documents (R8, R15), flow control +(the window per origin, folding, relays forwarding and returning acks), the incarnation fence (R20), the harness start +after the barrier (R17), native restart patterns (R16), and W1's publication list (a throwing derive, a chain A→B→C +where B throws, two owners alternating with a derive over both, reentrancy, a watch started after 150 frames, stop +during a running call, a retire with a derive over it, a reload during a routine call, GC counts of retained +revisions). + +--- + +## 29. Telemetry + +Chord reports traces, events and metrics through `@earendil-works/pi-telemetry`; one adapter per host +(`createHost({ telemetry })`, NOOP by default) decides sampling, buffering, export and filtering. + +- **Types:** the adapter is a `TelemetryContext` (`startSpan` plus optional readonly `metrics` and `propagation` + capabilities); only hosts hold it, to start roots. `Context.telemetry` is always a `TelemetrySpan` (the NOOP span by + default); every span carries its adapter's `metrics` and `propagation`, so code with a context can record metrics + and inject carriers. `withTelemetry(span, context)`, `traceIn(options, fn, context)` (context last; pi change with S1); `value()` and keys stay. + `Ctx` extends `Context`, so `ctx.telemetry` is the current span; app values such as the principal stay out of + Durable's context. `ctx.trace(name, attributes, fn)` creates a child span and gives `fn` a child `ctx`. +- **Adapters trust typed inputs:** no validation of malformed caller values and no try/catch around recording; only + carriers from another process are checked (keys present, strings), else the root. Callers never protect themselves + against broken adapters. +- **Across links:** `call` and `hold` carry the caller's carrier as a bounded header field (section 22); the provider + extracts it and runs the method under it; relays add a `chord.relay` span. + + ``` + chord.call Notes.add [tab A] + └─ chord.relay [server] + └─ chord.method Notes.add [session:S1] principal=guest + · event chord.commit docs=2 ops=5 + ``` + +- **Shared work** (one activation for several getters, a worker start) is its own root span linked to its first + trigger; each waiting span links to it. Routine runs, job runs, eviction, backoff retries and background closes are + root spans linked to their trigger where one exists. +- **Tiers:** spans for calls, methods, relays, holds, activations, opens, closes, loads, revocations, resolves, routine + runs and job runs; events for commits inside methods, retries and parking; metrics for state messages, WebSocket + bytes, commit durations, in-flight calls, active holds, open links, park durations and backpressure waits. IDs are + attributes of spans, never of metrics. +- **Nothing is measured or walked for telemetry:** only sizes the code already has. Errors carry a fixed category and + a message truncated to 500 characters; cancelling is not an error. +- **A typed telemetry schema** ships with Chord, as pi-ai (`AI_TELEMETRY_SCHEMA`) and Durable + (`DURABLE_TELEMETRY_SCHEMA`) have; call sites that only have a context use the generic `addEvent`. +- **Storage and reading:** `@earendil-works/chord/telemetry-sqlite` over `chord/sqlite` (Node and Cloudflare): one + row per settled span (events and links inline), metrics aggregated per series per 10 s, a 1 s batched flush with + retention (7 days), a bounded queue (dropped and counted). The shared read interface `TelemetryStore` (Jaeger + style): `findTraces` (name, attributes, errors, time window, limit, cursor) returns complete traces, `getTrace`, + `findMetrics`. Spans carry global ids and a trace id; links carry trace and span id. One renderer + (`pi-telemetry/render`) prints links and never follows them. A web trace viewer in the playground reads the store. +- **Durable** (done): submit and phase spans, stored carriers on submissions and tasks (`TaskOptions.trace`, creation + links), events after commit, report events; pi-ai emits `pi.ai.request`/`pi.ai.poll` from one observer in + `lazyStream`. Persisted Chord jobs store their start carrier. + +## 30. Cost model + +As little compute and memory as possible while staying correct; Delta was built for this (its own primitives are +practically free: `diffRevisions` on shared subtrees 0.018 ms, `applyImmutable` per small change 0.004 ms). Measured +before/after per hop: `apps/p2/docs/p3-reviews/v2-cost-model.md`. + +| Hop | Work | +| --- | --- | +| commit | Delta's `prepare` (copies and checks only placed values), adopt by pointer swap; storage writes the delta | +| reads in a transaction | the draft overlay, no copy; unchanged = no write | +| subscribers on the same host | the same value and ops by reference | +| derives | compute plus `diffRevisions`; unchanged parts stay the same objects (`mapShared`, cookbook) | +| send | one encode per touched hold member per commit at its origin (the member's Delta encoder); while the window is full, per op one path key and one map lookup (leaf index), and one encode per waiting member per freed slot | +| relay, per message | decode the envelope, swap the origin (one shallow copy), encode the envelope; parts copied once into the outgoing buffer, never parsed; per `bound`/`state` one FIFO entry; two bytes of origin per message upstream | +| relay, per hold | the address's params checked once (route admission, `authorize`) | +| receive | the compiled `MESSAGE` check on untrusted links; arguments validated once at the provider (compiled closed schema), params checked; one `ack` per `bound`/`state` message | +| heartbeat | one timestamp write per received message; one timer per host | +| apply | `applyImmutable`, structure shared | +| new hold | the value encoded once (unavoidable) | +| memory | one authoritative value per document per host; replicas and derived values share structure; one cached value per adapted binding | + +**Cost guards:** Chord reports full-value walks, deep copies, encodes, schema validations and materializations as +ordinary telemetry metrics, always on (no test mode; with the NOOP adapter they cost an empty call where such work +happens). Tests use the in-memory adapter and assert them per hop (e.g. one appended note to a 500 KB document with +10 viewers: no walks, no copies, one encode per hold part at its origin and none at relays, no validations); a `vitest bench` suite measures the hops. + +--- + +## 31. Rejected + +`replicatedState` and its source family as public API; a separate IndexedDB backend; generic JSON rows and +store-maintained index entries; last-write-per-row compaction; recording raw SQL statements or SQLite changesets; +plugins' own databases holding source data; rollback after losing the driver role; JSON-field operands in bulk +updates (until needed); a lazy-lifetime flag; a `lifetime` option; derive `previous`. Drain-based handover; version fences without transactions; object restart for state changes; Durable's Session as the +object store; volatile documents; per-group lines; generic call or request ids; optimistic transactions; an outbox for +transient delivery errors; revoking proxies on `.value`; resources replaced in place (`replace`, `deps`, stale +handles); registries created at runtime and entries resolved through mutable indirection (contributions that change +with their contributor, readable contributions and readers looking up the current value at use are fine, R25); generic +registry keys with a `conflict` flag; +`restart_required`; detached references counting toward presence; contracts taken from the running app; ceremony +(`native/` directories, `defineNative`, lints); weakening type checking; "magic" input-field matching for delete; a +factory trial; a migrate dry run; principal field patterns and re-stamping for revocation; `storeKey`; a consumer contract check during staging; reload votes; per-plugin rollback within a +host; `defaultProjectTrust: "always"`; loader variants A and B; a topology layer above the host API; a stream type in +Chord; callbacks as method arguments; `opaque` members; numeric middleware priorities; placement as data naming an +input field; host-name strings in app code; host values (start data and platform bindings are core objects); +consumer-side linger; registry composition as a staging check; a JSONL document storage; `tx.get` for documents; validating state and documents on commits or reads (migration results are validated once, R8); copying values on same-host calls; op-rewriting projections and a +view algebra; keeping natives across reloads by name; Chord ending the process on a storage failure; an `optional` +flag on references; cross-relay reset coordination; count limits that refuse calls or holds; relays acking per hop, relay resets +(`reset { hold }`, `reset { holds }`), relay recovery and id renumbering at relays (replaced by flow control per +origin); a byte window, credit messages, snapshot admission and debt; `reset` from a consumer; a round-robin of +relayed calls at the relay; `state: false` on references (C8); `inVariant`/`expectVariant` (C15); per-call deadlines. diff --git a/packages/chord/docs/next/durable-observers/book.md b/packages/chord/docs/next/durable-observers/book.md new file mode 100644 index 000000000..71f6f4833 --- /dev/null +++ b/packages/chord/docs/next/durable-observers/book.md @@ -0,0 +1,432 @@ +--- +title: In the Step: when Durable's views change, and who hears it +cover: In the / Step +kicker: An explainer · Chord v2 slice 6 and pi-durable · 2026-10-26 +subtitle: Why Durable's views must change inside the storage's commit, why the first build broke hosts, and what the "publication at adoption" proposal changes +stats: + - 1 | Durable commit, seen as one change + - 4 | regressions the first build had, all in one gap + - 7 | things that change for hosts compared with HEAD + - 5 | decisions for the owner +note: | + Written for the owner from the build of `/home/badlogic/pi-observers` (uncommitted), the four reviews + (`/tmp/observers-review-{astra,fable,opus,sol}.md`) and the four checks of the proposal + (`/tmp/observers-proposal-{astra,fable,opus,sol}.md`), whose probes ran against HEAD, the first build and the + proposal. Code is Durable's `session/session.ts` and `harness/*.ts` at HEAD, simplified. Nothing here is decided + beyond what the decision log says ("Slice 6 first decisions", refined 2026-10-26). +colophon: In the Step · first edition · 2026-10-26 · from the observers build, its reviews and the proposal checks +--- + +# Before we begin {.preface} + +Mario's phone shows a p3 conversation. Next to the transcript a small status line reads "last: tool call, 1 task +running". It is a Chord derive over two Durable views: the conversation view (the transcript) and the task graph (the +running tasks). + +The model asks for a tool. Durable makes one commit, seq 41, that appends the tool-call entry and creates the tool's +task. The phone should go from "last: assistant, 0 running" to "last: tool call, 1 running" in one step. Today it can +show "last: tool call, 0 running" in between: a state that never existed. Slice 6 also puts Durable and Chord into one +store with one commit step, where every owner's memory must show a complete prefix of the log (S1, W1). Both need the +same thing: Durable's views must change inside the storage's commit, not after it. + +This book explains where Durable publishes a commit today, why moving half of it broke hosts, and what moving all of +it changes. + +## Seven things to know + +:::findings +1. **Durable adopts inside the storage, but publishes after it.** `Storage.commit(writes, adopt, context)` calls + `adopt` once the batch is durable; Durable's views, scheduler and host listeners all run later, after + `Storage.commit` returned. +2. **Two watches tear one commit.** A derive over two Durable views sees seq 41 in one view before the other. +3. **The first build moved only Durable's internal observers.** Host listeners and telemetry stayed after the + storage. Everything a host could see in between came out in the wrong order. +4. **All four regressions live in one gap:** the time between `adopt` and the end of `Storage.commit`. +5. **The proposal removes the gap:** the whole publication happens inside `adopt`. +6. **It changes seven things a host could observe**, most of them timing nobody documented, one a bug fix, one a real + problem for Chord. +7. **The real problem has a small fix:** host listeners one microtask after `adopt`, outside Chord's batch (Opus's + variant). +::: + +## Words used throughout + +**Durable** is pi's agent harness; a **Session** is its one line of atomic commits. **Storage** is Durable's storage +interface; `Storage.commit(writes, adopt, context)` stores a batch and calls **`adopt(seq)`** once it is durable. +Durable's **trackers** hold each loaded document's value in memory. **Internal observers** are Durable's own commit +listeners (`observeCommits`): the conversation **view** mounts, the **task graph**, the **scheduler** and the +**submissions**. **Host listeners** are what an app registers with `subscribeCommits`. A **watch** is a +`WatchHandle`: its listener runs a microtask after a frame arrives. **The gap** is the time between `adopt` and the +moment `Storage.commit` resolves. In Chord's store, the **commit step** is the store-wide serialized part of a commit +(backend transaction plus adoption), and the **batch** is Chord's synchronous adopt, mark and flush around it. +Letters in traces: **A** = `adopt`, **R** = `Storage.commit` resolves, **H** = host listeners, **C** = +`session.commit()` resolves. + +# Today {#today lede="Where a Durable commit is adopted and where it is published, at HEAD."} + +A Durable commit runs on the Session's line from start to end. Inside, `#runCommit` hands the batch to the storage +with an `adopt` callback, waits for it, and only then publishes: + +```ts +// session.ts, #runCommit, at HEAD (simplified) +const adopt = (seq: Seq): void => { + adoption = { documents: tx.adopt(seq) }; // trackers swap to the new values +}; +seq = await this.#storage.commit(writes, adopt, context); +// ... back in the Session, after R: +this.#publish(seq, writes, documents, context); // telemetry events, internal observers, host listeners +return result; // C +``` + +The SQLite storage calls `adopt` right after `COMMIT` returned. JSONL calls it and then still awaits its own cleanup +of sidecar files before it resolves: + +```ts +// jsonl/storage.ts +adopt(seq); // A +await this.reclaimSidecars(reclamations, context); // still inside Storage.commit +return seq; // R +``` + +:::sequence animate | One Durable commit at HEAD. The trackers change inside the storage; everything that tells anyone about it happens after the storage returned. +actor session "Session" "the line" accent +actor storage "Storage" "JSONL" muted +actor obs "internal observers" "views, graph, scheduler" +actor host "host" "subscribeCommits" +actor watch "watch listeners" "microtasks" blue +session -> storage: commit(writes, adopt) +storage -> storage: write, marker durable +storage => session: A: adopt(41), trackers swap +storage -> storage: reclaim sidecars +storage --> session: R +session -> obs: telemetry events, then observers (views advance, waits resolve) +session -> host: H: subscribeCommits(41) +obs -> watch: frames queued, delivered next microtask +session --> session: C +::: + +Two things follow. A host listener always runs before any watch listener of the same commit: the observers queue the +watch deliveries as microtasks, and the host listeners run synchronously right after them. And nothing is published +until the storage is completely done. + +## The tear + +Chord follows a Durable view through its watch. Two views mean two watches, and each delivers in its own microtask: + +```ts +const status = self.derive([conversationView, taskGraph], ([view, graph]) => ({ + last: view.entries.at(-1)?.kind, + running: graph.tasks.filter((t) => t.status === "running").length, +})); +``` + +```trace +commit 41 (entry + task) + microtask 1: view watch delivers 41 → derive: last "toolCall", running 0 ← never existed + microtask 2: graph watch delivers 41 → derive: last "toolCall", running 1 +``` + +The phone receives both. Chord's rule W1 says a commit changes its sources inside one step and the derives over them +recompute once, right after. For Durable's views that means: they must change inside `adopt`, and Chord must follow +them as sources marked there, not through watches. + +## Why after is not good enough for the store + +In Chord's store, Durable's storage (`StoreStorage`) runs `adopt` inside the store's commit step, and the step lets +the next owner's commit start as soon as `adopt` returned. Durable's `#publish` runs later, after its `await`. In +between, a Chord object's commit can adopt, flush and recompute a derive that reads a Durable view still at 40. The +owner ruled out the alternative, a store that waits for Durable's continuation: no Durable special cases in the store. + +# The First Build {#first lede="Moving half the publication into adopt, and the four things that broke."} + +The first build moved Durable's internal observers into `adopt` and left the rest of `#publish` (host listeners, +telemetry events) after R. To keep host code from waking too early, it added `afterPublish(callback)`: observers queue +what wakes a host (resolving a wait, aborting a run, `onWake`), and the queue runs after the host listeners. + +```ts +const adopt = (seq: Seq): void => { + const documents = tx.adopt(seq); + this.#observeStored(seq, writes, documents); // internal observers, inside A +}; +seq = await this.#storage.commit(writes, adopt, context); +this.#publish(...); // host listeners, after R +this.#settle(); // afterPublish callbacks +``` + +Durable's state (views, the scheduler's idle state, the metric counts) now changes at A. Hosts hear about it at H, +after R. Everything that runs in between sees a commit that is half published. Four reviewers found four ways that +shows, each reproduced with a probe. + +:::steps | The four regressions ++++ Watch listeners before host listeners +The observers call `Watch.advance()` inside A; the watch delivers a microtask later. The storage's own `await` and the +Session's `await` resume later still, so the watch listener runs first. + +```trace +HEAD: A … R → H → watch +first build: A (watch queued) → watch → R → H +``` + +Even with the memory storage. With JSONL the watch listener runs before `Storage.commit` resolved; if the storage then +rejects, the watch has shown a commit whose `commit()` failed. ++++ waitForIdle answers early +The scheduler's idle state is updated in A. A host that calls `waitForIdle()` in the gap gets the fast path ("already +idle") and resumes before H. + +```ts +harness.subscribeCommits((p) => seen.push(p.seq)); +// a task-graph watch listener runs in the gap: +await harness.waitForIdle(ctx); // resolves before seen has the terminal commit +``` ++++ A late subscriber misses the commit +When nothing listened at A, the publication is never built. A host that subscribes during the gap gets nothing for +that commit; at HEAD the listener list was read after R, so it did. ++++ The metric drifts after a rejection +A storage that rejects after A (JSONL's cleanup fails): the scheduler counted the new task in A and queued its `+1` +for the metric; the failure seals the scheduler, which subtracts `-1`; the queued `+1` is dropped with the failed +commit. + +```trace +HEAD: [] (nothing observed, net 0) +first build: [-1] (net -1) +``` +::: + +Fable found three more of the same kind: a host listener that closes the Harness makes resolved waits reject; +scheduler line jobs queued in A jump ahead of a host listener's own commit; abort listeners moved after H. The +patches each reviewer suggested were more `afterPublish`. The owner called it what it is: special-casing. + +# Publication at Adoption {#proposal lede="One rule: a commit is published when it is adopted, all of it, inside adopt."} + +The gap is the bug. The first build created it by splitting the publication. The proposal moves all of it: + +```ts +const adopt = (seq: Seq): void => { + const documents = tx.adopt(seq); // trackers, as today + const publication = this.#publication(seq, writes, documents); + this.#events(writes); // pi.durable.entry / .submission, contained + this.#notifyInternal(publication); // views, graph, scheduler, submissions + this.#notifyHosts(publication); // subscribeCommits +}; +seq = await this.#storage.commit(writes, adopt, context); +// after R: touch documents, sweep, fail() for an error an observer threw; nothing is published here +``` + +The order inside is HEAD's: telemetry events, internal observers, host listeners. `afterPublish` disappears. Waits +need nothing special: an observer resolves a promise in A, and the waiter's continuation runs later by nature, after +the host listeners already ran synchronously. One rule replaces the special cases: `adopt` runs no task code. The +only synchronous path into task code is `AbortController.abort()` of a newly marked run (abort listeners run inside +`abort()`), so that one is queued with `queueMicrotask`. `adopt` never throws: every listener, every report and every +telemetry call inside it is contained. + +:::sequence animate | The same commit under the proposal. Everything happens inside A; R and C only end it. +actor session "Session" "the line" accent +actor storage "Storage" "JSONL" muted +actor obs "internal observers" "views, graph, scheduler" +actor host "host" "subscribeCommits" +actor watch "watch listeners" "microtasks" blue +session -> storage: commit(writes, adopt) +storage -> storage: write, marker durable +storage => session: A: adopt(41) +session -> session: trackers swap, telemetry events +session => obs: observers: views advance, waits resolve +session => host: H: subscribeCommits(41) +obs -> watch: watches, waits, abort: next microtasks +storage -> storage: reclaim sidecars +storage --> session: R +session --> session: C +::: + +What this does to the regressions, verified by Opus and Fable with probes against HEAD: + +| regression | HEAD | first build | proposal | +| --- | --- | --- | --- | +| watch listeners vs H | H, watch | **watch, H** | H, watch | +| `waitForIdle` in the gap | after H | **before H** | after H | +| metric after a rejection following A | 0 | **−1** | 0 (+1, −1) | +| `close()` from H, then `waitForTask` | resolves | **rejects** | resolves | +| scheduler jobs vs H's own commit | H first | **jobs first** | H first | +| stuck state, lost reports | none | yes | none | +| allocations per commit | baseline | more | about HEAD | + +HEAD's whole non-e2e Durable suite (73 files, 1404 tests) passes on a build of the proposal. The four tests the first +build added that encode its split fail, as they should, and are replaced. + +# What Changes for Hosts {#changes lede="Seven things a host, a task or a storage could observe, compared with HEAD."} + +Nothing in the public API changes. These are the differences in behaviour the four checks found. + +## 1. With JSONL, everything runs before the storage is done + +```trace +HEAD: marker → A → cleanup … → R → observers → H → watches, waits → C +proposal: marker → A { observers, H } → watches, waits, abort → cleanup … → R → C +``` + +Host and task code woken by a commit now runs while JSONL still cleans up its sidecars. It cannot reach the storage +through the Session: the line is still held, so a commit or read it starts waits. The memory, SQLite and IndexedDB +storages do no work after A, so for them only the microtask placement differs, and the order is HEAD's. No spec +section promised "after `Storage.commit` resolved". Verdict: harmless; the spec gets one sentence. + +## 2. Abort listeners fire after host listeners + +```ts +runtime.signal.addEventListener("abort", () => log(hostSaw(id))); +// HEAD: false (abort before H) proposal: true (abort after H) +``` + +At HEAD the scheduler aborted a newly marked run inside `#publish`, before the host listeners. Under the proposal the +abort is a microtask after A, so after H. No contract fixes this order. Decision for the owner. + +## 3. A commit the storage rejects after A is published + +A host keeps a replica from `subscribeCommits`. JSONL wrote the marker (the commit is durable), called `adopt`, then +failed while cleaning up and rejected. + +```trace +HEAD: durable, but no observer and no host listener ran → the replica misses it + → replica ≠ storage; waits of that commit reject; the Session fails +proposal: published at A → the replica gets it → replica = storage + → waits of that commit resolve; the Session fails after R +``` + +Durable's spec says a replica fed by `subscribeCommits` "stays equal to Storage"; HEAD breaks that, the proposal keeps +it. Spec invariant 2 changes from "published only after its storage commit succeeds" to "published once it is +durable, even when `commit()` then rejects". Decision for the owner; all four checks recommend it. + +## 4. A subscriber that arrives late misses the commit + +```ts +// host code that runs in the gap, between A and R: +harness.subscribeCommits(listener); // HEAD: gets commit 41; proposal: does not, 41 was published at A +``` + +Consistent with the rule: a listener added after the publication misses it, as one added after `#publish` did at +HEAD. Documented, nothing to decide. + +## 5. An observer that throws fails the Session after the host listeners + +At HEAD, a throwing internal observer failed the Session at once, inside `#publish`: close listeners and the +`session_failed` report came before the host listeners. Under the proposal, `adopt` records the error, lets the other +observers and the host listeners run, and the Session fails after R. The commit stands, as before. Failing inside A +instead would make Durable's storage guard reject the caller of a durable commit with `SessionFailed`. Verdict: keep +it so. + +## 6. Telemetry that throws no longer breaks memory + +At HEAD, `#publish` called `context.telemetry.addEvent` for each entry and submission without protection. A throwing +adapter rejected a commit that was already durable and adopted, skipped every observer and host listener, and left the +Session healthy: the scheduler and the views missed the commit for good. Inside `adopt` nothing may throw, so the +proposal contains it and reports it; the commit resolves. This fixes a HEAD bug. The report needs a type (`host_error` +with callback `"telemetry"` is the suggestion). The same containment must cover Durable's report path itself +(`safeReport` records a metric and a span event outside its `try`), or `adopt` is not truly throw-free. + +## 7. Host listeners would run inside Chord's batch + +This is the one that matters. In Chord's store, `StoreStorage` runs Durable's `adopt` inside the store's commit step, +inside Chord's batch, where Chord refuses reentrant calls (`self.tx`, `get`, `watch`, `close` throw +`invalid/reentrant`). Under the proposal the host listeners run inside `adopt`, so inside that batch: + +```ts +// p3 code that mirrors Durable's commits into a Chord object +harness.subscribeCommits((publication) => + chord((ctx) => audit.record(publication.seq, ctx), context)); // throws invalid/reentrant +``` + +The listener keeps every rule Durable states (synchronous, no Session calls) and still fails. Its error is reported; +an `onReport` that itself uses Chord throws too, and a throwing `onReport` is dropped, so the work vanishes without a +trace. And every host listener's run time is added to the store-wide commit step that every other object's commit +waits for. + +:::flow wide | Under the proposal, a host listener runs inside Chord's batch; Chord calls from it are refused. +row "Store commit step (one at a time for the whole store)" accent +box backend "backend transaction" "`BEGIN … COMMIT`" muted +box adopt "Durable's adopt" "trackers, views, scheduler" accent +box hostin "host listeners" "inside the batch" dashed +row "Chord" +box flush "flush" "derives, wire, watches queued" blue +box refused "`self.tx` from a listener" "`invalid/reentrant`" dashed +arrow backend -> adopt "durable" +arrow adopt -> hostin "proposal" +arrow hostin -> refused "Chord call" +arrow adopt -> flush "after the step" +::: + +# The Microtask Variant {#variant lede="Host listeners one microtask after adopt: same results, outside Chord's batch."} + +Opus proposed a variant that keeps every result of the proposal and moves only the host listeners out of the step. +The publication is still built in A and the internal observers still run there. The host listeners are dispatched in +one microtask that is queued before the internal observers run: + +```ts +// inside adopt, after tx.adopt(seq) +this.#events(writes); // contained +if (this.#commitListeners.size > 0) { + const listeners = [...this.#commitListeners]; + queueMicrotask(() => { + for (const listener of listeners) + contained(() => listener(stored, context), (error) => this.report(error, LISTENER, context)); + }); +} +for (const listener of [...this.#internalListeners]) + contained(() => listener(observed, context), (error) => { failure ??= { error }; }); +``` + +Microtasks run first in, first out. The host dispatch is queued before anything the observers queue, so the host +listeners still run before every watch delivery, every resolved wait, every abort and `onWake`. And it is a +microtask, so it runs after the storage's synchronous step: after Chord's batch and flush, where Chord calls work. + +:::sequence animate | The variant inside Chord's store. The step stays synchronous and short; host listeners run right after it, first of everything the commit woke. +actor store "store step" "one at a time" accent +actor durable "Durable adopt" "views, scheduler" +actor chord "Chord batch" "mark, flush" blue +actor host "host listeners" +actor later "watches, waits, abort" muted +store -> store: backend COMMIT +store => durable: adopt(41): trackers, events +durable -> host: queue H (microtask 1) +durable -> durable: observers: views advance (sources marked) +durable -> later: queue watches, waits, abort (microtasks 2…) +store => chord: flush: derives over both views recompute once +store --> store: step ends, next commit may start +host -> host: H runs: Chord calls allowed +later -> later: then watches, waits, abort +::: + +Chord needs nothing Durable-specific: only that its commit step and its flush are synchronous, which its design +already states. The store never knows Durable's continuation, as the owner required. + +What the variant gives up compared with the proposal: host listeners are no longer called synchronously from inside +`adopt`, but one microtask later, still on the Session line and still before C. Opus ran HEAD's full non-e2e suite +against the variant (1404 of 1404) and all twelve probes: the same results as the proposal. One detail to settle while +building: the listener list is copied at A, so a listener removed between A and the microtask is still called once; +copying it inside the microtask avoids that, but then a listener added in between is called too. + +# Decisions {.epilogue lede="What the owner decides before the rebuild."} + +1. **The variant or the proposal.** The variant (recommended by Opus) runs host listeners one microtask after `adopt`, + outside Chord's batch. The proposal runs them inside `adopt` and needs the documented rule "no Chord calls from + `subscribeCommits` or `onReport`". +2. **Abort listeners after host listeners** (change 2): acceptable, or must the abort keep HEAD's place before them. +3. **Publish a commit the storage rejects after `adopt`** (change 3): recommended by all four checks; it fixes HEAD's + replica gap and changes spec invariant 2. +4. **The report type of a throwing telemetry call** (change 6): suggested `host_error` with callback `"telemetry"`. +5. **HEAD's order inside the publication** (events, internal observers, host listeners) and a fully throw-free + `adopt`, the report path included. These are fixes rather than choices; listed so they are not lost. + +After the decisions: the spec rewrites (invariant 2, section 4 and its pseudo-code, the Storage contract, section 10 +failures), new tests in place of the four that encode the split (order with the memory storage and with a storage that +works after `adopt`, `waitForIdle` in the gap, publication on a rejection with a balanced metric, close in the gap, +Chord-like reentrancy, telemetry throws), a rebuild by a fresh builder, and the four reviewers again. + +# Sources {.appendix} + +| what | where | +| --- | --- | +| HEAD code | `packages/durable/src/session/session.ts` (`#runCommit`, `#publish`), `harness/view.ts`, `harness/task-graph.ts`, `harness/scheduler.ts`, `harness/submissions.ts`, `storage/jsonl/storage.ts` | +| first build | `/home/badlogic/pi-observers` (uncommitted) | +| reviews of the first build | `/tmp/observers-review-{astra,fable,opus,sol}.md` | +| the proposal | `/tmp/observers-proposal.md` | +| checks of the proposal | `/tmp/observers-proposal-{astra,fable,opus,sol}.md`; probes in `/tmp/obsprobe-*` | +| decisions so far | `packages/chord/docs/next/decisions.md`, "Slice 6 first decisions" | +| Chord rules | `packages/chord/docs/next/design.md` §11 (W1, the batch), §24 (Durable integration) | diff --git a/packages/chord/docs/next/durable-observers/index.html b/packages/chord/docs/next/durable-observers/index.html new file mode 100644 index 000000000..fe6961cd5 --- /dev/null +++ b/packages/chord/docs/next/durable-observers/index.html @@ -0,0 +1,517 @@ + + +In the Step: when Durable's views change, and who hears it + + + + +

+
+
An explainer · Chord v2 slice 6 and pi-durable · 2026-10-26
+

In the
Step

+

Why Durable's views must change inside the storage's commit, why the first build broke hosts, and what the "publication at adoption" proposal changes

+
1Durable commit, seen as one change
4regressions the first build had, all in one gap
7things that change for hosts compared with HEAD
5decisions for the owner
+

Written for the owner from the build of /home/badlogic/pi-observers (uncommitted), the four reviews +(/tmp/observers-review-{astra,fable,opus,sol}.md) and the four checks of the proposal +(/tmp/observers-proposal-{astra,fable,opus,sol}.md), whose probes ran against HEAD, the first build and the +proposal. Code is Durable's session/session.ts and harness/*.ts at HEAD, simplified. Nothing here is decided +beyond what the decision log says ("Slice 6 first decisions", refined 2026-10-26).

+
+
+ + +
+

Before we begin

+

Mario's phone shows a p3 conversation. Next to the transcript a small status line reads "last: tool call, 1 task +running". It is a Chord derive over two Durable views: the conversation view (the transcript) and the task graph (the +running tasks).

+

The model asks for a tool. Durable makes one commit, seq 41, that appends the tool-call entry and creates the tool's +task. The phone should go from "last: assistant, 0 running" to "last: tool call, 1 running" in one step. Today it can +show "last: tool call, 0 running" in between: a state that never existed. Slice 6 also puts Durable and Chord into one +store with one commit step, where every owner's memory must show a complete prefix of the log (S1, W1). Both need the +same thing: Durable's views must change inside the storage's commit, not after it.

+

This book explains where Durable publishes a commit today, why moving half of it broke hosts, and what moving all of +it changes.

+

Seven things to know

+
    +
  1. Durable adopts inside the storage, but publishes after it. Storage.commit(writes, adopt, context) calls +adopt once the batch is durable; Durable's views, scheduler and host listeners all run later, after +Storage.commit returned.
    2. +
  2. Two watches tear one commit. A derive over two Durable views sees seq 41 in one view before the other.
    3. +
  3. The first build moved only Durable's internal observers. Host listeners and telemetry stayed after the +storage. Everything a host could see in between came out in the wrong order.
    4. +
  4. All four regressions live in one gap: the time between adopt and the end of Storage.commit.
    5. +
  5. The proposal removes the gap: the whole publication happens inside adopt.
    6. +
  6. It changes seven things a host could observe, most of them timing nobody documented, one a bug fix, one a real +problem for Chord.
    7. +
  7. The real problem has a small fix: host listeners one microtask after adopt, outside Chord's batch (Opus's +variant).
    8. +
+

Words used throughout

+

Durable is pi's agent harness; a Session is its one line of atomic commits. Storage is Durable's storage +interface; Storage.commit(writes, adopt, context) stores a batch and calls adopt(seq) once it is durable. +Durable's trackers hold each loaded document's value in memory. Internal observers are Durable's own commit +listeners (observeCommits): the conversation view mounts, the task graph, the scheduler and the +submissions. Host listeners are what an app registers with subscribeCommits. A watch is a +WatchHandle: its listener runs a microtask after a frame arrives. The gap is the time between adopt and the +moment Storage.commit resolves. In Chord's store, the commit step is the store-wide serialized part of a commit +(backend transaction plus adoption), and the batch is Chord's synchronous adopt, mark and flush around it. +Letters in traces: A = adopt, R = Storage.commit resolves, H = host listeners, C = +session.commit() resolves.

+
+ +
I

Today

Where a Durable commit is adopted and where it is published, at HEAD.

+

A Durable commit runs on the Session's line from start to end. Inside, #runCommit hands the batch to the storage +with an adopt callback, waits for it, and only then publishes:

+
// session.ts, #runCommit, at HEAD (simplified)
+const adopt = (seq: Seq): void => {
+  adoption = { documents: tx.adopt(seq) };          // trackers swap to the new values
+};
+seq = await this.#storage.commit(writes, adopt, context);
+// ... back in the Session, after R:
+this.#publish(seq, writes, documents, context);      // telemetry events, internal observers, host listeners
+return result;                                       // C
+

The SQLite storage calls adopt right after COMMIT returned. JSONL calls it and then still awaits its own cleanup +of sidecar files before it resolves:

+
// jsonl/storage.ts
+adopt(seq);                                           // A
+await this.reclaimSidecars(reclamations, context);   // still inside Storage.commit
+return seq;                                           // R
+
Sessionthe lineStorageJSONLinternal observersviews, graph, schedulerhostsubscribeCommitswatch listenersmicrotaskscommit(writes, adopt)write, marker durableA: adopt(41), trackers swapreclaim sidecarsRtelemetry events, then observers (views advance, waits resolve)H: subscribeCommits(41)frames queued, delivered next microtaskC
Figure 1. One Durable commit at HEAD. The trackers change inside the storage; everything that tells anyone about it happens after the storage returned.
+

Two things follow. A host listener always runs before any watch listener of the same commit: the observers queue the +watch deliveries as microtasks, and the host listeners run synchronously right after them. And nothing is published +until the storage is completely done.

+

The tear

+

Chord follows a Durable view through its watch. Two views mean two watches, and each delivers in its own microtask:

+
const status = self.derive([conversationView, taskGraph], ([view, graph]) => ({
+  last: view.entries.at(-1)?.kind,
+  running: graph.tasks.filter((t) => t.status === "running").length,
+}));
+
commit 41 (entry + task)
+  microtask 1: view watch delivers 41   → derive: last "toolCall", running 0    ← never existed
+  microtask 2: graph watch delivers 41  → derive: last "toolCall", running 1
+

The phone receives both. Chord's rule W1 says a commit changes its sources inside one step and the derives over them +recompute once, right after. For Durable's views that means: they must change inside adopt, and Chord must follow +them as sources marked there, not through watches.

+

Why after is not good enough for the store

+

In Chord's store, Durable's storage (StoreStorage) runs adopt inside the store's commit step, and the step lets +the next owner's commit start as soon as adopt returned. Durable's #publish runs later, after its await. In +between, a Chord object's commit can adopt, flush and recompute a derive that reads a Durable view still at 40. The +owner ruled out the alternative, a store that waits for Durable's continuation: no Durable special cases in the store.

+
+ +
II

The First Build

Moving half the publication into adopt, and the four things that broke.

+

The first build moved Durable's internal observers into adopt and left the rest of #publish (host listeners, +telemetry events) after R. To keep host code from waking too early, it added afterPublish(callback): observers queue +what wakes a host (resolving a wait, aborting a run, onWake), and the queue runs after the host listeners.

+
const adopt = (seq: Seq): void => {
+  const documents = tx.adopt(seq);
+  this.#observeStored(seq, writes, documents);       // internal observers, inside A
+};
+seq = await this.#storage.commit(writes, adopt, context);
+this.#publish(...);                                    // host listeners, after R
+this.#settle();                                        // afterPublish callbacks
+

Durable's state (views, the scheduler's idle state, the metric counts) now changes at A. Hosts hear about it at H, +after R. Everything that runs in between sees a commit that is half published. Four reviewers found four ways that +shows, each reproduced with a probe.

+
The four regressions

Watch listeners before host listeners

The observers call Watch.advance() inside A; the watch delivers a microtask later. The storage's own await and the +Session's await resume later still, so the watch listener runs first.

+
HEAD:        A … R → H → watch
+first build: A (watch queued) → watch → R → H
+

Even with the memory storage. With JSONL the watch listener runs before Storage.commit resolved; if the storage then +rejects, the watch has shown a commit whose commit() failed.

+

waitForIdle answers early

The scheduler's idle state is updated in A. A host that calls waitForIdle() in the gap gets the fast path ("already +idle") and resumes before H.

+
harness.subscribeCommits((p) => seen.push(p.seq));
+// a task-graph watch listener runs in the gap:
+await harness.waitForIdle(ctx);          // resolves before seen has the terminal commit
+

A late subscriber misses the commit

When nothing listened at A, the publication is never built. A host that subscribes during the gap gets nothing for +that commit; at HEAD the listener list was read after R, so it did.

+

The metric drifts after a rejection

A storage that rejects after A (JSONL's cleanup fails): the scheduler counted the new task in A and queued its +1 +for the metric; the failure seals the scheduler, which subtracts -1; the queued +1 is dropped with the failed +commit.

+
HEAD:        []        (nothing observed, net 0)
+first build: [-1]      (net -1)
+
+

Fable found three more of the same kind: a host listener that closes the Harness makes resolved waits reject; +scheduler line jobs queued in A jump ahead of a host listener's own commit; abort listeners moved after H. The +patches each reviewer suggested were more afterPublish. The owner called it what it is: special-casing.

+
+ +
III

Publication at Adoption

One rule: a commit is published when it is adopted, all of it, inside adopt.

+

The gap is the bug. The first build created it by splitting the publication. The proposal moves all of it:

+
const adopt = (seq: Seq): void => {
+  const documents = tx.adopt(seq);                              // trackers, as today
+  const publication = this.#publication(seq, writes, documents);
+  this.#events(writes);                                         // pi.durable.entry / .submission, contained
+  this.#notifyInternal(publication);                            // views, graph, scheduler, submissions
+  this.#notifyHosts(publication);                               // subscribeCommits
+};
+seq = await this.#storage.commit(writes, adopt, context);
+// after R: touch documents, sweep, fail() for an error an observer threw; nothing is published here
+

The order inside is HEAD's: telemetry events, internal observers, host listeners. afterPublish disappears. Waits +need nothing special: an observer resolves a promise in A, and the waiter's continuation runs later by nature, after +the host listeners already ran synchronously. One rule replaces the special cases: adopt runs no task code. The +only synchronous path into task code is AbortController.abort() of a newly marked run (abort listeners run inside +abort()), so that one is queued with queueMicrotask. adopt never throws: every listener, every report and every +telemetry call inside it is contained.

+
Sessionthe lineStorageJSONLinternal observersviews, graph, schedulerhostsubscribeCommitswatch listenersmicrotaskscommit(writes, adopt)write, marker durableA: adopt(41)trackers swap, telemetry eventsobservers: views advance, waits resolveH: subscribeCommits(41)watches, waits, abort: next microtasksreclaim sidecarsRC
Figure 2. The same commit under the proposal. Everything happens inside A; R and C only end it.
+

What this does to the regressions, verified by Opus and Fable with probes against HEAD:

+

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
regressionHEADfirst buildproposal
watch listeners vs HH, watchwatch, HH, watch
waitForIdle in the gapafter Hbefore Hafter H
metric after a rejection following A0−10 (+1, −1)
close() from H, then waitForTaskresolvesrejectsresolves
scheduler jobs vs H's own commitH firstjobs firstH first
stuck state, lost reportsnoneyesnone
allocations per commitbaselinemoreabout HEAD
+

HEAD's whole non-e2e Durable suite (73 files, 1404 tests) passes on a build of the proposal. The four tests the first +build added that encode its split fail, as they should, and are replaced.

+ + +
IV

What Changes for Hosts

Seven things a host, a task or a storage could observe, compared with HEAD.

+

Nothing in the public API changes. These are the differences in behaviour the four checks found.

+

1. With JSONL, everything runs before the storage is done

+
HEAD:     marker → A → cleanup … → R → observers → H → watches, waits → C
+proposal: marker → A { observers, H } → watches, waits, abort → cleanup … → R → C
+

Host and task code woken by a commit now runs while JSONL still cleans up its sidecars. It cannot reach the storage +through the Session: the line is still held, so a commit or read it starts waits. The memory, SQLite and IndexedDB +storages do no work after A, so for them only the microtask placement differs, and the order is HEAD's. No spec +section promised "after Storage.commit resolved". Verdict: harmless; the spec gets one sentence.

+

2. Abort listeners fire after host listeners

+
runtime.signal.addEventListener("abort", () => log(hostSaw(id)));
+// HEAD: false (abort before H)    proposal: true (abort after H)
+

At HEAD the scheduler aborted a newly marked run inside #publish, before the host listeners. Under the proposal the +abort is a microtask after A, so after H. No contract fixes this order. Decision for the owner.

+

3. A commit the storage rejects after A is published

+

A host keeps a replica from subscribeCommits. JSONL wrote the marker (the commit is durable), called adopt, then +failed while cleaning up and rejected.

+
HEAD:     durable, but no observer and no host listener ran → the replica misses it
+          → replica ≠ storage; waits of that commit reject; the Session fails
+proposal: published at A → the replica gets it → replica = storage
+          → waits of that commit resolve; the Session fails after R
+

Durable's spec says a replica fed by subscribeCommits "stays equal to Storage"; HEAD breaks that, the proposal keeps +it. Spec invariant 2 changes from "published only after its storage commit succeeds" to "published once it is +durable, even when commit() then rejects". Decision for the owner; all four checks recommend it.

+

4. A subscriber that arrives late misses the commit

+
// host code that runs in the gap, between A and R:
+harness.subscribeCommits(listener);   // HEAD: gets commit 41; proposal: does not, 41 was published at A
+

Consistent with the rule: a listener added after the publication misses it, as one added after #publish did at +HEAD. Documented, nothing to decide.

+

5. An observer that throws fails the Session after the host listeners

+

At HEAD, a throwing internal observer failed the Session at once, inside #publish: close listeners and the +session_failed report came before the host listeners. Under the proposal, adopt records the error, lets the other +observers and the host listeners run, and the Session fails after R. The commit stands, as before. Failing inside A +instead would make Durable's storage guard reject the caller of a durable commit with SessionFailed. Verdict: keep +it so.

+

6. Telemetry that throws no longer breaks memory

+

At HEAD, #publish called context.telemetry.addEvent for each entry and submission without protection. A throwing +adapter rejected a commit that was already durable and adopted, skipped every observer and host listener, and left the +Session healthy: the scheduler and the views missed the commit for good. Inside adopt nothing may throw, so the +proposal contains it and reports it; the commit resolves. This fixes a HEAD bug. The report needs a type (host_error +with callback "telemetry" is the suggestion). The same containment must cover Durable's report path itself +(safeReport records a metric and a span event outside its try), or adopt is not truly throw-free.

+

7. Host listeners would run inside Chord's batch

+

This is the one that matters. In Chord's store, StoreStorage runs Durable's adopt inside the store's commit step, +inside Chord's batch, where Chord refuses reentrant calls (self.tx, get, watch, close throw +invalid/reentrant). Under the proposal the host listeners run inside adopt, so inside that batch:

+
// p3 code that mirrors Durable's commits into a Chord object
+harness.subscribeCommits((publication) =>
+  chord((ctx) => audit.record(publication.seq, ctx), context));   // throws invalid/reentrant
+

The listener keeps every rule Durable states (synchronous, no Session calls) and still fails. Its error is reported; +an onReport that itself uses Chord throws too, and a throwing onReport is dropped, so the work vanishes without a +trace. And every host listener's run time is added to the store-wide commit step that every other object's commit +waits for.

+
STORE COMMIT STEP (ONE AT A TIME FOR THE WHOLE STORE)CHORDbackend transactionBEGIN … COMMITDurable's adopttrackers, views, schedulerhost listenersinside the batchflushderives, wire, watches queued`self.tx` from a listenerinvalid/reentrantdurableproposalChord callafter the step
Figure 3. Under the proposal, a host listener runs inside Chord's batch; Chord calls from it are refused.
+
+ +
V

The Microtask Variant

Host listeners one microtask after adopt: same results, outside Chord's batch.

+

Opus proposed a variant that keeps every result of the proposal and moves only the host listeners out of the step. +The publication is still built in A and the internal observers still run there. The host listeners are dispatched in +one microtask that is queued before the internal observers run:

+
// inside adopt, after tx.adopt(seq)
+this.#events(writes);                                            // contained
+if (this.#commitListeners.size > 0) {
+  const listeners = [...this.#commitListeners];
+  queueMicrotask(() => {
+    for (const listener of listeners)
+      contained(() => listener(stored, context), (error) => this.report(error, LISTENER, context));
+  });
+}
+for (const listener of [...this.#internalListeners])
+  contained(() => listener(observed, context), (error) => { failure ??= { error }; });
+

Microtasks run first in, first out. The host dispatch is queued before anything the observers queue, so the host +listeners still run before every watch delivery, every resolved wait, every abort and onWake. And it is a +microtask, so it runs after the storage's synchronous step: after Chord's batch and flush, where Chord calls work.

+
store stepone at a timeDurable adoptviews, schedulerChord batchmark, flushhost listenerswatches, waits, abortbackend COMMITadopt(41): trackers, eventsqueue H (microtask 1)observers: views advance (sources marked)queue watches, waits, abort (microtasks 2…)flush: derives over both views recompute oncestep ends, next commit may startH runs: Chord calls allowedthen watches, waits, abort
Figure 4. The variant inside Chord's store. The step stays synchronous and short; host listeners run right after it, first of everything the commit woke.
+

Chord needs nothing Durable-specific: only that its commit step and its flush are synchronous, which its design +already states. The store never knows Durable's continuation, as the owner required.

+

What the variant gives up compared with the proposal: host listeners are no longer called synchronously from inside +adopt, but one microtask later, still on the Session line and still before C. Opus ran HEAD's full non-e2e suite +against the variant (1404 of 1404) and all twelve probes: the same results as the proposal. One detail to settle while +building: the listener list is copied at A, so a listener removed between A and the microtask is still called once; +copying it inside the microtask avoids that, but then a listener added in between is called too.

+
+ +
∴

Decisions

What the owner decides before the rebuild.

+
    +
  1. The variant or the proposal. The variant (recommended by Opus) runs host listeners one microtask after adopt, +outside Chord's batch. The proposal runs them inside adopt and needs the documented rule "no Chord calls from +subscribeCommits or onReport".
    2. +
  2. Abort listeners after host listeners (change 2): acceptable, or must the abort keep HEAD's place before them.
    3. +
  3. Publish a commit the storage rejects after adopt (change 3): recommended by all four checks; it fixes HEAD's +replica gap and changes spec invariant 2.
    4. +
  4. The report type of a throwing telemetry call (change 6): suggested host_error with callback "telemetry".
    5. +
  5. HEAD's order inside the publication (events, internal observers, host listeners) and a fully throw-free +adopt, the report path included. These are fixes rather than choices; listed so they are not lost.
    6. +
+

After the decisions: the spec rewrites (invariant 2, section 4 and its pseudo-code, the Storage contract, section 10 +failures), new tests in place of the four that encode the split (order with the memory storage and with a storage that +works after adopt, waitForIdle in the gap, publication on a rejection with a balanced metric, close in the gap, +Chord-like reentrancy, telemetry throws), a rebuild by a fresh builder, and the four reviewers again.

+
+ +
A

Sources

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
whatwhere
HEAD codepackages/durable/src/session/session.ts (#runCommit, #publish), harness/view.ts, harness/task-graph.ts, harness/scheduler.ts, harness/submissions.ts, storage/jsonl/storage.ts
first build/home/badlogic/pi-observers (uncommitted)
reviews of the first build/tmp/observers-review-{astra,fable,opus,sol}.md
the proposal/tmp/observers-proposal.md
checks of the proposal/tmp/observers-proposal-{astra,fable,opus,sol}.md; probes in /tmp/obsprobe-*
decisions so farpackages/chord/docs/next/decisions.md, "Slice 6 first decisions"
Chord rulespackages/chord/docs/next/design.md §11 (W1, the batch), §24 (Durable integration)
+
+ +

In the Step · first edition · 2026-10-26 · from the observers build, its reviews and the proposal checks

+ + + diff --git a/packages/chord/docs/next/explainer/book.md b/packages/chord/docs/next/explainer/book.md new file mode 100644 index 000000000..05a98d9aa --- /dev/null +++ b/packages/chord/docs/next/explainer/book.md @@ -0,0 +1,4507 @@ +--- +title: Chord v2: a guide to building apps that reload +cover: Chord v2 / A Guide +kicker: A guide · Chord v2 target design · 2026-10-20 +subtitle: How to build an app like p2 from nothing on Chord v2 and pi-durable, one object at a time, so that new plugin code takes over while the agent keeps running +stats: + - 11 | layers in the rebuild, L1 to L11 + - ~1,300 | lines built so far (L1 contracts) + - 6–7k | target size of Chord v2, without tests + - 0.004 ms | Delta's apply per small change +note: | + This guide teaches the **designed** API of Chord v2 (`packages/chord/src/next`, export `@earendil-works/chord/next`) + from `packages/chord/docs/next/design.md` and the newer decision log `packages/chord/docs/next/decisions.md`, which wins + where the two differ. Most of it is **not built yet**: a first kernel of about 10k lines is frozen as a reference and + Chord v2 is being rebuilt layer by layer; only the L1 contract layer exists (`src/next/contracts/`, commit + `7d2bb8ea8`). Code in this guide is therefore the target, not something you can run today. Every chapter says what is + built. API names are working names. Notes marked *review* are points that are undecided, or where the sources + disagree; they name the files. The reader is assumed to know pi-durable; Chord is explained from the start. +colophon: Chord v2 · a guide · first edition · 2026-10-20 · based on design.md (target design v2), redesign-decisions.md (review rounds 2 and 3), chord-next-walkthrough.md, layer-1.md, contracts/README.md, the p3 functionality map, questions.md and the pi-durable README +--- + +# Before we begin {.preface} + +It is late in the evening. Mario is on his phone, connected to his laptop at home through the Radius relay. In session +S1 the agent is twenty minutes into a refactoring; right now it is running a test suite through `bash`. A colleague, +Ann, follows the session read-only from her own browser. Mario notices that the notes panel sorts notes the wrong way +round. He fixes one line in the notes plugin on the laptop and presses **Reload**. + +What should happen is easy to say. The notes panel on both screens shows the new order. The test run keeps running. The +agent's turn continues as if nothing happened. Ann's view does not flicker into an error. Nothing that was written is +lost, and no process restarts. + +The obvious ways to get there fail. **Restarting the processes** loses the turn in flight, the open terminal and every +connection; pi-durable would resume the turn, but the test run and the connections are gone. **Hot-swapping modules** +in place, as front-end tooling does, leaves the old code alive in closures, timers, event listeners and sockets that +nobody knows about: the old sort function keeps running inside a callback someone stored, a file watcher opened by the +old code is never closed, and two versions of the plugin now write the same data without any order. + +Chord v2 is the runtime that makes the first outcome normal. It does so by being strict about one thing: **every piece +of state, every open OS handle and every piece of running work has exactly one owner, an object, and objects only +declare what they need.** Chord itself activates, connects, reloads and ends them. Because Chord knows every timer, +watcher, transaction and reference an object has, it can swap the object's code while its data stays, restart only +what touches the operating system, and route calls between browser tabs, the server and per-session worker processes +without the plugin author writing any of that. + +This guide teaches Chord v2 by building an app called **Studio**: a small version of p2 (Pi Mobile), with a server, +one worker process per session, browser tabs as clients, and an agent from pi-durable on each worker. The first +chapters grow one plugin, `@studio/notes`, until it uses every concept; the last three chapters assemble the rest of a +p2-like app from the same parts. + +:::review | Review: what exists today +Nothing in this guide runs yet except the contract layer. `packages/chord/src/next/contracts/` (L1 part 1: `remote`, +`local`, `method`, `evolve`, `checkAccepts`, the schema helpers) is built and reviewed. The rest of L1 (inputs, +argument checks, principals, errors) is a draft spec (`packages/chord/docs/next/layer-1.md`). The frozen kernel in +`src/next/*.ts` implements an older version of the design (host values, a development freeze, JSONL storage, +descriptions on the wire); do not read it as the target. Build on the design, not on the frozen code. +::: + +## Ten things to know + +:::findings +1. **An app is a tree of hosts.** One server, any number of workers it starts on demand, any number of clients (tabs, + TUIs) that connect to it. Clients reach workers only through the server, which relays. +2. **Objects are the unit of everything.** An object type is a contract (name, input, placement, members, version); an + object is one instance per input and caller. The first `get` creates it, later gets share it, the last release ends + it. +3. **Factories declare, Chord activates.** A factory is synchronous: it declares documents, references, derived + values, OS handles, routines and jobs through `self`, and returns the members. Chord opens, binds and starts them, + and undoes them in reverse. +4. **Links stamp who is calling.** A caller's data never says who it is; the link it arrived on does. Each caller gets + its own object and the variant of members it may use. +5. **Shared data lives once, behind views.** A `local` shared store holds the data; per-caller `remote` views expose + what each caller may see. Only stable identities persist documents. +6. **State changes only in transactions.** Documents are JSON values written in `self.tx`; commits are atomic for + every viewer and travel as small operations, never as whole values. +7. **Failures look like disconnects.** A reference survives link drops, restarts and failures: calls park or fail, + values stay readable, and Chord retries by itself. There are no request ids. +8. **Everything long-lived has a declared form.** Standing behaviour is a routine, work that outlives a call is a job, + input from a person is a flow, behaviour from other plugins comes through a registry. Nothing else may hold a timer + or a callback. +9. **Reload swaps pure objects in place and restarts native ones.** Objects without OS handles get the new code with + their data and running work intact; objects with OS handles close and reopen them. A broken change leaves the whole + host on its old code. +10. **pi-durable runs inside a core object.** The agent harness is a host-lifetime object on each worker; models, + environments and tools reach it through registries, and tools reach Chord through `chord(fn, context)`. +::: + +## Words used throughout + +A **host** is one participant of the app: a process, a browser tab or a Durable Object, identified by a logical +identity and an **incarnation** that changes on every start. A **link** connects two hosts. The **server** is the one +addressable root host; a **worker** is a host the server starts for one session; a **client** is a tab or TUI, not +addressable. A **plugin** is a directory with a `package.json`; its code for one kind of host is a **facet**. **Base +code** is the app's own code (Chord calls its facet the core facet); it restarts the hosts when it changes, while +plugins reload. An **object type** (or **contract**) describes objects; an **object** is one instance; its **input** is +its key among objects of that type, and its **placement** computes from the input on which host it lives. A **factory** builds an +object; **`self`** is what a factory declares through. A **principal** is who calls, stamped by the link; a +**variant** is the set of members a principal gets. A **reference** is what `get` returns. A **readable** is a value +plus its ordered changes. A **document** is a named JSON value an object owns, written in a **transaction**. A +**derive** is a value computed from readables. A **native** is an OS-level value (a socket, a watcher, a process) +opened with `self.open`; an object with natives is a **native object**, one without is a **pure object**. A +**routine** is standing behaviour, a **job** is owned work that outlives its call, a **flow** is input into running +work. A **registry** is an owner's extension point, filled with **contributions**. **`ctx`** is what every call +receives: cancellation, caller, call-scoped references. A **reload** swaps a plugin's code inside running hosts; a +**restart** ends a process; a **revision** names the complete code set the hosts run. + +:::note | Reading the code in this guide +Imports are shown once per file and then omitted. `obj`, `str`, `arr` and friends are Chord's schema helpers (they +return ordinary TypeBox schemas). Helper functions that belong to the app (`authenticate`, `watchDir`, `uuidv7`) are +named and not shown. Where the design gives no name, the guide says it uses a working name. +::: + +# The Shape of an App {#shape lede="Which processes a Chord app has, how they find each other, and the start code that connects them."} + +Before any plugin exists, Studio needs processes and connections between them. In Chord each participant is a host, +and the app's own start code creates every host, connects them and decides how workers start. Chord deliberately does +none of that itself: a laptop app forks child processes, a test runs everything in one process, a Cloudflare +deployment routes through the edge. This chapter writes Studio's start code: `server.ts`, `worker.ts`, the browser +entry, and a test setup. + +## Three roles + +A Node app on Chord has one shape: + +:::flow wide | The three roles. Clients connect to the one server; the server starts workers and relays between them. +row "Clients: not addressable" muted +box tab1 "browser tab" "owner, local" +box tab2 "browser tab" "guest Ann" +box tui "TUI" "owner" +row "Server: exactly one" accent +gap +box server "server" "`Server()`" accent +gap +row "Workers: one per session, on demand" blue +box w1 "worker" "`Session({ id: \"S1\" })`" blue +box w2 "worker" "`Session({ id: \"S2\" })`" blue +box w3 "worker" "`Session({ id: \"S3\" })`" dashed +arrow tab1 -> server "WebSocket" +arrow tab2 -> server "WebSocket" +arrow tui -> server "WebSocket" +arrow server -> w1 "IPC" +arrow server -> w2 "IPC" +note "A worker for S3 does not exist until something asks for an object that lives there." +::: + +| Role | Addressable | Who it is comes from | Lifecycle | +| --- | --- | --- | --- | +| client | no | the authentication of its link | connects by itself | +| server | yes, exactly one | its own declaration | runs | +| worker | yes, many, by a typed address | stamped by the server, its parent | started on demand by the app | + +The **server** holds what all sessions share (the session list, access, settings) and starts workers. A **worker** +runs one session: its agent, its files, its terminal connections. **Clients** are browser tabs and TUIs. Nobody can +send a request *to* a client, so a client can only serve objects that nobody outside it needs; chapter IV names them +`local` types. + +Things that look like hosts but are not: a code mode sandbox (a worker thread inside a worker, owned by an object), a +remote environment's daemon (it speaks its own protocol, owned by an object), the TUI's sandbox behind its launcher. +A detached process such as a terminal host could become another kind of child of the server. + +## Host kinds and addresses + +Every addressable host has a typed address. The app declares its **host kinds** once, in base code: + +```ts +// studio/src/hosts.ts +import { hostKind, obj, str } from "@earendil-works/chord/next"; + +export const Server = hostKind("server"); +export const Session = hostKind("session", obj({ id: str() })); + +Server(); // the address of the server +Session({ id: "S1" }); // the address of the worker of session S1 +``` + +There are no host-name strings anywhere in app code. Chord serializes addresses internally; your code only calls the +kinds. An object type names its host with a placement function from its input to an address (chapter II), so the +type checker knows that `Session({ id: session })` needs a `session` field. + +## Links + +A **link** is an ordered, two-way channel of messages between two hosts; it only delivers. Chord ships three: a memory +pair for tests, IPC for a Node child process forked with `serialization: "advanced"`, and WebSocket. All three carry +the same binary message format and the same flow control (chapter XVII). Recovery policy belongs to the link adapter: +IPC needs no heartbeat, a WebSocket through a relay needs one, because TCP does not notice that a laptop went to sleep. + +Each link has a **direction**. A host has at most one link to its parent (`connect`) and any number of links from +below (`accept` for clients, `route` for workers). Requests arriving on a link act as the principal the link was given; +chapter III explains why that matters. + +## The server + +```ts +// studio/src/server.ts +import { createHost, ipcLink, webSocketLink } from "@earendil-works/chord/next"; +import { fork } from "node:child_process"; +import { WebSocketServer } from "ws"; +import { Server, Session } from "./hosts.ts"; +import { coreServerFacet } from "./core/server.ts"; +import type { P } from "./principal.ts"; + +const server = createHost

({ + address: Server(), + principal: { kind: "server" }, // who this host's own code acts as + storage: await openServerStorage(dataDir), // persisted documents (chapter V); memory if omitted + onError: (e) => { if (e.kind === "storage_failed") process.exit(75); else log(e); }, +}); + +await server.load([ + { name: "@studio/core", facet: coreServerFacet(config) }, // base code, under a fixed name + ...(await loadPlugins("server")), // [{ name, facet }] from the plugin loader +]); + +const wss = new WebSocketServer({ server: http, path: "/chord", maxPayload: 64 * 1024 * 1024 }); +wss.on("connection", (ws, req) => { + const principal = authenticate(req); // app code: cookie, loopback, device token + if (!principal) return ws.close(4401); + server.accept(webSocketLink(ws), { principal }); +}); + +server.route(Session, { + authorize: (p, { id }) => + p.kind === "owner" || p.kind === "server" || (p.kind === "guest" && p.session === id), + resolve: async ({ id }, { signal }) => { + const principal: P = { kind: "worker", session: id }; + const child = fork("worker.ts", [], { serialization: "advanced", signal }); + child.send({ id, principal, init: await initFor(id, signal) }); // the app's own start message + return { link: ipcLink(child), principal }; // Chord stamps `principal` on it + }, +}); +``` + +Each call has one job: + +| Call | Meaning | +| --- | --- | +| `createHost

({ address?, principal?, ...options })` | with an address: an addressable host whose own code acts as `principal`; without: a client. `P` is the app's principal type | +| `load([{ name, facet }])` | the facets this host runs, each with its plugin's `package.json` name; duplicates and invalid names are refused. Base code passes its core facet the same way | +| `accept(link, { principal })` | a link from a client; every request on it acts as `principal` | +| `route(Kind, { authorize, resolve })` | how to reach hosts of a kind. `authorize(principal, address)` runs before anything starts; `resolve(address, { signal })` returns `{ link, principal }` | +| `connect(link, { principal })` | the link to the parent, at most one, replaced after a reconnect; `principal` says who the parent is | +| `root(fn, { principal?, signal? })` | a call scope for code outside objects (a CLI command): gives `fn` a `ctx` (chapter VI) | + +Chord has no `listen`. Entry points differ too much: a loopback WebSocket server, a tunnel, the Radius relay +delivering channels, a Durable Object edge, a memory link in a test. Each entry point authenticates and calls +`accept`. Radius, for example, is one more line: + +```ts +radius.onChannel((ws) => + server.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } })); +``` + +## The worker + +```ts +// studio/src/worker.ts +const { id, principal, init } = await startMessage(process); // app code: the first IPC message +const worker = createHost

({ + address: Session({ id }), + principal, // { kind: "worker", session: id } + storage: await openSessionStorage(init.dataDir, id), // one storage per session +}); +await worker.load([ + { name: "@studio/core", facet: coreWorkerFacet(init) }, // serves the start data (chapter IX) + ...(await loadPlugins("worker")), +]); +worker.connect(ipcLink(process), { principal: { kind: "server" } }); +``` + +## A browser tab + +```ts +// studio/web/main.tsx +const client = createHost

(); // no address: a client +await client.load(await loadPlugins("web")); +client.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }); +const notes = await client.get(Notes, { session: "S1" }); // tab → server → worker S1 +``` + +## The first `get`, step by step + +The tab asks for the notes of S1. No worker for S1 runs yet. + +:::sequence animate | The first get for an object on a worker that is not running. The server authorizes, the app starts the worker, Chord stamps the link. +actor tab "browser tab" "owner" +actor server "server" accent +actor app "route(Session)" "app code" muted +actor worker "worker S1" blue +tab -> tab: placement: Session({ id: "S1" }) +tab -> server: hold Notes { session: "S1" } +server -> server: not mine: route(Session) +server -> app: authorize(owner, { id: "S1" }) +app --> server: true +server => app: resolve({ id: "S1" }) +app -> worker: fork + start message +worker -> worker: createHost, load, connect +app --> server: { link, principal: worker S1 } +server => worker: hold (stamped: owner) +worker -> worker: placement check, run factory +worker --> server: bound: snapshot +server --> tab: bound: snapshot +::: + +`resolve` runs once per address while its link lives. When the link drops (the worker crashed or the supervisor +stopped it) and references to that worker still exist, Chord calls `resolve` again by itself, with a per-address +backoff, so a tab that only watches recovers without a click. A worker start that already produced a link finishes +even if the request that triggered it went away. + +**Incarnations** make restarts visible. Chord mints a fresh id (a uuidv7) on every host start and sends it in the link +handshake; the other side sees a restart as a new incarnation of the same identity, and references to it re-hold +(chapter VI). + +## Tests: everything in one process + +The same code runs in a test with memory links. `resolve` builds the worker host in memory and returns one end of the +pair: + +```ts +server.route(Session, { + authorize: () => true, + resolve: async ({ id }) => { + const [up, down] = memoryLinkPair(); + const principal: P = { kind: "worker", session: id }; + const worker = createHost

({ address: Session({ id }), principal }); + await worker.load([{ name: "@studio/core", facet: coreWorkerFacet({ id, dataDir }) }, notesWorker]); + worker.connect(up, { principal: { kind: "server" } }); + return { link: down, principal }; + }, +}); +``` + +Memory links deliver the same encoded messages asynchronously, so a test runs the production protocol path. + +## What stays app code + +Starting processes, the start message, authentication, supervising and stopping idle workers are app code. Chord +provides one readable for the last one: `host.activity`, with `{ references, calls, jobs, lingering }` (viewers, calls +in flight, unsettled jobs, objects kept warm after their last reference). Studio's core reports it to the server +together with the agent's running tasks, and the server's supervisor stops a worker when everything has been zero for +five minutes, as p2 does. Chord counts; it has no idle policy. + +## Host options and timeouts + +`createHost` takes the options introduced later in this guide: `storage` (V), `principalKey` and `revokedBy` (III, +VII), `telemetry` (XVII) and `onError`. `onError` receives background errors and `{ kind: "storage_failed" }`; every +app must handle the latter, because Chord never ends the process. Studio exits with its own code so its supervisor +restarts it. + +Timeouts are host defaults with per-thing overrides; the full table is in the appendix. The two you meet first: +`readyTimeout` (60 s: how long an activation and an awaited `get` wait) and the park deadline (60 s: how long a call +waits through a link drop or a restart before failing `unavailable`). + +:::note | Built so far +Nothing in this chapter is built in Chord v2 yet: hosts, links and routes are layers L6 and L7. The frozen kernel has +working versions with the same shape (`createHost`, `connect`, `accept`, `route`, `load`), which the rebuild +simplifies. +::: + +# A First Object {#first-object lede="A plugin, a contract, a factory, and a browser tab that calls it on another process."} + +Studio's first feature is session notes: a list of short notes per session, shown in a panel, that anyone allowed can +add to. Several places need "the notes of S1": the owner's tab, a guest's tab, later a download route and an agent +tool. If each built its own copy, the copies would drift apart and nobody would know when to clean them up. In Chord +there is **one object** for "the notes of S1", everyone asks for it by its type and input, and Chord routes them all to it. + +## The plugin's files + +A plugin is a directory with a `package.json`. Its `name` (npm style, globally unique and stable) is the plugin's +identity: Chord finds its stored documents, reports its errors and orders its contributions by it, never by a +directory name or load order. + +| File | Runs on | Holds | +| --- | --- | --- | +| `package.json` | read by the loader | `"name": "@studio/notes"` | +| `contract.ts` | every host that imports it | object types: names, shapes, method signatures; no running code | +| `worker.ts` | every session worker | the worker facet | +| `server.ts` | the server | the server facet (later: the download route) | +| `web.tsx` | browser tabs | the web facet: the panel | + +## The contract + +An object type is a contract: a name, an **input** (the object's key among objects of this type), a **placement** +(on which host it lives, computed from the input), the **members** (state you can read and methods you can call) and a **version**. + +```ts +// notes/contract.ts +import { arr, int, method, obj, remote, str, type Static } from "@earendil-works/chord/next"; +import { Session } from "@studio/core/hosts"; + +export const Note = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int() }); +export type Note = Static; + +export const Notes = remote("notes", { + version: 1, + input: obj({ session: str() }), // the key: { session: "S1" } + placement: ({ session }) => Session({ id: session }), // input → the host's address + variants: { + editor: { + state: { notes: obj({ items: arr(Note) }) }, + methods: { + add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), // returns the new id + remove: method(obj({ id: str() })), // no result + }, + }, + }, +}); +``` + +`remote` means objects of this type can be reached from any host, across links (chapter IV introduces `local`). The +schema helpers (`obj`, `str`, `int`, `arr`, `map`, `opt`, `lit`, `oneOf`, `union`, `bool`, `num`, `nul`, `bytes`, +`unknown`) return ordinary TypeBox schemas; short options such as `{ min, max }` bound lengths or values. Schemas must +describe data: no callbacks, no class instances, no custom codecs. + +`method()` has four forms: `method()` (no arguments, no result), `method(args)`, `method(args, result)` and +`method(undefined, result)`. + +A contract contains no code that runs, so every host can import it. When you declare it, Chord materializes its +description and computes a **fingerprint**, a hash of its input, variants and members. Two hosts compare contracts by +version and fingerprint (chapter XVI). Placement is not part of the fingerprint. + +## Placement + +The input says *which* object; `placement` says *where* it lives. The caller runs the placement function on the input and sends the resulting address with its request; the server +routes by it. The provider runs the function too and refuses an object that is not its own with `invalid/placement`. +So a worker for S2 never builds "Notes for S1", even if a caller has an outdated rule. Because placement is a typed +function, a `Session` placement on a type whose input has no `session` field does not compile. + +Inputs are **normalized** before anything else: defaults are applied, unknown fields removed, and a canonical key +computed. `{ session: "S1" }` and `{ session: "S1", extra: 1 }` are the same object. + +## The facet and the factory + +A **facet** is a plugin's setup for one kind of host. It runs once when the plugin loads on that host and tells the +host which object types it serves: + +```ts +// notes/worker.ts +import { defineState } from "@earendil-works/chord/next"; +import { defineFacet } from "@studio/core"; // Studio's defineFacet: principals typed as Studio's P (chapter III) +import { Note, Notes } from "./contract.ts"; + +const NoteList = defineState("notes.list", { // a document (chapter V); in memory for now + schema: obj({ items: arr(Note) }), + version: 1, + initial: () => ({ items: [] }), + checkpointWhen: () => true, +}); + +export default defineFacet((host, self) => { + host.serve(Notes, ({ session }, principal, self) => { + const list = self.state(NoteList); // THE note list of this object + return { + variant: "editor", + notes: list, // a document handle is a readable: expose it as state + add: ({ text }, ctx) => + self.tx(async (tx) => { + const id = uuidv7(); + (await tx.draft(list)).items.push({ id, text, by: "someone", at: Date.now() }); + return id; + }, ctx), + remove: ({ id }, ctx) => + list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx), + }; + }); +}); +``` + +Read `host.serve(Notes, factory)` as "this host builds `Notes` objects like this". The factory receives the +normalized input, the caller's principal (chapter III) and `self`, through which it declares what the object needs. +It returns the members of one variant, tagged with `variant`. + +**Factories declare; Chord activates.** The factory is synchronous and has no side effects. It does not await, open +files or start timers. `self.state(NoteList)` does not load anything: it declares that the object has this document +and returns a **handle** that stays valid for the object's whole life. Chord then performs the activation: it loads the +document, binds references, opens OS handles, starts routines, and only then lets calls in. The rule exists for +reload: a reload runs the factory again, and anything the factory did itself would happen twice. + +## Calling it + +```tsx +// notes/web.tsx +export function NotesPanel({ session }: { session: string }) { + const notes = useRef(Notes, { session }); // app-side hook over host.get / release + const items = useReadable(notes?.notes)?.items ?? []; + return ( +

+ {items.map((n) =>

{n.text}

)} + notes?.add({ text })} /> +
+ ); +} +``` + +Underneath the hooks is the reference API: + +```ts +const notes = await client.get(Notes, { session: "S1" }); // live and hydrated once awaited +notes.notes.value.items; // the current list +notes.notes.subscribe((v) => render(v.items)); // every committed change +const id = await notes.add({ text: "check the flaky test" }); +notes.release(); // the object may end now +``` + +An awaited reference is **live and hydrated**: every state member has a value. State members are **readables**: +`.value` is the current value, `subscribe` is called after every commit. The first `get` for `{ session: "S1" }` runs +the factory; every later `get` for the same input and caller shares that object. The object ends when the last +reference is released (a type may keep it a little longer with `linger`). + +:::sequence animate | A call to add. Arguments are checked once at the provider; the commit reaches the viewer as one small change. +actor tab "browser tab" +actor server "server" muted +actor worker "worker S1" blue +actor obj "Notes { S1 }" accent +tab -> server: call add { text } +server -> worker: relay (header only) +worker -> obj: strip + validate args +obj => obj: self.tx: push note, commit +obj --> worker: result: id +worker --> server: state: ops of the commit +worker --> server: result +server --> tab: state, then result +::: + +## Two rules for members + +**Methods return data, never objects.** A method that wants to point you to another object returns its address +(`{ session: "S2" }`), and you `get` it yourself. That keeps every reference owned by someone who releases it. + +**No callbacks as arguments.** A callback carries code, and everything it closes over, from one object into another. +Suppose an inbox object passed `onFile` to another object, which stored it. After a reload of the inbox's plugin, the +other object still calls the *old* `onFile`; after the inbox ends, it calls into an object whose file watcher is +already closed. Chord forbids this by construction: members are data (JSON plus bytes). Behaviour from elsewhere goes +through registries (chapter XIII), multi-step work through jobs (XI), input from a person through flows (XII). + +## What is wrong with this version + +This first version works for one person. It has two problems, which the next two chapters fix. Everyone gets the +`editor` variant, including a read-only guest. And, less obvious, the owner and a guest do not see the same notes: as +chapter III shows, a remote object exists per caller, so each caller's `Notes` object has its own `NoteList`. + +:::note | Built so far +`remote`, `method`, the schema helpers, `Static` and fingerprints are built (`src/next/contracts/`). Serving, `get`, +references and documents are layers L3 to L5. +::: + +# Who Is Calling {#principals lede="Principals: how Chord knows who calls, why a caller cannot lie about it, and how the answer picks the object and its members."} + +The owner may add and remove notes. Ann, a read-only guest, may only read them and should not even see who wrote +what. The object must therefore know who is calling, and the caller must not be able to claim to be someone else. In +Chord the answer is a **principal**, a small JSON value the app defines and the links stamp. + +## The app's principal type + +Chord defines no principal kinds; the app does. Every variant has a `kind`. Studio uses p2's set: + +```ts +// studio/src/principal.ts +export type P = + | { kind: "owner"; access: "local" | "relay" | "device"; device?: string } + | { kind: "guest"; session: string; mode: "read" | "write"; grant: string; revision: number } + | { kind: "server" } + | { kind: "worker"; session: string }; +``` + +| `kind` | Who | +| --- | --- | +| `owner` | the person running the app, on loopback, through the relay, or on a paired device | +| `guest` | someone invited into one session, with a mode and the grant (and its revision) that let them in | +| `server` | the server's own code calling a worker | +| `worker` | a session worker calling the server | + +## Links stamp principals + +A principal never comes from a request's data. The server authenticates a client once, when its WebSocket connects, +and stamps the result on the whole link (`server.accept(link, { principal })`). The server tells Chord who each worker +is when it resolves it (`return { link, principal }`). When the server relays a client's request to a worker, Chord +writes the client's principal into the forwarded request; a principal field coming from a client is always +overwritten, and only a trusted parent link may carry one. + +Where principals come from is app code. Studio's sharing feature creates a share link per session and mode; a guest +who opens it gets a guest record (grant `g1`) and a cookie; `authenticate(req)` turns the cookie into +`{ kind: "guest", session: "S1", mode: "read", grant: "g1", revision: 1 }` on every new connection. Chord never sees +cookies or grants, only the finished principal. + +## The caller is part of the identity + +A `remote` object exists once per **(type, input, principal key)**. The **principal key** is a string computed from +the principal; by default its canonical JSON (sorted keys). The app may pass its own `principalKey` to `createHost`, +and must pass the same function on every host, or identities split silently. + +```ts +const principalKey = (p: P) => + p.kind === "owner" ? `owner:${p.access}:${p.device ?? ""}` + : p.kind === "guest" ? `guest:${p.grant}:r${p.revision}` + : p.kind === "worker" ? `worker:${p.session}` + : p.kind; +``` + +The design wants the key to include how the client is connected and the grant's revision, so that two access paths +never share objects and a changed grant never reuses the old grant's object: + +| Caller | Principal key | `Notes { S1 }` object | +| --- | --- | --- | +| owner, tab A, local | `owner:local:` | 1 | +| owner, tab B, local | `owner:local:` | 1 (shared with tab A) | +| owner through the relay | `owner:relay:` | 2 | +| guest g1, revision 1 | `guest:g1:r1` | 3 | +| guest g1 after a change to write mode, revision 2 | `guest:g1:r2` | 4 (new) | + +## Variants + +The contract lists named sets of members; the factory receives the principal and returns the variant that principal +may use: + +```ts +// notes/contract.ts (version 1, final form for now) +variants: { + editor: { + state: { notes: obj({ items: arr(Note) }) }, + methods: { add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), remove: method(obj({ id: str() })) }, + }, + reader: { + state: { notes: obj({ items: arr(PublicNote) }) }, // no `by`: authors are hidden + }, +}, +``` + +```ts +// notes/worker.ts +host.serve(Notes, ({ session }, principal, self) => { + if (canWrite(principal, session)) return { variant: "editor", notes: …, add: …, remove: … }; + return { variant: "reader", notes: … }; +}); +``` + +A reader has no `add` at all. Chord refuses a call to a member that is not in the caller's variant, so `add` needs no +check of its own. Three rules keep variants compatible: a member name has the same kind (state or method) in every +variant that has it; a method has the same arguments and result in every variant; only state schemas may differ per +variant (the reader's notes have no authors). `remote(...)` enforces them at the type level, and at runtime for +contracts that arrive as data. + +## How a facet knows `P` + +`P` is Studio's type, not Chord's, and every host is created with it (`createHost

`). A plugin's facets are written +and compiled separately, then handed to `host.load` as values, so the type checker cannot infer `P` from the host into +the factory's `principal`. Studio's base code therefore exports its own `defineFacet`, already bound to `P`, and every +facet of every plugin imports that one instead of Chord's: + +```ts +// studio/src/facet.ts (base code, exported as @studio/core) +import { bindFacet } from "@earendil-works/chord/next"; +import type { P } from "./principal.ts"; + +export const defineFacet = bindFacet

(); // a facet's principals are Studio's P +export type { P, Revocation } from "./principal.ts"; +``` + +```ts +// notes/worker.ts (server.ts, web.tsx and tui.ts import the same defineFacet) +import { defineFacet } from "@studio/core"; + +export default defineFacet((host, self) => { + host.serve(Notes, ({ session }, principal, self) => { // principal: P + if (principal.kind === "guest" && principal.mode === "read") return { variant: "reader", notes: … }; + … + }); +}); +``` + +The import is cheap: `@studio/core` is a host package plugins already import at runtime (the contract imports `Session` +from it), `P` is a type-only import, and `bindFacet` is a one-line function. The host checks the pairing: `load` +accepts only facets typed for its own `P`, so a facet compiled against another app's principal type, or one built with +Chord's untyped `defineFacet`, does not compile into Studio's start code. Client facets (`web.tsx`, `tui.ts`) use the +same `defineFacet`; they rarely see a principal, because clients serve only `local` types. + +None of this is checked at runtime, and none of it needs to be: the principal a factory receives is the one the link +stamped. + +:::review | Review: binding facets to the app's principal type (open) +The design shows `defineFacet((host, self) => …)` without saying how `P` reaches the factory. The frozen kernel takes +it as an explicit type parameter, `defineFacet

(…)`, defaulting to `{ kind: string }` +(`packages/chord/src/next/facet.ts`), and the playground writes `defineFacet(…)`. The pre-bound +`defineFacet` above (`bindFacet` is a working name) is the owner's preferred direction, recorded as an open point in +`packages/chord/docs/next/layer-1.md` §4. +::: + +## Narrowing on the caller's side + +Owner and guest run the same `web.tsx`, and which variant they get is known only at runtime. The types force every +caller to narrow before using variant members: + +```ts +const notes = await client.get(Notes, { session: "S1" }); +notes.add({ text: "hi" }); // type error: `add` exists only in "editor" +if (inVariant(notes, "editor")) await notes.add({ text: "hi" }); +expectVariant(notes, "editor"); // or assert (throws) +function Editor({ notes }: { notes: RefIn }) { … } +``` + +The protection is not the type checker; it is Chord refusing the call at the provider. A client that ignores the +types gets the same error. + +## Admission before a worker starts + +Principals also decide whether a caller may reach a host at all. `route(Session, { authorize })` runs before Chord +resolves an address, so a guest of S1 asking for anything in S2 is refused at the server, and no worker starts for +them. Finer authority lives in the per-principal view's methods; there is no asynchronous "may this caller activate +this object" guard. An object whose address carries authority (say, a login flow by id) may exist; what a caller can +see and do is decided by the variant its factory picks. + +## Inside methods + +`ctx.principal` is the caller of the current call. In Studio's notes the editor's `add` uses it to record the +author: + +```ts +add: ({ text }, ctx) => store.add({ text, by: authorOf(ctx.principal) }, ctx), +``` + +Where principals act, in summary: + +| Purpose | Where | +| --- | --- | +| identity: which object a caller gets | principal key | +| variant: which members | the factory | +| admission: whether a host is reachable | `route.authorize` | +| attribution and fine checks | `ctx.principal` in methods | +| revocation | `revokedBy` (chapter VII) | + +## The bug from chapter II + +Now the bug is visible. The owner's `Notes { S1 }` and Ann's `Notes { S1 }` are two objects (rows 1 and 3 above), and +each declared its own `NoteList`. The owner's notes are not in Ann's list. Per-caller objects must not hold the data; +they must be views over something that exists once. That is the next chapter. + +# One Store, Many Views {#shared-store lede="Local types, and the pattern every Chord plugin uses: shared data once in a store, per-caller views in front of it."} + +Every caller has its own `Notes` object, which is what makes per-caller variants possible. But data must exist once. +The fix is the most common pattern in Chord: a **shared store**, a `local` object that holds the data and does the +heavy work once, behind thin per-principal **views**, the `remote` objects callers get. + +## `local` types + +```ts +// notes/contract.ts +export const NoteStore = local("notes.store", { + version: 1, + input: obj({ session: str() }), + members: { + notes: obj({ items: arr(Note) }), // state: a schema + public: obj({ items: arr(PublicNote) }), + add: method(obj({ text: str(), by: str() }), str()), // methods: method(...) + remove: method(obj({ id: str() })), + }, +}); +``` + +A `local` contract has flat `members` and no variants. The differences: + +| | `remote` | `local` | +| --- | --- | --- | +| reachable from | any host, across links | its own host only | +| identity | (type, input, principal key) | (type, input) | +| principals, variants | yes | no | +| factory | `(input, principal, self)` | `(input, self)` | +| arguments | stripped to the provider's schema and validated, for every caller | see the review note below | +| values on the same host | passed by reference, never copied | passed by reference, never copied | +| `ctx.principal` | the caller | none | +| may persist documents | no (chapter V) | yes | + +A local object has no principal: it is trusted code on its own host, acting for the host. So attribution ("who wrote +this note") is passed as data (`by`), by the view that knows who called. Clients serve only local types, because +nobody can address a client. + +## The pattern + +```ts +// notes/worker.ts +export default defineFacet((host, self) => { + host.serve(NoteStore, ({ session }, self) => { + const list = self.state(NoteList); + const pub = self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) })); + return { + notes: list, + public: pub, // computed once, for every reader + add: ({ text, by }, ctx) => + self.tx(async (tx) => { + const id = uuidv7(); + (await tx.draft(list)).items.push({ id, text, by, at: Date.now() }); + return id; + }, ctx), + remove: ({ id }, ctx) => list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx), + }; + }); + + host.serve(Notes, ({ session }, principal, self) => { + const store = self.get(NoteStore, { session }); // a reference, declared: held for the view's life + if (canWrite(principal, session)) return { + variant: "editor", + notes: store.notes, // the store's readable, exposed as-is + add: ({ text }, ctx) => store.add({ text, by: authorOf(principal) }, ctx), + remove: (args, ctx) => store.remove(args, ctx), + }; + return { variant: "reader", notes: store.public }; + }); +}); +``` + +:::flow wide | One store, three views. The redaction runs once in the store; both readers read the same value. +row "Per-principal views (remote)" blue +box ve "Notes { S1 }" "owner · editor" blue +box vr1 "Notes { S1 }" "guest Ann · reader" blue +box vr2 "Notes { S1 }" "guest Bob · reader" blue +row "Shared store (local)" accent +gap +box store "NoteStore { S1 }" "notes, public (a derive)" accent +gap +arrow ve -> store "self.get" +arrow vr1 -> store "self.get" +arrow vr2 -> store "self.get" +note "Authority is the view's: the reader variant has no add. The store does not check; it is trusted and acts for the host." +::: + +`self.get(NoteStore, { session })` in a factory declares a **reference** that the view holds for its whole life; Chord +binds it during activation and waits until the store is live (chapter VI). The view exposes the store's readables +directly. Owner, Ann and Bob now give three `Notes` objects, one `NoteStore`, one note list and one redacted list. + +Any heavy or shared part belongs in a store like this: a pi-durable watch, a file watcher, a sandbox, a search index. +Base code uses the same pattern with the same primitives; nothing about it is special to core. + +## Derived values, briefly + +`self.derive(name, sources, fn)` computes a value from readables and recomputes it when a source changes; viewers get +only the difference. The reader's list above is one. Chapter VIII covers derives in depth, including the one footgun +that matters: `stripAuthor` must return the *same* object for an unchanged note, or every commit costs the size of the +whole list. The tool for that is `mapShared`. + +## Arguments are checked at the provider + +Trusted code is not type-checked code: plugins are bundled with esbuild without type checks, and agents write +JavaScript plugins. So the provider checks arguments itself (decision R7): every method's arguments are stripped to +the **provider's own** schema (unknown fields removed) and validated with it, whoever the caller is, and never with a +schema a peer sent. Schemas are compiled lazily on first use and cached per schema object. Bytes are checked as leaves +(`instanceof Uint8Array` plus an optional length bound), never walked. Stripping copies only when a field was removed; +the caller's value is never mutated (proposal Q12, in the L1 draft). Results and state are not validated: they come +from the provider. + +On the same host nothing is copied: arguments and results are passed by reference. Values are **immutable by +convention** (Delta's rule, as in Durable): nobody mutates a value it received, and Chord neither copies nor freezes +values to enforce it. There is no development mode that would freeze them; one behaviour everywhere (R9). + +:::review | Review: are local arguments validated? (Q15, open) +Design §5's table says `local` arguments are "neither validated nor copied"; §3 and R7 say arguments are validated +"for every caller". Review round 3 raised this as Q15 (apply R7 to `local` too, or keep `local` as the one exception), +and `layer-1.md` lists it as open. Until it is decided, write local methods so that they do not depend on either +answer: trust the shape, but never trust content (lengths, ids) that came from a remote caller without the view +checking it. `chord-next-walkthrough.md` concept 6 still describes the older rule ("validated only when the caller is +not a host principal"), which R7 replaced. +::: + +## Choosing between the two + +Use a `remote` type for what callers address: views, things with per-caller authority. Use a `local` type for what +exists once per input on one host: stores, caches, owners of OS handles, and the core objects a host provides +(chapter IX). If a type is reached across a link, it is remote; if a type persists documents, it is local (or a +host-lifetime object, chapter V). + +# Writing It Down {#documents lede="Documents and transactions: how an object stores data, changes it atomically, survives restarts and migrates it."} + +Notes must be stored, changed safely while others read them, sent to viewers as small changes, and survive a worker +restart. An object's data lives in **documents**: named JSON values the object owns, changed only in +**transactions**. The design is a port of pi-durable's store, so most rules will look familiar; this chapter states +them for Chord. + +## Defining a document + +```ts +// notes/worker.ts +const NoteList = defineState("notes.list", { + schema: obj({ items: arr(Note) }), + version: 1, + initial: () => ({ items: [] }), + persist: true, // stored; default: memory only + checkpointWhen: (_value, _ops, info) => info.deltasSinceBase > 64, +}); + +const Draft = defineState("notes.draft", { // one unfinished note per author + schema: obj({ text: str() }), + version: 1, + initial: () => ({ text: "" }), + keyed: true, // many per object, by key + persist: true, + checkpointWhen: () => true, // small: a full base per write +}); +``` + +`checkpointWhen` is required on every definition, with no default (R14, as in Durable since pi `5c1736f82`): it says +when a commit stores a complete base instead of a delta, which bounds how much is replayed on load. Small documents +checkpoint on every write; growing ones after a number of deltas. + +A document's root is always a JSON object (lists live in a field). Documents hold JSON only; bytes are a value and wire +type, not a document type. + +## Singleton and keyed + +```ts +host.serve(NoteStore, ({ session }, self) => { + const list = self.state(NoteList); // THE note list of this object + const drafts = self.state(Draft); // all drafts of this object + … +}); +``` + +| | singleton (no `keyed`) | keyed | +| --- | --- | --- | +| how many | one per object | many per object, by a short key | +| loaded | at activation, kept in memory | on demand | +| read | `list.value` (synchronous) | `await drafts.get(key)`: the value or `undefined` | +| list | (it is one value) | `for await (const key of drafts.keys())` pages keys | +| expose as state | yes: the handle is a readable | no: expose a method or a singleton index | + +There is no live key list. A collection shown live is a singleton (an object holding a list or a map); a large +collection is a method returning pages. + +## Transactions + +All writes go through `self.tx(fn, ctx)`. Inside, `await tx.draft(handle, key?, { tags? })` returns a draft of the +current document, or a draft over `initial()` if it is absent. Drafts record operations (assignments, `push`, +`Object.assign`); asking for the same document twice in one transaction returns the same draft. `tx.retire(handle, +key?)` removes a document. There is no `tx.get`: reading is drafting, and a draft is an overlay over the current +value, so nothing is copied. A draft you do not change commits nothing, so reading never stores an absent document. + +```ts +saveDraft: ({ author, text }, ctx) => + self.tx(async (tx) => { (await tx.draft(drafts, author)).text = text; }, ctx), +``` + +Shorthands for one document: `handle.change(fn, ctx)` (keyed: `handle.change(key, fn, ctx)`) (loads the document, then runs the synchronous `fn` on +its draft) and `handle.retire(ctx)` (keyed: `handle.retire(key, ctx)`). + +### Four patterns + +Two decisions (R12 and R13, both as in Durable) shape how transactions are written. **Drafts are revoked when the +transaction settles**, so a transaction returns at most plain values, never something taken from a draft. And +**there is no existence check inside a transaction.** + +```ts +// 1. Return data: write in the transaction, read after the commit. +await self.tx(async (tx) => { (await tx.draft(list)).items.push(note); }, ctx); +return list.value.items.length; // the committed revision, or a newer one + +// 2. Remove if present: retire of an absent document is a no-op. +await drafts.retire(author, ctx); + +// 3. Create with tags if missing: the draft creates it, only if it changes. +await self.tx(async (tx) => { + (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; +}, ctx); + +// 4. Branch on existence: read first, outside the transaction. +if ((await drafts.get(author)) === undefined) { … } +``` + +### Durable's rule + +Inside the callback, only `await tx.*`. No other objects, no network, no models, no processes, no second `self.tx`, no +reads past `tx`. Breaking it is not an error but a hang: the object's line is held while the callback waits. Fetch +first; decide and write inside. + +```ts +// Wrong: a fetch inside the transaction holds the object's line. +await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title: await fetchTitle(url) }); }, ctx); +// Right: fetch, then write. +const title = await fetchTitle(url, ctx.signal); +await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title }); }, ctx); +``` + +## What a commit does + +:::flow wide animate | The commit sequence, Durable's. Nothing is copied, frozen or revalidated; viewers get one batch per commit. +row "Inside self.tx" accent +box line "wait for the line" "per object" +box run "run the callback" "drafts record ops" accent +box prep "prepare" "Delta checks placed values" +box write "storage batch" "deltas or a base" accent +row "After the write" blue +box adopt "adopt" "pointer swap" +box local "notify locally" "once, with the ops" blue +box wire "send" "one batch per link" +box revoke "revoke drafts" "touching one throws" +arrow line -> run +arrow run -> prep +arrow prep -> write +arrow write -> adopt "stored" +arrow adopt -> local +arrow local -> wire +arrow wire -> revoke +note "An error or an abort before the storage write discards everything. Storage commits are serialized per host." +::: + +Lines are per object, never across objects; reads outside a transaction do not take the line. Subscribers on the same +host get the new value and the commit's operations by reference. Remote viewers get the commit's operations, never the +document again; chapter XVII shows how one commit touching several holds still arrives as one message. + +## Tags and deleting by tag + +Tags are typed tokens set when a document is created: + +```ts +export const SessionTag = defineTag("studio.session"); // base code, shared by every plugin +await self.tx(async (tx) => { (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; }, ctx); +// when the session is deleted +await store.retireByTag(SessionTag, "S1"); // every document tagged S1 +``` + +The design names the operation `store.retireByTag(tag, value)` without saying who holds the store; session delete is +one of the app decisions still open (#18, chapter XIX). + +## Who may persist + +`persist: true` is allowed only in `local` objects and `lifetime: "host"` objects: identities without a principal. A +`remote` object's identity contains the principal key, which changes with the access path and the grant revision. Its +stored documents would scatter across access paths and be orphaned after a reconnect. The rule is enforced at the type +level (`self.state` in a remote factory accepts only non-persisted definitions) and at declaration ("persist needs a +stable identity: put the documents into a local shared store"). This is the second reason for the shared store. + +What survives a host teardown (a crash, a restart) is exactly what is persisted. In-memory documents live as long as +their object: they survive reloads and native restarts of a live object, and go when it ends. Everything else +(natives, derives, routines, references) is rebuilt on activation. + +## Facet documents + +A facet has its own identity (its plugin's name) and may hold documents too, for plugin-wide data such as +preferences. Objects the facet serves may read and write their facet's documents, and one transaction may span the +object's own documents and its facet's (lock order: own line, then the facet's). The facet never touches a served +object's documents. + +## Ownership across plugins + +No plugin reads or writes another plugin's documents, Chord or Durable. Data crosses plugins only through state and +methods. Data that core enforces lives in a format core owns and is handed to core through its API when the data is +created (as with Durable's `createConversation({ init })`), never written into core's documents. + +## Versions and migration + +A change of shape or meaning raises `version` and adds a pure `migrate(old, from)`: + +```ts +const NoteList = defineState("notes.list", { + schema: obj({ items: arr(NoteV2) }), // NoteV2 adds `pinned` + version: 2, + migrate: (old, from) => ({ items: old.items.map((n) => ({ ...n, pinned: false })) }), + initial: () => ({ items: [] }), + persist: true, + checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64, +}); +``` + +Documents migrate on access by the new definition: singletons during activation (hydration counts as access), keyed +ones on their next access. The migrated base is persisted with the next commit; right after a bump, a read-only +transaction writes it. + +Chord does not walk values against schemas. Instead, each definition computes a **schema fingerprint** once +(descriptions and titles ignored), and every stored record keeps the fingerprint it was written with. On access Chord +compares them: the same version with a different fingerprint fails the object `broken` ("schema changed without a +version bump"). So every structural change needs a bump, an additive one with a trivial `migrate`. The one exception +to "no value checks" (R8): the result of `migrate` and of `initial()` is validated against the new schema once, when +produced. A failed check, a throwing `migrate`, or old code meeting a newer document refuses the access, keeps the +stored data and fails the owning object. A changed meaning without a schema change cannot be detected; that is on +you. + +## Storage + +Behind the store is a `DocumentStorage`, passed as `createHost({ storage })`; memory by default. The server has one +shared storage, each worker one for its session. + +```ts +interface DocumentStorage { + commit(writes: readonly DocumentWrite[]): Promise; // create/change/retire, atomic + find(address: DocumentAddress): Promise; + load(id: string): Promise; // version, value, deltasSinceBase + scan(query: { owner: string; definition: string }, limit: number, cursor?: string): + Promise<{ keys: string[]; next?: string }>; + byTag(tag: string, value: string, limit: number, cursor?: string): Promise<{ ids: string[]; next?: string }>; + mintId(): Promise; +} +``` + +Chord ships a memory storage and a SQLite storage (over `@earendil-works/chord/sqlite`, the driver layer that moved +from Durable; Node and Durable Object drivers). Storages check only metadata (incarnation, version, sequence, address +occupancy, fingerprint) and write the supplied delta or base; they never rebuild or validate content on a commit. +Chord documents never commit together with Durable transcripts: Chord has its own storage, not an adapter over +Durable's. Your own storage can prove itself with `registerDocumentStorageConformance(runner, name, withStorage)` from +`@earendil-works/chord/testing`. + +Keyed persisted documents that nobody reads, writes or watches for 10 minutes leave memory and reload on the next +access. A storage error ends the store and closes the host; `onError` receives `{ kind: "storage_failed" }` and the +app decides (chapter I). + +## Kinds of state + +| Kind | For | API | +| --- | --- | --- | +| document | data the object owns | `self.state`, `self.tx` | +| derive | computed values | `self.derive` (chapter VIII) | +| source readable | external data: Durable views, files, logs | value plus ordered changes; feeds derives | +| paged method | large or queryable collections | a method returning pages (chat history) | +| plugin database | full-text search, large blobs | a native opened by its owner (chapter IX) | + +App state lives in Chord documents. Only transcript-bound data, which must fork and rewind with a conversation, lives +in Durable. + +:::review | Review: does an absent singleton read `undefined` or `initial()`? (Q11, open) +Design §9 says "whether a document exists is read outside the transaction (`handle.value` / `handle.get` give +`undefined` for an absent one)" and also "a singleton that does not exist yet reads `initial()` through its handle; +retiring resets it to `initial()`". Review round 3 proposes: singletons always read `initial()`, only absent keyed +documents read `undefined`, and singleton existence needs no API. This guide follows the proposal (`list.value` is +never `undefined`); it is not decided. +::: + +:::note | Built so far +Documents, transactions and storages are layer L3, a port of Durable's session store; not started. The frozen kernel +has a working version with a JSONL storage, which the design rejected. +::: + +# Holding On {#references lede="References, the object lifecycle and ctx: what a caller sees while objects start, fail, restart and move, and why nobody writes retry code."} + +A tab shows the notes of S1 for an hour. During that hour the laptop sleeps, the worker is stopped for idleness and +started again, the notes plugin reloads twice and the store's storage hiccups once. The panel should show the notes the +whole time, say "reconnecting" when it cannot reach them, and recover by itself. In Chord this is what a +**reference** does; plugin code contains no retry loops and no request ids. + +## Three ways to hold + +| Where | API | Released | +| --- | --- | --- | +| a factory | `self.get(Type, input)` | when the object ends | +| a call | `await ctx.get(Type, input, { state? })` | when the call ends | +| outside objects | `await host.get(Type, input, { signal, state?, parkTimeout? })` | `ref.release()` | + +Use `self.get` for what an object needs its whole life (the view's store), `ctx.get` for what one call needs, and +`host.get` only in UI code, tests and entry points. `{ state: false }` binds for methods only, without receiving state. +A reference keeps its object alive while held. + +## The object lifecycle + +```text +(get) → activating → live → ending → ended + └→ failed (backoff) → activating … +``` + +**Activating:** Chord binds the declared references and waits until they are live and hydrated, loads singleton +documents, opens natives (one after another, in declaration order), computes derives and starts routines. The object +goes live when all of that succeeded. Calls wait behind its **gate** until then. + +**Ending:** routines stop, the object's calls and jobs are aborted as their rules say, natives close in reverse order, +references are released. An object with `lifetime: "referenced"` (the default) ends when its last reference goes, +after the type's `linger` if it has one. A `lifetime: "host"` object (only for types without a required input) lives +as long as the host. A running, finishing or blocked job counts as a reference (chapter XI). + +## `ref.state` + +Every reference reports what is going on as a tagged value; listeners get each new one: + +```ts +type RefState = + | { kind: "loading" } // first hold, reconnecting, restarting + | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // undefined: no automatic retry + | { kind: "released" }; +``` + +| Situation | Calls | Values | +| --- | --- | --- | +| link drop | park until reconnected | readable, `loading` | +| object loading or restarting | park until live | readable, `loading` | +| rollout: provider ahead or installing | park until installed | readable | +| object `failed` | reject at once with the cause | `failed`, with cause and `retryAt` | + +Values stay readable through all of this: the panel keeps showing the last notes. Parked calls share one deadline (60 +s by default, `parkTimeout` per `get`), then fail `unavailable`. Otherwise a call ends only through its caller's +signal; there is no per-call deadline. A call that was running on an instance that ends fails with an error and is +not resent: Chord cannot know whether it took effect. + +**Errors look like disconnects.** A consumer handles a failed call exactly as a dropped link: show the error, let the +user retry. Work that must not run twice carries its own idempotency key as data (Durable's `requestId`, a job's +`key`). + +:::steps | The worker restarts while a tab watches the notes ++++ Watching +The tab holds `Notes { S1 }`. `ref.state` is `live`; `notes.notes.value` has 40 notes. ++++ The worker goes away +The supervisor stops worker S1 (or it crashes). The server's IPC link to it drops. The server tells the tab that its +hold is gone; the tab's reference goes `loading`. The 40 notes stay readable. ++++ A click meanwhile +Ann's colleague clicks **Add**. The call **parks** on the reference; nothing is sent yet. The deadline is 60 s. ++++ Chord resolves again +Holds on S1 still exist, so Chord calls `route(Session).resolve({ id: "S1" })` again, with the per-address backoff. +The app forks a new worker: a new **incarnation** of `Session({ id: "S1" })`. ++++ Re-hold +The server re-holds every hold on S1 over the new link. The new worker activates `NoteStore { S1 }` (its notes load +from storage) and the views; it sends fresh snapshots. The tab's reference goes `live`. ++++ The parked call runs +The parked `add` is sent now and succeeds. The panel shows 41 notes. Nobody wrote recovery code. +::: + +## Failure and retry + +Any broken part fails the **whole** object: a half-working object could break rules that rely on its documents +changing together. Chord tags the cause by where it failed: + +```ts +type FailureCause = + | { kind: "transient"; error: ChordError } // an open threw, a dependency was unreachable: automatic backoff + | { kind: "broken"; error: ChordError }; // factory, derive, hydration or migrate threw: retried only by a trigger +``` + +**Transient** causes get a backoff per address: 1 s, doubling to 60 s; concurrent gets share one attempt; success +resets it. **Broken** causes would fail the same way again (a code bug, invalid data), so they are retried only by a +**trigger**: a plugin reload, `ref.retry()` (a Retry button), or a readable the factory declared with +`self.retryOn(...readables)`. Only an opener may declare its own failure broken, with `throw broken(error)`, for +example for invalid configuration. A failed reference stays held; on success it goes `loading`, then `live`. + +```ts +host.serve(EnvConnection, ({ env }, self) => { + const config = self.get(EnvConfig, {}); // a facet or registry value, outlives the attempt + self.retryOn(config.envs); // a config change retries a broken open + const ssh = self.open(async ({ signal }) => { + const c = config.envs.value[env]; + if (!c) throw broken(new Error(`unknown environment ${env}`)); + return connect(c, signal); + }, (s) => s.close()); + … +}); +``` + +`retryOn` sources must live independently of the attempt (facet documents, registry values); a member of one of the +object's own references is a type error, since it goes away with the attempt. + +## Waiting for dependencies + +An object whose references are not live yet stays `loading` and goes live the moment they are, bounded by +`readyTimeout` (60 s). The decision for the failure case (R18) is what makes recovery automatic: + +:::sequence animate | Dependency recovery. The view fails transient with the store's cause, keeps its reference, and is retried the moment the store goes live. +actor tab "tab" +actor view "Notes { S1 }" blue +actor store "NoteStore { S1 }" accent +tab -> view: hold +view -> store: self.get (activation) +store -> store: open fails: disk not mounted +store --> view: failed (transient) +view --> tab: failed: transient, cause inside +note view..store: the view keeps its reference; its state is the retry trigger +store -> store: backoff retry: open succeeds +store --> view: live +view -> view: retried at once (backoff cancelled) +view --> tab: live +::: + +A dependency that fails, or does not become live in time, fails the dependent **transient** with the dependency's +cause inside, never broken: the cause is the dependency's. The dependent keeps its references while failed, and Chord +uses their state as a retry trigger. Factory authors write `self.get` and know none of this. After an object is live, +a dependency leaving `live` does not change it: its calls that use the dependency get the dependency's error, or park +while it loads. + +**Cycles are rejected** (Q5). If A's activation gets B and B's gets A, neither would ever lose its last reference. +Every object instance has a uuidv7 instance id; on one host Chord follows the waiting edges of activating objects, and +across hosts a hold made during an activation carries the chain of instance ids. A cycle fails the activation +`broken`, naming it ("A → B → A"), and keeps no references. + +## No optional references + +`get` has no `optional` flag. When a dependency's plugin may be missing, turn the dependency around: the dependent +owns a registry and the other plugin contributes to it. Radius uses forwards from the ports plugin, but ports is +optional, so radius owns `radius.forwards` and ports contributes its forwards; without ports the table is empty and +radius works (chapter XIII). + +## `ctx` + +Every call gets a `ctx` as its last argument: + +| Field | Meaning | +| --- | --- | +| `signal` | fires when the caller cancelled or the object ended; pass it to everything you wait on | +| `principal` | the caller (remote types only) | +| `incarnation` | the calling host's incarnation, for calls that arrived over a link (chapter X) | +| `get` | references released when the call ends | +| `telemetry`, `trace(name, attributes, fn)` | the current span; a child span with a child `ctx` (chapter XVII) | +| `jobId`, `progress` | in a job's `run` (chapter XI) | + +```ts +exportText: async ({}, ctx) => { + const catalog = await ctx.get(Catalog, {}); // released when the call ends + const title = catalog.sessions.value[session]?.title ?? session; + const res = await fetch(renderUrl, { method: "POST", body: render(title, list.value), signal: ctx.signal }); + return await res.text(); +}, +``` + +There are no code versions and no request ids in `ctx`. Durable's `Context` is the same module (`chord/context`): +`ctx` extends it, so passing `ctx` into pi-durable needs no conversion. Code outside objects gets a `ctx` from +`host.root(fn, { principal?, signal? })`; inside Durable tools it uses `chord(fn, context)` (chapter XIV). + +:::note | Built so far +References, `ctx` and the lifecycle are layers L4 and L5. `ChordError` and `FailureCause` are specified in the L1 +draft (`layer-1.md`), with codes `invalid`, `contract_mismatch`, `forbidden`, `gone`, `not_ready`, `released`, +`too_large`, `unavailable` and `aborted`, and a finer `reason` such as `invalid/args`. +::: + +# Taking Access Away {#revocation lede="Revocation: how removing a guest ends their objects on every host at once, and why reconnecting does not bring access back."} + +The owner removes Ann from S1. Her tab still has a WebSocket open and a `Notes { S1 }` view on worker S1. Both must end +now, on the server and on the worker, and her tab's automatic reconnect must not restore anything. Chord provides the +mechanism; the app decides who is blocked and remembers it. + +## Labels, not predicates + +The app declares, once, which **revocation labels** each principal carries. Revoking means revoking one label; +everything carrying it is hit. + +```ts +// studio/src/principal.ts +export type Revocation = { grant: string } | { device: string } | { session: string }; + +export const revokedBy = (p: P): Revocation[] => + p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] + : p.kind === "owner" && p.access === "device" && p.device ? [{ device: p.device }] + : []; + +const server = createHost({ address: Server(), principal: { kind: "server" }, revokedBy }); +// every worker is created with the same revokedBy +``` + +| Principal | Labels | +| --- | --- | +| guest Ann (grant g1) in S1 | `{ grant: "g1" }`, `{ session: "S1" }` | +| guest Bob (grant g2) in S1 | `{ grant: "g2" }`, `{ session: "S1" }` | +| owner on the paired phone | `{ device: "phone" }` | +| owner, local | none | + +| Call | Hits | +| --- | --- | +| `server.revoke({ grant: "g1" })`: remove Ann | Ann | +| `server.revoke({ session: "S1" })`: archive S1 | Ann and Bob | +| `server.revoke({ device: "phone" })`: sign out the phone | the phone | + +A label is plain data because revocation is an event that must travel from the server to the workers; a predicate +function could not be sent. + +## What `revoke` does + +`host.revoke(r)` ends every object and closes every link whose principal `p` has `r` in `revokedBy(p)` (compared by +canonical JSON; never the host's own principal), and passes `r` to its children, which apply their own `revokedBy`. + +:::sequence animate | Removing guest Ann. The label travels down; her objects end on every host; her reconnect is refused by the app's own record. +actor owner "owner tab" +actor server "server" accent +actor worker "worker S1" blue +actor ann "Ann's tab" "guest g1" muted +owner -> server: Sharing.remove({ grant: "g1" }) +server -> server: app: mark g1 blocked (document) +server => server: revoke({ grant: "g1" }) +server -> ann: close link +server => worker: revoke { grant: "g1" } +worker -> worker: end Notes { S1 } as g1 +ann -> server: reconnect +server -> server: authenticate: g1 blocked +server --> ann: refused (4401) +::: + +On worker S1, for `revoke({ grant: "g1" })`: + +| Object | Principal | Hit | +| --- | --- | --- | +| `Notes { S1 }` | owner, local | no | +| `Notes { S1 }` | guest g1 | **yes**: ends | +| `Notes { S1 }` | guest g2 | no | +| `NoteStore { S1 }` | none (local) | no | + +Ending Ann's view releases its reference to the store, which keeps serving the others. + +## Chord persists nothing here + +`revoke` cuts live access only. Lasting denial lives in the app's data (the sharing plugin's guest records), which +`authenticate` checks on every new connection. After a server restart nothing live exists to revoke; every client +authenticates again and blocked grants are refused there. + +**No re-stamping.** A changed grant (Ann moves from read to write) is a revocation of the old grant. Her link closes, +her tab reconnects and authenticates with the new grant revision, which gives a new principal key, and her references +re-hold by themselves, now as an editor. Reconnecting never restores revoked authority. + +**Revocation beats jobs** (R11). Normally a running job keeps its object alive after the last reference is released +(a closed tab does not abort an import). Revoking a principal ends its objects at once regardless; their jobs get the +abort signal and the normal close timeout. + +## Work in shared stores + +Revocation ends objects whose principal carries the label. A shared store has no principal, so work it runs *on +behalf of* someone (a question waiting for Ann's answer, a shell command a guest started) is not ended by `revoke`. +Such a store keeps the grant as data with the work and reconciles from the grant owner's live state with a routine +(chapter X): + +```ts +// in a shared store that runs work for guests +const sharing = self.get(Grants, {}); // the sharing plugin's live grants +self.routines.on(sharing.live, async (live, ctx) => { + for (const run of shell.running.value) + if (run.args.grant && !live[run.args.grant]) shell.abort(run.id); // idempotent +}); +``` + +There is no revocation hook: a hook can be missed (a crash between revoke and hook), a routine over current state +cannot. The window between the revocation and the routine's run is accepted (R10); a step started in it still runs. + +## Who does what + +| Chord | The app | +| --- | --- | +| stamps principals on links | what a grant is; share links, cookies, `authenticate` | +| keeps one object per principal key | defines `Revocation` and `revokedBy` | +| on `revoke(r)`: ends matching objects and links, passes `r` down | decides when to revoke; stores who is blocked | + +# Computed Values {#derives lede="Derives, the one footgun that makes them expensive, mapShared, and presence: who is looking."} + +The reader's notes without authors, a note count, the number of people viewing: values computed from other values. +Storing copies would go stale; recomputing them on every read would be wasteful and could not be watched. A +**derive** is computed once per change and sent to viewers as a difference. + +## `self.derive` + +```ts +const count = self.derive("count", [list], ([l]) => l.items.length); +const summary = self.derive("summary", [list, store.pinned], ([l, pinned], previous) => ({ + total: l.items.length, + pinned: pinned.length, +})); +``` + +`self.derive(name, sources, (values, previous) => value)`. Sources can be documents, readables, other derives and +state members of references. The value is recomputed when a source changes and diffed into operations for the wire. +Derived values are adopted before ordinary observers run and share their source commit's wire batch, so a viewer +never sees a note list and a count from different commits. + +`previous` is the derive's last value, kept across factory re-runs (reloads) by name. It is a cache, empty after a +host teardown, so a derive must be correct without it. A derive that throws fails its whole owner `broken`. + +## How the difference is found + +Chord compares the new value with the previous one using Delta's `diffRevisions`. Parts that are the **same objects** +as last time cost one `===` each. Parts that are new objects must be compared by content. + +**The footgun.** A derive that builds new objects for unchanged data makes every commit cost the size of the whole +value, per view. The reader list from chapter IV, written naively: + +```ts +self.derive("public", [list], ([l]) => ({ items: l.items.map((n) => ({ id: n.id, text: n.text, at: n.at })) })); +// one note added to 2,000: 2,001 new objects, every one compared by content +``` + +**`mapShared(f)`** keeps one result per source element: `f` runs only for new or changed elements, unchanged ones +return their previous result object. + +```ts +import { mapShared } from "@earendil-works/chord/delta"; + +const stripAuthor = mapShared((n: Note): PublicNote => ({ id: n.id, text: n.text, at: n.at })); +self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) })); +// one note added: one new object, 2,000 identical references; the diff emits one insert +``` + +:::review | Review: mapShared is planned, not available +Design §12 introduces `mapShared` as coming "from `@earendil-works/chord/delta`", but nothing in `packages/chord/src` +defines it yet, and Delta does not yet treat bytes as atomic leaves in `diffRevisions`. Design §28 lists both as the +step "Delta in pi", before the layered rebuild. Until then, keep a `Map` from source element to result yourself, which +is what `mapShared` will do. +::: + +## Cookbook + +| Case | How | +| --- | --- | +| select a field, look up by key, a small aggregate | compute it; the value is small | +| filter or map a list | `mapShared` over the elements | +| redaction | preferably separate documents per visibility; otherwise `mapShared`, computed once in the shared store, never per view | +| join two sources | `mapShared` plus the looked-up source's revision | +| a window over a large list | take a section of the source elements without rebuilding them; cache per-element measurements by identity; keep the window's start stable so changes arrive as appends | +| a large view combined from several sources | re-path the sources' operations, as Durable's `ConversationView` does; application code never interprets ops | + +**Container width** (R24): structural sharing is O(change) along a path, not across a container's width. Changing one +item of a flat array copies the array's slots, and `mapShared` visits every element once (one identity comparison +each). Large collections are keyed documents or bounded windows. A benchmark of long arrays is to set the documented +limit. + +## Readables that are not documents + +Three small readables round out the toolkit: + +- `constant(value)`: never changes. Start data uses it (chapter IX). +- `self.writable(initial)`: a host-local readable of any in-process value (functions included), compared by identity, + with `set`. It never goes on the wire; it exists for registry contributions that change (chapter XIII). +- Source readables from elsewhere: a Durable view or document watch is already a Chord readable, because Durable and + Chord v2 share the readable type (`chord/src/state.ts`), so it can feed a derive or be exposed as state directly. + +:::review | Review: exposing a writable (Q14, open) +A `writable` may hold functions; nothing yet stops a factory from returning one as a state member, which would reach +the wire encoder. Review round 3 proposes that only members with a JSON schema in the contract are exposable, checked +when the object is served. Undecided. +::: + +## Presence: who is looking + +`host.references(Type)` is a readable of who holds references to objects of a type on this host: input, principal and +client. Client holds carry an opaque origin id, so two tabs are two clients. + +```ts +host.serve(NoteStore, ({ session }, self) => { + const viewers = self.derive("viewers", [host.references(Notes)], ([refs]) => + refs.filter((r) => r.input.session === session).length); + return { …, viewers }; +}); +``` + +A link drop is a departure: accurate and simple. Smoothing (a phone switching networks makes the count flicker) is the +UI's job, for example a debounced count. Counts per person are derived by the app from principals and origins. There +is no consumer-side linger: a closed tab closes its link and keeps nothing. The provider's `linger` keeps a +just-closed object warm, and that also keeps the worker alive through `host.activity`. + +**"Leaving" is never derived from an unmount.** A UI reload unmounts and remounts components; a component unmounted by +a reload has not left. Leaving is explicit navigation, or presence (a dropped reference). This replaces p2's presence, +viewer and login-abandon leases with one mechanism. + +:::note | Built so far +Readables and derives are layer L2 (a readable is today's `replicatedState`, derives reuse today's derived cells over +it). Not started in the rebuild. +::: + +# Things the OS Gives You {#natives lede="Start data, host-lifetime objects and natives: how a plugin opens a file watcher or a socket so that it is closed exactly once, also across reloads."} + +Studio's notes get an inbox: a text file dropped into the session's `inbox/` directory becomes a note. That needs a +file watcher, a **native**: an OS-level value (a file handle, a socket, a process, a database connection) that cannot +be JSON, cannot be copied, must be opened, and must be closed exactly once. It also needs the session's data +directory, which the plugin does not know. This chapter covers both. + +## Start data is an object + +A worker knows things before any plugin loads: its session id, the data directory, init data from the server's start +message. There is no special mechanism for them. Start data is an ordinary `local` object with `lifetime: "host"` and +constant state, served by base code's core facet: + +```ts +// studio/src/core/contract.ts +export const Init = obj({ id: str(), dataDir: str(), cwd: str(), env: opt(str()) }); +export const SessionInfo = local("studio.session-info", { + version: 1, + input: obj({}), + members: { init: Init }, +}); + +// studio/src/core/worker.ts +export const coreWorkerFacet = (init: Static) => defineFacet((host) => { + host.serve(SessionInfo, () => ({ init: constant(init) }), { lifetime: "host" }); + // … the Harness and other core objects (chapter XIV) +}); +``` + +Any plugin on the worker declares a reference and reads it **after activation**: in an opener, a derive, a routine or +a method, never synchronously in the factory. + +```ts +const info = self.get(SessionInfo, {}); +// later: info.init.value.id, info.init.value.dataDir +``` + +Paths are derived by the plugin from generic start data (`join(dataDir, id, "notes-inbox")`), never provided per +plugin. Platform bindings work the same way: on a Durable Object, a core object's `self.open` returns the binding +(`state`, `env`) and its close does nothing. + +## `self.open` + +```ts +const watcher = self.open( + async ({ signal }) => watchDir(path, { signal }), // open: during activation + (w, reason) => w.close(), // close: when the object ends or restarts + { closeTimeout: 10_000 }, // optional; host default 30 s +); +watcher.value; // the watcher; reading it before activation throws not_ready +``` + +`self.open` returns a handle with `.value` (type `Native`), not a proxy. The rules: + +- **Natives live only in objects.** Not at module level, not in the facet. A module-level watcher would survive a + reload with nobody to close it. +- **Order:** natives open one after another in declaration order (a later opener may read an earlier handle's + `.value`) and close in reverse. The object goes live after all opens succeeded. +- **The value never changes during an attempt.** No replacement, no versions, no stale-handle logic. To change a live + native, the object restarts. +- **A failed open** makes the object `failed`: transient with backoff by default, broken if the opener says so + (`throw broken(error)`). A partly failed activation closes the natives already open, with reason `end`; the next + attempt opens all of them anew. +- **`close` gets a reason:** `{ kind: "reload" }` (the plugin reloads and this native object restarts), `{ kind: + "stop" }` (the host shuts down) or `{ kind: "end" }` (the last reference ended, or the type was removed). Each + `close` is bounded by `closeTimeout`, then reported as a possible leak while the rest continues. A throwing `close` + does not stop the others; `host.close()` collects the errors, background ends report to `onError`. +- **Behaviour around a native goes into routines** (chapter X). Listeners on natives outside routines are forbidden. + +An object with at least one `self.open` is a **native object**; one without is a **pure object**. The difference +decides what a reload does (chapter XV): pure objects are swapped in place, native objects restart. + +## The inbox + +The inbox must run as long as the worker, whether anyone looks at the notes or not. So it is a `lifetime: "host"` +object. The facet cannot `self.get` a type it serves, and the inbox has no meaningful input, so it is an object +without input that reads its session from the start data: + +```ts +// notes/contract.ts +export const NoteInbox = local("notes.inbox", { version: 1, input: obj({}), members: {} }); + +// notes/worker.ts +host.serve(NoteInbox, (_input, self) => { + const info = self.get(SessionInfo, {}); + const watcher = self.open(async ({ signal }) => { + const { id, dataDir } = info.init.value; // read in the opener: after activation + return watchDir(join(dataDir, id, "notes-inbox"), { signal }); + }, (w) => w.close()); + + self.routines.loop("import", async (ctx) => { // chapter X + const store = await ctx.get(NoteStore, { session: info.init.value.id }); + for await (const file of watcher.value.files(ctx.signal)) { + await store.add({ text: await readFile(file, "utf8"), by: "inbox" }, ctx); + await rm(file); + } + }); + return {}; +}, { lifetime: "host" }); +``` + +Note `ctx.get(NoteStore, …)` in the routine rather than `self.get` in the factory. The factory runs before the +session id is known (start data is read after activation), so a host-lifetime object without input gets +session-scoped objects with `ctx.get` in a routine or method (R22). A `get` whose input is computed after activation is +to be added only if a real case needs it. + +## Natives with work in flight + +A restart is sequential (R16): close the old native, then open the new one. When the old one still has work running, +use one of two patterns: + +- **Drain in the background.** `close` detaches the old resource and returns at once; it disposes of it when its work + finishes, or after `closeTimeout` at most. Chord opens the new one immediately. Code mode's sandbox does this: a + script running during a reload finishes on the old sandbox (for up to 30 s, Q7; a longer one is aborted and its turn + fails), new scripts start on the new one. +- **Swap through a registry.** The resource is a registry contribution (chapter XIII). Readers always look up the + current one at use; a reader that must wait for one uses a cancellable "wait until present" with the park deadline. + +Chord aborts only the object's own calls and jobs on close. Other work that uses the native (Durable tool calls on a +sandbox) is the owner's business: `close` may drain instead of tearing down. What an old native does after its owner +restarts is also the owner's `close`: a model runtime keeps working until collected, a closed SSH connection fails its +in-flight `exec`. + +## Natives that follow a relocation + +A session can move to another directory. The inbox watcher must follow. Restarting the object on every move would be +heavy-handed; instead the native is a stable manager opened once, and a routine on the session location switches its +internal resource and bumps a generation, so late events from the old location are dropped: + +```ts +const watcher = self.open(async ({ signal }) => new SwitchableWatcher({ signal }), (w) => w.close()); +self.routines.on(location.cwd, (cwd) => watcher.value.switchTo(join(cwd, ".notes-inbox"))); +``` + +## Call-scoped natives + +A native that lives exactly as long as one call (a temporary SSH session for a directory listing, a file handle for a +download) is opened and closed inside the method with `try/finally`. There is no `ctx.open`: + +```ts +list: async ({ path }, ctx) => { + const session = await ssh.open(config, ctx.signal); + try { return await session.readDir(path, ctx.signal); } + finally { session.close(); } +}, +``` + +## Core vs. plugin + +`SessionInfo` comes from base code, `NoteInbox` from a plugin; both use the same primitives. Where code lives decides +which it is. **Base code** (the app's `src/`, imported statically by the start code under a fixed name such as +`@studio/core`) is always there, not switchable, and a change restarts the hosts. **A plugin** (its own directory, +found by listing, named by its `package.json`) is switchable and reloads. Core is what the app always needs and +everything builds on: the harness, the session list, access, the session location. Optional features are plugins; +code mode is a plugin. + +:::note | Built so far +Natives and host-lifetime objects are layer L4. `constant` is decided (R23), not built. +::: + +# Standing Behaviour {#routines lede="Routines, reconciling instead of notifying, and how two owners keep a rule between them without a shared transaction."} + +Notes older than 30 days should go every hour. Inbox files should be imported as they arrive. Notes of an archived +session should become read-only. Each of these is behaviour that stands, not a call someone makes. Written as bare +`setInterval`s and event listeners, it would keep running old code after a reload and touch state from nowhere. Chord +has one declared form for it: the **routine**. + +## Four kinds + +```ts +self.routines.every("cleanup", HOUR, (ctx) => dropOlderThan(30 * DAY, ctx)); // due time survives reloads +self.routines.on(settings, async (value, ctx) => applyLimit(value, ctx)); // a readable or event source +self.routines.loop("import", async (ctx) => { // runs until the object ends + while (!ctx.signal.aborted) await importNext(ctx); +}); +self.routines.custom("ticker", ({ signal, run }) => { // anything else + const timer = setInterval(() => run(tick), 1000); + return () => clearInterval(timer); // cleanup undoes the setup +}); +``` + +Routines start when the object goes live, stop when it ends, and restart with the new code on every reload of their +plugin. Each run gets its own `ctx`; errors are caught and reported, never uncaught. Routines are named (`on` +optionally by its source) so Chord recognizes them across reloads; that is how `every` keeps its due time. + +## Serialized per routine + +The new instance of a routine starts only after the old handler in flight has settled; only that routine waits, for at +most the routine handover timeout (10 s), after which it starts anyway and the stuck handler is reported. + +The reason, as a trace: a supervisor routine reconciles pending worker restarts. The old handler is midway through +`restart(W1)` when the plugin reloads. Without serialization the new routine starts at once, sees W1 still pending, and +restarts it a second time. + +## Reconcile, don't track + +A routine **reconciles from the current value**: it makes the world match what it sees now. It writes no "in +progress" flags, and it starts a job for long work (chapter XI). A handler that only cares about the latest value +compares with the current value itself; there is no `overlap` option. Subscribing to the object's own documents is +gapless: every commit is seen. + +No bare timers or callbacks that touch state outside routines, jobs or calls. That is what lets Chord stop all of an +object's behaviour when it ends or reloads. + +## Rules between two owners + +Every thing has exactly one owner, the smallest object it belongs to. Some rules span two: "notes of an archived +session are read-only" involves the catalog (which knows archiving) and `NoteStore` (which holds notes). Two tempting +designs are wrong: + +- **A cross-owner transaction** (archive and lock the notes atomically) would make the catalog know every plugin and + couple their storage. +- **A notification** (`catalog` calls `notes.archive(session)`) can be lost: the worker is not running, the link drops, + the process crashes between the archive and the call. + +The design's rule: **the owner of the dependent data enforces the rule** by reconciling from the other owner's state +with an idempotent routine. + +```ts +// NoteStore: an archived session's notes are read-only +const flags = self.state(NoteFlags); // { archived: boolean }, NoteStore's own document +const catalog = self.get(Catalog, {}); // the server's session list: another owner +self.routines.on(catalog.sessions, async (sessions, ctx) => { + const archived = sessions[session]?.archived ?? false; + if (flags.value.archived !== archived) await flags.change((d) => { d.archived = archived; }, ctx); +}); +// add and remove refuse while flags.value.archived +``` + +After a crash the routine runs again with the current value. The catalog does not know the notes plugin exists. The +same pattern closes a forward of an archived session in the ports plugin, aborts a job whose grant was revoked +(chapter VII), and keeps any invariant between plugins. + +## Reading without starting a worker + +The session list in the sidebar shows each session's note count. Getting `NoteStore { S1 }` for every session would +start every worker. The answer is plain publishing between two objects of one plugin, no Chord mechanism: the worker +side reports a summary to a server-side object by a method call from a routine, and clients read the server object. + +```ts +// notes/contract.ts +export const NoteCounts = remote("notes.counts", { + version: 1, input: obj({}), placement: () => Server(), + variants: { + owner: { state: { counts: map(int()) } }, + worker: { methods: { report: method(obj({ session: str(), count: int() })) } }, + }, +}); + +// notes/server.ts +const Counts = defineState("notes.counts", { + schema: obj({ bySession: map(obj({ count: int(), incarnation: str() })) }), + version: 1, initial: () => ({ bySession: {} }), persist: true, // survives a server restart + checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64, +}); + +host.serve(NoteCountStore, (_input, self) => { // local, lifetime "host" + const counts = self.state(Counts); + return { + counts: self.derive("counts", [counts], ([c]) => mapValues(c.bySession, (e) => e.count)), + report: ({ session, count, incarnation }, ctx) => + counts.change((d) => { + const last = d.bySession[session]; + if (last && last.incarnation > incarnation) return; // uuidv7: an older worker, ignore + d.bySession[session] = { count, incarnation }; + }, ctx), + }; +}, { lifetime: "host" }); + +host.serve(NoteCounts, (_input, principal, self) => { + const store = self.get(NoteCountStore, {}); + if (principal.kind === "worker") return { + variant: "worker", + report: ({ session, count }, ctx) => { + if (session !== principal.session) throw forbidden(); // a worker reports only itself + return store.report({ session, count, incarnation: ctx.incarnation }, ctx); + }, + }; + return { variant: "owner", counts: store.counts }; +}); + +// notes/worker.ts, inside NoteStore +const counts = self.get(NoteCounts, {}); // a reference to the server +self.routines.on(count, (n, ctx) => inVariant(counts, "worker") && counts.report({ session, count: n }, ctx)); +``` + +The server object keeps the latest value per session (in a document if it must survive a server restart) and +**ignores reports from an older worker incarnation** (incarnations are uuidv7, so they order by start time). A worker that was replaced may still deliver a late report; the +fence is the caller's incarnation, which Chord stamps on every call from the link handshake (R20): `ctx.incarnation`, +next to `ctx.principal`, not part of the principal or its key, never sent as data, so it cannot be forged. Session +status, subagent lists, unread marks and artifact lists in p2 all follow this pattern. The store is local and has no +caller incarnation of its own, so the view passes it on as data. + +:::review | Review: `ctx.incarnation` across a relay (Q13, open) +A worker's call to the server arrives directly, so `ctx.incarnation` is the worker's. But a call relayed through the +server (a tab's call to a worker) carries the server's incarnation on the worker's side, because the worker's +handshake is with the server. Review round 3 proposes that relays stamp the origin host's incarnation, as they stamp +the principal. Undecided; the pattern above only needs the direct case. +::: + +:::note | Built so far +Routines are layer L8. The design treats `every` as one of four forms over one runner; review round 3 proposed cutting +to `on` and `loop` only (C4), not decided. +::: + +# Work That Outlives the Call {#jobs lede="Jobs: owned work started by a method that keeps running after its caller leaves, with cancel, dedupe and a recorded outcome."} + +Importing a 50 MB text file into notes takes a minute. The tab that started it may close; the import must go on, be +cancellable, not run twice when the tab retries after a link blip, and its result must be recorded. A method cannot do +this (it dies with its caller), a routine is standing behaviour rather than a request, and a Durable task is for work +that must resume after a crash. Chord's form for it is the **job**, built in pi-durable's sandwich: **intent** (a +transaction), **effect** (anything), **outcome** (a transaction in the current code). + +## Defining and running a job + +```ts +// notes/worker.ts +const ImportJob = defineJob("notes.import", { + version: 1, + args: obj({ path: str() }), + progress: obj({ lines: int() }), + result: obj({ added: int() }), + concurrency: 1, // one import at a time per NoteStore +}); + +const ImportResults = defineState("notes.import-result", { + schema: obj({ outcome: union([obj({ added: int() }), obj({ error: str() })]) }), + version: 1, initial: () => ({ outcome: { added: 0 } }), keyed: true, + checkpointWhen: () => true, +}); + +host.serve(NoteStore, ({ session }, self) => { + const list = self.state(NoteList); + const results = self.state(ImportResults); + const imports = self.job(ImportJob, { + run: async ({ path }, ctx) => { // 2. effect + let lines = 0; + for await (const line of readLines(path, ctx.signal)) { + await addNote({ id: `${ctx.jobId}:${lines}`, text: line, by: "import" }, ctx); // keyed by job id + ctx.progress({ lines: ++lines }); + } + return { added: lines }; + }, + finish: async (outcome, tx) => { // 3. outcome, current code + (await tx.draft(results, outcome.id)).outcome = + outcome.kind === "done" ? outcome.result : { error: outcome.error }; + }, + }); + return { + …, + imports: imports.running, // live list of runs + startImport: ({ path, requestId }, ctx) => + imports.start({ path }, ctx, { key: requestId }), // 1. intent (refined below) + abortImport: ({ id }) => imports.abort(id), + }; +}); +``` + +`defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })`; `progress` and the outcome +are typed by the schemas. + +:::sequence animate | The job sandwich. The intent commits before anything runs; the effect runs outside any transaction; the outcome commits in the current code. +actor tab "tab" +actor store "NoteStore { S1 }" accent +actor run "run" "effect" blue +tab -> store: startImport({ path, requestId }) +store => store: 1. intent: busy check, dedupe by key, commit +store --> tab: { id } +store -> run: start after the commit +run -> run: read lines, add notes (keyed by jobId) +run --> store: progress { lines } (throttled) +note tab..store: the tab may close here: the job counts as a reference +run --> store: settles: { kind: "done", result } +store => store: 3. record outcome, finish(outcome, tx), ack +::: + +## The three steps + +**1. Intent.** `job.start(args, ctx, { key? })` is shorthand for `tx.start(job, args, { key? })` inside a +transaction, which may contain other writes. The busy check (`concurrency`) and deduplication (the same key returns the +existing run) are atomic and in the same commit; `run` starts only after the commit. Run ids are uuidv7. + +**2. Effect.** `run(args, ctx)` may do anything: processes, network, other objects. It writes no document itself; it +returns the outcome. External writes carry a dedupe key derived from `ctx.jobId` (above: note ids), so an interrupted +attempt cannot apply them twice. On a reload `run` keeps its old code. + +**3. Outcome.** When `run` settles, Chord records the outcome in its own record, then calls the **current** code's +`finish(outcome, tx)` and acknowledges the run in the same commit. The outcome is tagged: + +```ts +type Outcome = { id: string } & ({ kind: "done"; result: R } | { kind: "failed"; error: "threw" | "aborted" | "lost" }); +``` + +A throwing `finish` changes nothing and is retried with a backoff (1 s to 60 s), and after the next reload. `finish` +is optional and follows Durable's transaction rule. + +## Dedupe ends with the run + +Deduplication covers **unsettled runs only** (Q9). Once `finish` has committed, Chord drops the run's record and key +and releases its waiters' references; it keeps nothing beyond a run's lifetime. Dedupe after completion is the +owner's. Studio records which run a request started, in the same commit as the intent, and `finish` records the +result by run id (`ImportResults` above): + +```ts +const Requests = defineState("notes.import-request", { + schema: obj({ run: str() }), version: 1, initial: () => ({ run: "" }), keyed: true, persist: true, + checkpointWhen: () => true, +}); + +startImport: async ({ path, requestId }, ctx) => { + const prior = await requests.get(requestId); // read outside the transaction (R13) + if (prior) return { id: prior.run }; // started before, maybe finished long ago + return self.tx(async (tx) => { + const run = await tx.start(imports, { path }, { key: requestId }); // dedupes a concurrent retry + (await tx.draft(requests, requestId, { tags: [SessionTag(session)] })).run = run.id; + return { id: run.id }; + }, ctx); +}, +``` + +Two concurrent retries both miss `prior`, but the key makes `tx.start` return the same unsettled run. The client +watches `imports` for progress and reads the result by run id once it is gone from `running`. + +Why Chord does not keep completed runs itself: either 100,000 keyed imports would stay in memory forever, or a +retention limit would let a retried key rerun its effect. The owner knows which results matter. + +## Watching, waiting, aborting + +- **`running`** is a readable of `{ id, version, args, progress, phase }`. `phase` is `running`, `finishing` (outcome + recorded, `finish` not yet committed) or `blocked` (with the reason). `concurrency` counts only active runs. The + panel shows a progress bar from it. +- **Progress** is in memory, throttled and sent as deltas. For output, use `OutputBuffer` and `Progress` from + `@earendil-works/chord/output`: bounded windows, adaptive throttling, no write amplification. +- **Waiting:** `start` returns a run with `outcome(ctx)`. The caller's abort ends only its wait, never the run. +- **Abort is a request:** `job.abort(id)` fires the run's `ctx.signal`; the run decides what it returns (a result + that already arrived stays a success). A run that ignores the signal keeps its slot until it ends. `finish` runs in + Chord's own context. + +## Lifetime, persistence, revocation + +**Jobs keep their owner alive.** A running, finishing or blocked job counts as a reference to its object, which ends +only once all its jobs are settled. Starting an import and closing the tab does not abort it. Unsettled runs also +count as activity for the worker's keep-alive. + +**Revocation takes precedence** (R11). Revoking a principal ends its objects at once, whatever jobs remain; the jobs +get the abort signal and the normal close timeout. A job runs as its object's principal. + +**Persistence:** jobs are in memory by default and gone with the process. `persist: true` (only for stable +identities) keeps the record: a run live at a crash ends `lost` at the next activation and `finish` records it. +Nothing is rerun automatically. + +**Versions:** a run started under an older job version keeps its version; `finish` and `running` see it through pure +`accepts` adapters, and stored outcomes are never rewritten. A version bump without an adapter does not refuse the +reload: the old outcome is `blocked`, kept and reported until a later reload brings the adapter. + +**Native objects:** a job in a native object ends `aborted` when the object restarts on a reload (chapter XV). In a +pure object it keeps running on its old `run` code. + +## What is not a job + +| Work | Use | +| --- | --- | +| request and response that may die with its caller | a method | +| input into running work (a pasted code, an approval) | a flow (chapter XII) | +| work that must resume after a crash | a Durable task | +| standing behaviour | a routine | + +:::review | Review: jobs are the largest open simplification (#9, C3) +Decision #9 (deferred until login is ported): a login or a long relocation is a job whose state consumers want to +address and watch (`LoginFlow { id }`). Options: (a) jobs stay records inside their owner, watched through `running`; +(b) a job is an object of its own, addressed by its id, living until done. Separately, review round 3 proposed cut C3: +shrink jobs to a runner over documents (no job versions, `accepts`, `blocked`, `persist`) or move them out of the +kernel. Not decided. `chord-next-walkthrough.md` concept 17 still shows the older outcome shape `{ ok: true, result }`; +the design tags outcomes by `kind`. +::: + +# Asking a Person {#flows lede="Flows and the Questions service: how running work waits for an answer from someone who may be on any tab, may reconnect, and may answer twice."} + +A provider login asks for a pasted code, then an account choice. A dangerous tool call waits for the owner's +approval. The agent asks everyone present a question. In each case running work waits for a person, who may be on any +tab, may lose the connection halfway, may answer late or twice. Two patterns cover this: a **flow** kept as its +owner's state, and the **Questions** service, a shared store for questions. + +## A flow is state, not a call + +The tempting design is a long-held server call, `step()`, that blocks until the next step. A link drop fails it, and +the consumer cannot tell whether it missed a step. Chord's answer is that the owner keeps the flow as **state**, which +re-holds by itself after any drop: + +```ts +// models/contract.ts (sketch from the design) +start: method(obj({ provider: str() }), obj({ id: str() })) // uuidv7 +answer: method(obj({ id: str(), step: int(), value: str() }), Answered) // ok | "step no longer active" +cancel: method(obj({ id: str() })) + +// LoginFlow { id }: state +type FlowState = + | { kind: "running"; step?: { id: number; prompt: LoginPrompt }; notices: Notice[] } + | { kind: "done" | "failed" | "cancelled"; notices: Notice[] }; +``` + +The UI watches the state and renders the current step. A late or second answer to a step that is gone is refused. A +promise-based callback inside the work (pi-ai's `prompt` during a login) stays **inside** the owner: its job keeps the +resolver, and nothing crosses an object boundary. + +Imperative consumers (tests, scripts, a CLI, an agent tool) use a generic client helper over the same state, not a +second protocol: + +```ts +async function* steps(flow: LoginFlowRef, ctx: Ctx) { + let last = 0; + for (;;) { + const s = await until(flow.state, (v) => v.kind !== "running" || (v.step !== undefined && v.step.id > last), ctx); + if (s.kind !== "running") return s.kind; + last = s.step!.id; + yield s.step!; + } +} + +for await (const step of steps(await ctx.get(LoginFlow, { id }), ctx)) + await login.answer({ id, step: step.id, value: answerFor(step) }, ctx); +``` + +`last` keeps steps from being lost or repeated across re-holds. + +:::review | Review: where a flow lives (#11, deferred) +Decision #11, deferred until login is ported: (a) a keyed document of its owner, or (b) an object `LoginFlow { id }` +living until done; tied to #9 (jobs as objects). The `LoginFlow { id }` address above assumes (b). Also undecided is +how "leaving" ends a login (p2 cancels when the starting owner leaves everywhere); with presence it is a dropped +reference, never an unmount (chapter VIII). +::: + +## The Questions service + +The Questions service (`apps/p2/docs/questions.md`) is a p2 design, not a Chord feature, built from Chord's +primitives: a shared store with per-principal views. The asker gets `ask`; viewers get the open questions and +`answer`. + +```ts +export type Question = + | { kind: "text"; message: string; secret?: boolean; help?: string } + | { kind: "select"; message: string; options: readonly { id: string; label: string }[] } + | { kind: "confirm"; message: string; allow: string; deny: string }; + +type Policy = { mode: "first" } | { mode: "quorum"; count: number } | { mode: "all"; join?: "live" | "snapshot" }; + +ask(question, { + audience, // a readable or a list of principals + policy, + key?, // stable per asking flow: a re-ask reattaches to the pending question + deadline?, // then resolve with what is there; `missing` names who did not answer +}, ctx): Promise; +``` + +Its rules: + +- **The store is the truth.** Pending questions and their resolvers live in the store; `open` is a readable over it, + filtered per viewer. Nothing is a document, so a reload or migration cannot leave a ghost question. +- **One settlement point.** Register the waiter and the abort listener before publishing the question; an answer, an + abort or the deadline atomically takes the entry out before resolving. A late or second answer gets "no longer + active". Answers are validated against the stored question (a select id must be one of the options). Never await an + answer inside a transaction. +- **Policies over a live audience.** `first` resolves with the first answer; `quorum(n)` after n answers; `all` when + every member of the audience has answered. With `join: "live"` (the default), someone who joins becomes required and + someone who leaves drops out. +- **Authorization.** The asker is stamped by the host (a worker asks only in its own session's scope; secret questions + only from server askers). Each question stores who may answer it; a guest's send access does not imply approval + rights. Secret answers never appear in `open`, documents or logs. +- **Link drops and crashes.** A rejected `ask` is not a denial. Askers retry with the same `key` until their own signal + aborts; a pending `ask` counts as an active call for keep-alive. +- **Where it lives.** On the host where the asking work runs: session questions on the session's worker, login + questions on the server. UIs reach both through the usual relay. + +The store in Chord v2 terms: + +```ts +host.serve(QuestionStore, ({ scope }, self) => { + const pending = self.open(async () => new PendingQuestions(), (p) => p.rejectAll("store closed")); + const open = self.derive("open", [pending.value.readable], ([all]) => all); // data only, no resolvers + return { open, ask: (args, ctx) => pending.value.ask(args, ctx), answer: (args, ctx) => pending.value.answer(args, ctx) }; +}); + +host.serve(Questions, ({ scope }, principal, self) => { + const store = self.get(QuestionStore, { scope }); + if (isHost(principal)) return { variant: "asker", ask: (args, ctx) => store.ask({ ...args, asker: principal }, ctx) }; + if (!mayView(principal, scope)) throw forbidden(); + return { + variant: "viewer", + open: self.derive("mine", [store.open], ([all]) => openFor(all, principal)), + answer: ({ id, value }, ctx) => store.answer({ id, value, by: principal }, ctx), + }; +}); +``` + +The pending map with its resolvers is in-process state with promises, so it is a native of the store (opened and +closed with it). Its `close` rejects every waiter; askers retry with their key against the new store after a reload. + +:::review | Review: questions.md predates the v2 rules +`apps/p2/docs/questions.md` writes the store as `self.resource(...)` with a `replace` hand-over, which the design +rejected ("resources replaced in place"), and opens Chord call scopes in Durable hooks with `host.root`, which Q6 +replaced with `chord(fn, context)` (chapter XIV). Design §20 says the Questions store "must live in an object (it +predates natives only in objects)". The sketch above is this guide's translation, with the derive per viewer; filtering +per viewer in the store, once per variant, may be cheaper for large audiences. The `ask` signature passes typed +objects as arguments, which is fine (data), but the `audience` readable form cannot cross a link: across links, +audience must be data (principals) or computed by the store. +::: + +## Choosing + +| Case | Use | +| --- | --- | +| a multi-step login in one owner | a flow: the owner's job keeps the resolver, state shows the steps | +| a tool needs the owner's approval | Questions, `first`, audience the owner | +| the agent asks everyone present | Questions, `all`, audience from presence | +| deploy needs 2 of 3 approvals | Questions, `quorum(2)` | +| a notice (auth URL, device code, progress) | not a question: the asking work's own progress state | + +# Extension Points {#registries lede="The facet, registries and the HTTP component: how plugins add behaviour to each other without callbacks, and how the owner stays in charge."} + +The notes plugin wants a download route, `GET /s/:session/notes.txt`. A logs plugin wants to time every request. +Several plugins add model providers, tools for the agent, commands for the TUI. Someone must combine these +contributions, decide conflicts, and replace one plugin's part as a whole when it reloads. Between plugins exactly two +things pass: object types and **registries**. + +## The facet, completed + +The facet has appeared since chapter II as the place that serves types. It is also itself a **pure object** with the +host's lifetime: + +```ts +// notes/server.ts +export default defineFacet((host, self) => { + const prefs = self.state(NotesPrefs); // plugin-wide documents + self.contribute(Http, { routes: { "GET /s/:session/notes.txt": exportNotes } }); + host.serve(NoteCountStore, …, { lifetime: "host" }); + host.serve(NoteCounts, …); +}); +``` + +- It may hold documents, references, routines and contributions; **no natives** (those need an object with its own + lifetime). +- Its identity is the plugin's name, so its persisted documents survive restarts and reloads. +- Served objects go live after the facet and may read and write its documents (chapter V). +- `host.serve` and `self.contribute` are valid only while the setup runs. The facet never `self.get`s a type it + serves. + +## Registries + +A registry is an extension point owned by someone (a Chord component, the app, a plugin). Each contributor +contributes **at most one value** per registry; the table is an immutable, ordered list of `(contributor, value)`; +the registry's `compose` turns it into the value consumers use. + +```ts +// owned by the HTTP component +export const Http = defineRegistry("chord.http", { + version: 2, + accepts: { 1: (old: HttpContributionV1): HttpContribution => upgrade(old) }, + compose: (table, { report }) => buildRouter(table, report), // pure; always delivers a value +}); + +self.contribute(Http, { routes: { … }, wraps: [ … ] }); // a contributor +const router = host.registry(Http).value; // a consumer: live +``` + +**`compose` runs at runtime, whenever the table changes:** after a reload's commit, when a contributor goes live or +ends, when a contribution changes. It is not a staging check: contributions can be dynamic (known only once a native +is open), so no check before a reload could be complete. `compose` always delivers a value with its fallback built in +(e.g. the first by plugin name wins) and reports conflicts through `report`, which become owner notices and telemetry +events. Readers get a plain value and never handle conflicts. Only if `compose` throws is the registry in an error +state, and readers get an error when they use it. Work that started earlier keeps the value it read. + +**Rules:** + +- **Who contributes** (R21): only facets and `lifetime: "host"` objects; only their `self` has `contribute` (a type + error elsewhere). Every object may read registries. +- **A contribution is in the table exactly while its contributor is live:** a facet's while its plugin is loaded; an + object's from `live` until it ends, fails or restarts. +- **Order** is sorted by plugin name: deterministic on every host and across reloads, and deliberately meaningless. + Meaningful order comes from the owner's composition (HTTP phases, below). Helpers: `keyed(table, entriesOf)`, + `ordered(table, itemsOf, orderOf?)`. +- **One authoritative declaration** (Q8): only the defining plugin's facet declares the registry to the host + (`compose`, version, `accepts`). Other plugins' tokens only identify it by name and types. A reload of the defining + plugin replaces the declaration atomically and the table recomposes. If the defining plugin is not loaded, the + registry has no value (readers get an error like a failed reference) and contributions wait in the table. +- **Versions** (Q8): a contribution carries the registry version its plugin was built against; older ones are adapted + by the defining facet's pure `accepts`. One without a matching version or adapter is left out and reported naming + plugin, registry and versions; its plugin keeps working. Contributions are in-process values (often functions) and + are not schema-checked. +- **Use sites guard contributions.** A contribution is untrusted code (third-party plugins are a goal). Every use site + catches throws and rejections when calling an entry, reports naming the contributor, and continues with the others. +- **The owner decides** conflicts, order and selection. Registries add behaviour; they never replace an owner's checks. + +## Natives through registries + +Registries are the only way a native reaches another plugin. An entry lives exactly as long as its contributor; +consumers look up at each use (or follow the composed value) and keep nothing from an entry beyond a use. The models +plugin's host-lifetime object contributes its runtime; the harness looks it up at every request. After a models reload +the next request gets the new runtime, while a running stream finishes on the old one. + +A handler that stores `const r = entry()` in a module variable and uses it for days has a bug: after the contributor +restarts it holds a closed native. + +## Contributions that change + +Some contributions are known only at runtime: MCP tools appear when a server connects. A contribution is either fixed +for its contributor's life (dynamic parts behind handles read at use) or a **readable**, which Chord subscribes to; +on every change Chord replaces that entry and runs `compose` again. For in-process values, `self.writable(initial)` is +a host-local readable of any value (no JSON, no wire, compared by identity) with `set`: + +```ts +const ext = self.writable(mcpExtension([])); // a Durable Extension, no tools yet +self.contribute(HarnessExtensions, ext); +self.routines.on(mcp.tools, (tools) => ext.set(mcpExtension(tools))); +``` + +There is no remove call. An entry leaves when its contributor ends, or after a reload whose new code no longer +contributes. A live contributor with nothing to offer sets its readable to the registry's empty value (an extension +without tools, empty routes); the table keeps exactly one entry per live contributor. + +## Optional dependencies + +Registries also replace optional references (chapter VI). Radius forwards connections to ports the ports plugin +manages, but ports is optional. So radius owns `radius.forwards` and ports contributes its forwards; without ports the +table is empty and radius works. + +## The HTTP component + +HTTP is an optional Chord component (`@earendil-works/chord/next/http`) built on a registry. A route gets a plain +request summary (method, path params, query, principal, provenance such as local or relayed, selected headers, parsed +and capped JSON bodies, uploads in chunks) and a `ctx` whose `principal` is the request's, so `ctx.get(Notes, …)` gives +exactly the view that person may see. The route returns a description; the component does the HTTP work once (ranges +with 206 and 416, HEAD, download headers with a safe filename, `nosniff`, CSP and its nonce, caching, streaming, abort +on disconnect): + +```ts +type HttpResponse = + | { kind: "json"; value: JsonValue } | { kind: "page"; title: string; body: string } + | { kind: "bytes"; name: string; mime: string; size: number; read(offset: number, length: number): Promise } + | { kind: "redirect"; to: string } | { kind: "error"; status: number }; + +// notes/server.ts +self.contribute(Http, { + routes: { + "GET /s/:session/notes.txt": async (req, ctx) => { + const notes = await ctx.get(Notes, { session: req.params.session }); // the requester's view + const text = new TextEncoder().encode(render(notes.notes.value)); + return { kind: "bytes", name: "notes.txt", mime: "text/plain", size: text.byteLength, + read: async (offset, length) => text.subarray(offset, offset + length) }; + }, + }, +}); +``` + +Routes compose with `keyed`: two contributors with one pattern is a reported conflict, and the first by plugin name +wins. **Wraps** are middleware in phases the owner defines (`observe`, `headers`, `transform`, outermost first), +within a phase in contributor order. No numeric priorities: plugins have no single registration order, so Fastify's +lifecycle phases and encapsulation are the model, not Express. + +```ts +// logs/server.ts +self.contribute(Http, { + wraps: [{ + name: "access-log", phase: "observe", scope: "all", // default scope "own": only own routes + wrap: async (req, next, ctx) => { + const t = Date.now(); + const res = await next(req); + log(req, res, Date.now() - t); + return res; + }, + }], +}); +``` + +`route` narrows a wrap to one route; a `route` that does not exist is a reported conflict. Wraps may add a security +boundary on top of an owner's checks, never instead of them. A request uses the router as it was when it started. +Request resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary. + +## Channels and big bytes + +**Channels** are WebSocket byte channels (`send`, `onData`, `close`) for terminals, separate from object links and +with their own acknowledged windows, so a large state message never holds up keystrokes. + +**Big transfers are chunked by the owner,** not by Chord: bytes are a schema type (`bytes()`), carried natively +(never base64), and a 50 MB upload is a protocol of plain calls: + +```ts +beginUpload({ name, size }, ctx) → { upload } +append({ upload, offset, bytes }, ctx) // about 1 MiB per chunk; the awaited call is the backpressure +finishUpload({ upload }, ctx) → { id } +read({ id, offset, length }, ctx) → bytes +``` + +After a link drop the `append` in flight fails like any call; the writer asks the owner for the stored offset and +resumes. Unfinished uploads expire through the owner's routine. Chord has no stream type: it would need its own rules +for drops, cancellation, relays and restarts, which this offset protocol already survives. + +:::note | Built so far +Registries are layer L9. The HTTP component, channels and the chunked-transfer helper are deferred until after the +kernel and a first real proof (the harness, one tool with an approval pause, a code mode reload); until then the +playground serves its pages with plain app code. Their design stays the target. +::: + +# The Agent {#agent lede="pi-durable inside Chord: the harness as a core object, models, environments and tools through registries, and how a tool reaches Chord objects and people."} + +Each Studio worker runs a pi-durable harness for its session. The harness must run for the worker's whole life, +whether anyone watches or not. Its models come from the models plugin, its execution environments from the envs +plugin, its tools from every plugin, and all of those reload independently. This chapter assumes you know pi-durable +and shows only how it plugs into Chord. + +## Shared foundations + +Durable and Chord v2 share modules: Durable imports Chord's `context`, `delta`, `json`, `output` and `sqlite`, and v2 +imports the same, so `Context`, `Draft` and `JsonValue` are one type each. A v2 readable is today's +`replicatedState` (`chord/src/state.ts`), so Durable's `viewState()` and document watches can be exposed as Chord state +members without conversion. Only plain data crosses between Durable and Chord objects. + +## The harness is a core object + +```ts +// studio/src/core/harness.ts (base code) +import { Harness as DurableHarness } from "@earendil-works/pi-durable"; + +export const ChordHostKey = createContextKey>("studio.chord-host"); +export const HarnessModels = defineRegistry("studio.harness-models", { version: 1, compose: firstModels }); +export const HarnessEnvironments = defineRegistry("studio.harness-envs", { version: 1, compose: firstEnv }); +export const HarnessExtensions = defineRegistry("studio.harness-extensions", { + version: 1, + compose: (table, { report }) => extensionSnapshot(table, report), // one Durable Extension per plugin +}); +const HarnessRuntime = defineRegistry<() => DurableHarness, () => DurableHarness>("studio.harness-runtime", { + version: 1, compose: onlyEntry, // core-private: one contributor +}); + +host.serve(Harness, (_input, self) => { + const info = self.get(SessionInfo, {}); + const harness = self.open(async ({ signal }) => { + const { id, dataDir } = info.init.value; + const base = withContextValue(ChordHostKey, host, withAbortSignal(signal, BACKGROUND_CONTEXT)); + return DurableHarness.open(await openNodeSqliteStorage(join(dataDir, id, "durable.sqlite")), { + models: () => host.registry(HarnessModels).value, // read at each use + env: (target) => host.registry(HarnessEnvironments).value(target), + settings: () => currentSettings(), + registry: durableRegistry(host.registry(HarnessExtensions)), // a RegistryReader over the composed value + }, base); + }, (h) => h.close(BACKGROUND_CONTEXT)); + self.contribute(HarnessRuntime, () => harness.value); // for core's conversation objects + return { … }; +}, { lifetime: "host" }); +``` + +- **Lifetime `"host"`:** it runs for the worker's whole life, also while nobody holds it, because Durable tasks + continue. +- **Base code:** it lives in the core facet, so a plugin reload never restarts it; only a base-code change (a host + restart) does. +- **Registries, looked up at each use:** Durable's `models` and `settings` accept functions (pi `7ec1f648e`), so each + generation gets the current value. After a models reload the next request uses the new runtime; a running stream + finishes on the old one. +- **Durable's storage is its own.** Chord documents never commit together with Durable transcripts. + +The design writes the call as `Durable.open(storage, { … })`; pi-durable's actual entry point is +`Harness.open(storage, options, context)`, used here as `DurableHarness` to keep it apart from the core object +`Harness`. + +## Start order + +At host start Chord loads every facet and waits until every facet and every `lifetime: "host"` object **other than the +harness** is live or failed: the **start barrier**. Then the registries hold the contributions of every plugin that +loaded, and the harness opens and resumes scheduling (R17). An object that must contribute at start, like the model +runtime, is therefore `lifetime: "host"`. Nothing waits for specific contributions; what is missing shows where it is +used: + +| Missing | What happens | +| --- | --- | +| a tool | its call gets an error result the model sees | +| a model | the generation reports "model unavailable" and waits, not fails, until a model appears | +| a task definition | the task stays blocked (`missing_task`) until the plugin is installed or the task is aborted | +| a plugin that failed | reported with its error; a facet in backoff contributes later and the registries recompose | + +The same holds when a session's database is copied to an instance with other plugins: missing tools, sections and +hooks are skipped, documents of missing plugins stay untouched, stored selections are never rewritten. The UI must +show blocked tasks with their reason and offer to abort them; a turn waiting on one hangs until then. + +:::review | Review: the harness exception to the barrier (C17, R17) +R17 makes the harness start after the barrier, a special case in the barrier's rule. Review round 3 proposed cut C17: +no harness-specific exception; the harness is an ordinary object the app's start code holds after `load` resolves. +Not decided. `chord-next-walkthrough.md` concept 28 still says the harness's activation *is* the start barrier, which +R17 replaced. +::: + +## Tools that call Chord + +Durable tools, hooks and sections receive Durable's `ToolExecutionApi` (or `HookApi`) and a `Context`, never a Chord +`ctx` (Q6). Core put the host into the harness's base context under `ChordHostKey`, and every phase context inherits +it. A core helper opens a Chord call scope from any Durable context: + +```ts +// studio/src/core/chord.ts +export function chord(context: Context, fn: (ctx: Ctx) => Promise): Promise { + const host = context.value(ChordHostKey); + if (!host) throw new Error("not inside the Studio harness"); + return host.root(fn, { signal: context.abortSignal, telemetry: context.telemetry }); +} +``` + +The scope runs as the host principal; attribution is data; references taken with `ctx.get` are released when `fn` +settles. There is no wrapping of tools and no app-specific `defineTool`. The notes plugin gives the agent a tool: + +```ts +// notes/worker.ts +const addNote = defineTool({ + name: "notes_add", // tools are named _ + description: "Add a note to this session's notes panel", + parameters: Type.Object({ text: Type.String({ maxLength: 2000 }) }), + execute: (args, api, context) => chord(async (ctx) => { + const info = await ctx.get(SessionInfo, {}); + const store = await ctx.get(NoteStore, { session: info.init.value.id }); + const id = await store.add({ text: args.text, by: `agent:${api.conversationId}` }, ctx); + return { output: [{ type: "text", text: `added note ${id}` }] }; + }, context), +}); + +export default defineFacet((host, self) => { + self.contribute(HarnessExtensions, defineExtension({ name: "@studio/notes", tools: [addNote] })); + … +}); +``` + +A reload of the notes plugin replaces its extension in the table; Durable pins a registry snapshot per task phase, so +a tool call already running finishes on the old code and the next phase uses the new one. + +## Tools that need a person + +Clients are not addressable, so a tool cannot call the user's tab. A tool that needs the user goes through an owner's +state: the Questions service (chapter XII). The model-facing `ask_user` tool and an approval hook both call +`Questions.ask`; tabs render the open questions and answer them. + +```ts +hook(ToolTask, { + beforeTool: async (call, api, context) => { + if (!needsApproval(call)) return undefined; + const decided = await api.memo("approval", context); // a rerun does not ask again + if (decided !== undefined) return decided ? undefined : { block: "Denied" }; + const allowed = await chord(async (ctx) => { + const q = await ctx.get(Questions, { scope: { session } }); + for (;;) { + try { + const r = await q.ask( + { kind: "confirm", message: `Run ${call.name}?`, allow: "Allow", deny: "Deny" }, + { audience: approvers, policy: { mode: "first" }, key: api.taskId }, + ctx, + ); + return r.answer; + } catch (error) { + if (ctx.signal.aborted) throw error; // a link blip is not a denial: retry + await backoff(ctx); + } + } + }, context); + await api.memo("approval", allowed, context); + return allowed ? undefined : { block: "Denied by the user" }; + }, +}); +``` + +Durable runs `beforeTool` in the tool task's call phase; after the execute checkpoint a crash retries execution +without hooks, so the stored decision covers that retry. The approval must cover the final arguments: put the +approval hook after hooks that rewrite arguments. + +## Conversations for the UI + +Chat panels need a conversation's live view, submit, steer, stop. The design leaves p3's conversation API to the p3 +design; one construction that follows the rules: a core `ConversationStore { conversation }` (local) opens the Durable +watch as its own native, getting the Durable harness from the core-private `HarnessRuntime` registry contribution (a +native may reach another object only through a registry entry), and exposes a bounded window as a derive. Per-principal +`Conversation { session, conversation }` views in front of it check who may send (chapter XIX). + +## Code mode + +Code mode is a plugin. It owns its `pi-codemode` sandbox as a native; its tool entry looks up the **current** sandbox +when `execute()` starts, so a call pinned by a phase before a reload (waiting in an approval hook, say) starts on the +new sandbox. On `reload` and `stop` the old sandbox is sealed and drained in `close`: running scripts finish or hit +their deadline, for at most 30 s (Q7). After a worker stop Durable replays as it does today. + +## What lives where + +| Data | Where | +| --- | --- | +| transcript-bound: entries, documents that fork and rewind with a conversation | Durable | +| app state: session list, notes, drafts, grants, settings | Chord documents | +| data core enforces on conversations (read-only one-shots, cwd followers) | a core per-conversation policy document, written when the conversation is created (`createConversation({ init })`) | + +**Telemetry:** pass the host's adapter as `HarnessOptions.telemetry`; Durable stores trace carriers on submissions and +tasks (chapter XVII). + +# Reload and Rollout {#reload lede="What happens when you press Reload: per host, per object, and across the server, the workers and every tab."} + +Back to the preface. Mario changed the notes plugin's sort order and pressed Reload. Everything the earlier chapters +asked of plugin authors (synchronous factories, natives only in objects, routines instead of timers, data in +documents) exists so that this moment is safe. This chapter shows what Chord does with it. + +## On one host + +```text +1. load new bundles; an import or syntax error rejects +2. staging facet setups and every live object's new factory run with a recording self + (declarations only, nothing activated); any throw rejects +3. commit the host switches to the new code atomically +4. after per object: pure objects swap, native objects restart +``` + +**Rollback is all-or-nothing per host.** Any load or staging error leaves the whole host on its previous code, so a +host is always completely at one revision and its status is one value. Runtime states (a registry conflict, a failed +open, a migration) are not reasons to roll back; after the commit, errors stay with one object. There is no factory +trial, no migrate dry run, and no consumer contract check in staging: an incompatible consumer fails at bind with +`contract_mismatch` and a notice naming plugin and type. + +:::steps | Reloading @studio/notes on worker S1, object by object ++++ Load and stage +The new bundles load. The facet setup and the factories of `Notes` (three views), `NoteStore { S1 }` and `NoteInbox` +run with a recording `self`: they declare, nothing activates. Nothing throws, so the host commits to the new code. ++++ NoteStore: pure, swapped in place +`NoteStore` has no `self.open`. Its new methods and derives take over; `NoteList`, the drafts, its references and the +running import job stay. Routines restart, serialized: the new `cleanup` waits for the old handler in flight. The gate +opens. ++++ Running calls finish on old code +An `add` that was running when the swap happened finishes on the old code with its old handles; its transaction still +commits. Only new calls get the new handles. Exception (R15): if the reload raised a document's version, old code that +touches the migrated document throws, the document stays intact, and that call fails once. ++++ The import job keeps its old run +The job's `run` keeps the code it started with. When it settles, the **new** `finish` records the outcome. ++++ The views: pure, swapped +The three `Notes` views swap in place. Owner, Ann and Bob keep their references; the reader derive is recomputed with +the new sort and the difference is sent. Nobody sees a reconnect. ++++ NoteInbox: native, restarts +`NoteInbox` has a watcher. Its running work ends `aborted`, the watcher closes with reason `reload`, the new factory +runs, the watcher reopens. Its documents would stay. Whether a file dropped during the restart is seen is the +watcher's business: Studio's `watchDir` scans the directory once when it opens. +::: + +**Why natives restart instead of being handed over:** a native has no line, no transactions and no schema. If old and +new code shared it, they would interleave on it unchecked. In practice restarts are cheap: the Durable harness is base +code, terminal PTYs live in a separate terminal host process, and model runtimes, sandboxes and a search index rebuild +in milliseconds. Only reloading the plugin that owns SSH connections (running remote commands die) or the relay socket +(relayed connections drop) hurts. + +**Removing a type:** its objects end (routines stop, natives close, references end with an error); in-memory documents +go with them; persisted ones stay until retired. **Consumers elsewhere** see a restarting object like any object that +briefly goes away: `loading`, parked calls, then `live`. + +**UI facets reload too.** A reload unmounts and remounts components. That is why "leaving" (abort a login, close an +overlay) is never derived from an unmount (chapter VIII). + +## Across hosts + +A **revision** is (server incarnation, sequence) and names the complete desired code set: every facet with its +digest, not a batch of changes. Every install reconciles everything missing, and a host reports a revision only when it +runs that complete set. A revision also carries the code-set digest (a hash over the facet bundles' content hashes), +so a server restart with unchanged code triggers no reinstall. + +The order is **server, workers, clients**. Each host reports `installed`, `installed_failed` (with the objects that +failed) or `rejected` (with the reason; it stays on its previous code). A server that rejects stops the rollout. A +host that starts or reconnects installs the current revision before anything is routed to it. Installs are serialized +per host and bounded (60 s). + +Calls keep flowing during a rollout. When a call reaches a provider on another revision, the mismatch response +carries the provider's revision and status: + +| Provider | The call | +| --- | --- | +| ahead, or behind and installing | `updating`: values readable, calls park (with the park deadline) | +| rejected | fails at once; retried on the next revision | +| same revision | `contract_mismatch`: a real mismatch, final | + +A host checks its own install status first: if its own install was rejected, a call fails at once with the reason +instead of waiting on a provider. Calls parked on a binding that is released end with an error and are not resent. +There are no cross-host transactions and no votes. + +:::sequence animate | A tab calls into a worker that is still installing. The call parks and runs once the worker reports the revision. +actor tab "tab" "revision 8" +actor server "server" "revision 8" accent +actor worker "worker S1" "installing 8" blue +server => worker: install revision 8 +tab -> server: call Notes.add (new contract) +server -> worker: relay +worker --> server: updating (rev 7, installing) +server --> tab: updating +tab -> tab: park (deadline 60 s) +worker --> server: installed 8 +server --> tab: provider at 8 +tab -> server: call Notes.add (resent from park) +server -> worker: relay +worker --> tab: result +::: + +## Start barrier + +At host start and first load there is one more rule (chapter XIV): load every facet (the core facet with its start +data among them), wait until every facet and `lifetime: "host"` object other than the harness is live or failed, then +resume dependent work. Plugin reloads need no barrier. + +## Loading and the dev loop + +- **Roots:** plugins come from built-in, global (`/plugins`), package (npm or git, installed with + `--ignore-scripts`) and project roots, found by directory listing; no source parsing. The server selects which load. + Dependencies between plugins resolve by contract name at runtime. Names are prefixed: objects `.`, + tools `_`. +- **Loader D:** one base bundle per surface (host packages and the app runtime) plus one script per facet, with host + packages passed in (no module map). Old generations are reclaimed by the garbage collector; a stable filename per + (facet, digest), and V8's compilation cache off on hosts that reload in place. The loader is an adapter: static + imports (tests, Durable Objects) work the same, with the app passing the plugin name to `load`. +- **Dev loop:** a source watcher marks plugins dirty and never applies anything. A reload control (the playground's web + shell, a CLI, an agent tool such as p2's `reload_plugins`) reloads the changed plugins, or restarts all hosts for + base code. Reloads are always explicit. + +:::review | Review: project plugins (#14, deferred) +Per-directory plugins with trust prompts and snapshots (`apps/p2/docs/plugin-loading.md`, still the reference for +roots, trust and scopes) need their own pass over this design. Until then, treat the project root as designed but not +settled. +::: + +:::note | Built so far +Reload and rollout are layer L10, the last before telemetry. The frozen kernel has a reload path with host values and +a development freeze, both removed from the design. +::: + +# Changing Contracts {#contracts lede="How a contract gets a new version while old tabs, old workers and old plugins are still talking to it."} + +Chapter XV ended with a tab that calls into a worker on another revision. That case is short-lived: the rollout +finishes and both sides run the same code. Other cases last longer. A phone kept a tab open overnight with yesterday's +web bundle. A plugin from another author was built against `Notes` version 1 and is not updated yet. In both, a +consumer holds a contract that the provider no longer has. This chapter is about what the provider does then. It is +also the one part of Chord v2 that is built: everything in the sections on `evolve`, `checkAccepts` and fingerprints +runs today in `packages/chord/src/next/contracts/`. + +## What has a version + +Everything that is stored or crosses a boundary has a version. Each surface handles an old version its own way: + +| Surface | Version | When an old one shows up | Chapter | +| --- | --- | --- | --- | +| object type (contract) | `version` plus fingerprint | `evolve` and `checkAccepts` on the provider; otherwise `contract_mismatch` | this one | +| document | `version` plus schema fingerprint | pure `migrate` on access; a schema change without a bump fails the object `broken` | V | +| job | `version` | `accepts` on outcomes and `running`; `blocked` without an adapter | XI | +| registry | `version` | `accepts` adapts contributions; one without a match is left out and reported | XIII | +| native | none | the object restarts on reload | IX | +| Durable tasks and documents | Durable's `version` | Durable's `migrate` | XIV | +| link protocol | protocol version | refused; hosts update together, a web page reloads | XVII | +| UI components, Chord and app APIs | none yet | exposed as they are; an SDK will version them later | | + +The rule behind every row is the same as for documents: **every change bumps the version**, additive ones included. +Chord never guesses compatibility from the shape of two schemas. + +## A history, not a file of versions + +A contract history starts with a **base**: a full `remote()` or `local()` declaration with an explicit `version`. The +`Notes` contract from chapters II and III is a base at version 1. Studio now wants pinned notes: notes get a `pinned` +flag and editors get a `pin` method. Instead of editing the declaration, the plugin continues the history: + +```ts +// notes/contract.ts +import { bool, evolve, nul } from "@earendil-works/chord/next"; + +export const NoteV2 = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int(), pinned: bool() }); +export const PublicNoteV2 = obj({ id: str(), text: str(), at: int(), pinned: bool() }); + +export const NotesV2 = evolve(Notes, (e) => + e.state("editor", "notes", obj({ items: arr(NoteV2) })) + .state("reader", "notes", obj({ items: arr(PublicNoteV2) })) + .addMethod("editor", "pin", obj({ id: str(), pinned: bool() }), nul()), +); +``` + +`evolve(prev, (e) => …)` appends one version to `prev`'s history: `NotesV2.version` is 2, and `NotesV2` still knows +version 1 exactly as it was declared. Every version is **materialized and fingerprinted at declaration**. That is what +makes old code safe: an old plugin bundle that contains its own copy of `Notes` version 1 computes the same +fingerprint as the history inside `NotesV2`. There is no build step, no generated file and no fingerprint copied by +hand. + +The edits: + +| Edit | Changes | Kind | +| --- | --- | --- | +| `e.state(variant, member, schema)` | one state member of one variant | remote | +| `e.addState(variant, member, schema)` / `e.removeState(variant, member)` | adds or removes a state member | remote | +| `e.state(member, schema)`, `e.addState(member, schema)`, `e.removeState(member)` | the same, flat | local | +| `e.args(name, schema)` / `e.result(name, schema)` | one side of a method | both | +| `e.method(name, args, result)` | both sides of a method | both | +| `e.addMethod(variant, name, args, result)` | adds a method (or an existing one to another variant, same signature) | remote | +| `e.addMethod(name, args, result)` | adds a method | local | +| `e.removeMethod(name)` | removes a method from every variant | both | + +Edits chain. The `method(...)` wrapper appears only in base declarations; edits take the schemas directly, so a +method without a result is added with `nul()`. The rules of chapter III still hold across versions: a method has one +signature in every variant, and a member's kind never changes, not even after it was removed and added again. + +**What `evolve` cannot do:** change the input or the set of variants. Both change what an object *is* (its identity, +who gets which members), and no adapter can turn one into the other. That needs a new history base, below. + +## The provider says what it accepts + +`NotesV2` says what changed. It does not say how version 1 consumers are served; the **provider** decides that, with +mappings that live in its own code, never in the contract. The worker facet declares them with `checkAccepts`: + +```ts +// notes/worker.ts +import { checkAccepts } from "@earendil-works/chord/next"; + +const toV1 = ({ pinned, ...note }: Static) => note; +const toPublicV1 = ({ pinned, ...note }: Static) => note; + +const notesAccepts = checkAccepts(NotesV2, { + 1: { + state: { + editor: { notes: (current) => ({ items: current.items.map(toV1) }) }, + reader: { notes: (current) => ({ items: current.items.map(toPublicV1) }) }, + }, + methods: {}, // add and remove did not change; pin is new + }, +}); +// notesAccepts.versions: [{ version: 1, fingerprint: Notes.fingerprint }] + +host.serve(NotesV2, notesFactory, { accepts: notesAccepts }); // working name for the objects-layer option +``` + +`checkAccepts` is **pure**: it checks the mappings without calling them and returns only the version metadata of the +old versions it lists. A provider that lists nothing accepts only its current version. The objects layer later +compares a consumer's version and fingerprint against this list. + +The mappings point in the direction the data flows: + +| Member in the old version | Mapping | Direction | +| --- | --- | --- | +| state, changed | `(currentMember) => oldMember` | provider's current value to what the old consumer expects | +| state, removed | `(currentVariantState) => oldMember`, or `"refuse"` | receives the whole current state of that variant | +| method arguments, changed | `args: (old) => current` | old caller's arguments to current ones | +| method result, changed | `result: (current) => old` | current result back to the old shape | +| method, changed or removed | `{ call: async (oldArgs, ctx) => oldResult }`, or `"refuse"` | replaces the method for old callers | +| member added since | nothing | an old consumer never asks for it | + +A changed state member cannot be refused: the old consumer still has that member and must get a value. A removed one +can be computed from what the variant has now, or refused. `e.method(name, args, result)` replaces both sides and +therefore needs both mappings, even if one side stayed the same (then an identity mapping). + +**Required, at type level and again at runtime.** For every member that exists in the old version and changed or was +removed since, the mapping is required by the type of `checkAccepts`. Because plugins are bundled without type +checks, the same check runs at declaration: a missing mapping, a key for a member that needs none, an unknown version +or a wrong shape throws, naming the member's path (for example `accepts[1].state.reader.notes`). An absent `state` or +`methods` section counts as empty and may be left out only when nothing there needs a mapping. + +**Conservative with no-op edits.** An edit that sets a schema to what it already was, or changes a member and then +changes it back, still requires an identity mapping. TypeScript erases constraints such as `{ max: 2000 }` from the +static type, so two schemas that look the same to the compiler may differ; Chord does not try to tell. + +### Old versions map straight to the current one + +Studio later replaces `remove` with `archive`, which keeps the note in a trash list: + +```ts +export const NotesV3 = evolve(NotesV2, (e) => + e.removeMethod("remove").addMethod("editor", "archive", obj({ id: str() }), nul()), +); + +const notesAccepts = checkAccepts(NotesV3, { + 1: { + state: { editor: { notes: (c) => ({ items: c.items.map(toV1) }) }, reader: { notes: (c) => ({ items: c.items.map(toPublicV1) }) } }, + methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } }, + }, + 2: { methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } } }, +}); +``` + +Each entry maps **one old version directly to the current one**, not through the versions in between. Version 1 +needs the state mappings (its notes have no `pinned`) and the `remove` replacement; version 2 needs only the +`remove` replacement, and leaves `state` out because its state is unchanged. A member that an old version never had +needs nothing for that version, even if it was added, changed and removed later. Dropping support for version 1 is +deleting its entry: version 1 consumers then get `contract_mismatch`. + +## Binding an old consumer + +A hold carries only the consumer's contract **version and fingerprint**, never a description of the contract (R6). +The provider decides with its own schemas: + +:::flow wide animate | What a provider does with a hold. The decision uses only the provider's own history; a peer's schema is never an input. +row "The hold arrives" +box hold "hold" "version 1, fingerprint F1" blue +box cur "equal to current?" "version and fingerprint" +box acc "accepted old version?" "F1 from the provider's own history" +row "Outcome" accent +box live "bind" "current members" +box adapted "bind adapted" "through the mappings" accent +box mismatch "`contract_mismatch`" "or updating, during a rollout" muted +arrow hold -> cur +arrow cur -> live "yes" +arrow cur -> acc "no" +arrow acc -> adapted "yes" +arrow acc -> mismatch "no" +note "A mismatch fails only that consumer's binding. The provider object and its other bindings are not affected." +::: + +Once a version 1 tab is bound, it is served as if version 1 were current: + +:::steps | A version 1 tab on a version 3 worker ++++ Bind +The tab holds `Notes { S1 }` with version 1 and fingerprint F1. The worker's history computes F1 for version 1, and +`notesAccepts` lists version 1, so the hold binds adapted. The tab gets the `editor` variant as version 1 defined it: +`notes`, `add`, `remove`. It never sees `pin` or `archive`. ++++ State through the mapping +Chord serves the adapted state like a derive of the provider's state: it applies the mapping to the current value +and keeps the result per binding. The snapshot the tab receives has no `pinned` fields. ++++ A commit +The owner pins a note from a current tab. The worker sends the current tab the draft's ops. For the version 1 tab, +Chord applies the mapping to the new value and sends the difference to its last adapted value (`diffRevisions`). The +mapping produced nothing new for a pin, so the version 1 tab receives no change at all. A large list is never resent +whole on every commit. ++++ A call +The tab calls `remove({ id })`. Its arguments are checked against the provider's own version 1 schema, then the +`call` replacement runs `archiveNote` on version 3 code. Its `null` result is what version 1 expects. Had the method +only changed, `args` would have mapped the arguments forward and `result` the result back. ++++ The worker moves on +A later revision drops the version 1 entry. When the tab re-holds (or the worker restarts and the tab binds again), +the hold gets `contract_mismatch`, and Studio's panel shows "reload to update" instead of failing silently. +::: + +## A new history base + +Some changes cannot be adapted. Studio decides that notes belong to a project, not a session: the input changes from +`{ session }` to `{ project }`. That is a new base, written in full, with an explicit version that continues the +published numbering: + +```ts +export const NotesV4 = remote("notes", { + version: 4, // explicit: the published history used 1 to 3 + input: obj({ project: str() }), + placement: () => Server, + variants: { + editor: { state: { notes: obj({ items: arr(NoteV2) }) }, methods: { add: method(…), pin: method(…), archive: method(…) } }, + reader: { state: { notes: obj({ items: arr(PublicNoteV2) }) } }, + }, +}); +// checkAccepts(NotesV4).versions is empty +``` + +A new base has no history before its version and **cannot accept older versions**, not even for an additive change. +Every consumer built against versions 1 to 3 gets `contract_mismatch` and must update. Later `evolve` calls continue +from version 4. The same applies to a change of the variant set. Do not spread an older declaration into a new one or +copy its fingerprint; the contract objects are branded and `evolve` and `checkAccepts` reject copies. + +## Local contracts + +`local` contracts (chapter IV) evolve the same way, with a flat members map and no variants: + +```ts +export const SessionInfoV2 = evolve(SessionInfo, (e) => e.state("init", InitV2)); +checkAccepts(SessionInfoV2, { 1: { state: { init: (current) => toInitV1(current) }, methods: {} } }); +``` + +Same-host consumers of a `local` type are usually reloaded together with their provider, but not always: another +plugin on the same host may be built against an older version. The rules are identical; only the variant argument is +missing. + +## Implementing a contract + +`Implementation` is the type of everything a provider must supply for the current version: one state +getter per member and variant, and one asynchronous method per method, each receiving `(args, ctx)`: + +```ts +const impl: Implementation = { + state: { editor: { notes: () => list.value }, reader: { notes: () => publicList.value } }, + methods: { add: async ({ text }, ctx) => addNote(text, ctx), remove: async ({ id }) => removeNote(id), pin: async (a) => pin(a) }, +}; +``` + +Keep declaration types **inferred** (no annotation on `NotesV2`): the inferred type carries the complete history, and +`checkAccepts` needs it to know what changed. + +:::review | Review: L1's implementation shape and the factory's return value +`Implementation` in `src/next/contracts/types.ts` is `{ state: { variant: { member: getter } }, methods }`. The +design's factories (`design.md` §6, used throughout this guide) return `{ variant, member: value, method: fn }`. +The objects layer (L4) has to connect the two, and also define how `serve` receives the checked accepts (this guide +uses the working name `{ accepts }` as a third argument) and what `"refuse"` does at runtime. The natural reading, +used in the stepper above, is that a refused method call and a hold that needs a refused state member end with +`contract_mismatch`; no source says so. +::: + +## The fingerprint + +The fingerprint is a 64-bit hash (`hash64`, portable across runtimes) of a **normal form** of the contract's +schemas. The normal form decides what counts as a change: + +| Ignored | Significant | +| --- | --- | +| order of object keys, `required` and `enum` (sorted) | order of union and intersection branches (`anyOf`, `oneOf`, `allOf`) and of tuples | +| annotations on schema nodes: `title`, `description`, `examples`, `$comment`, `deprecated` | defaults, formats, patterns, bounds, `$ref` names | +| placement | input, variants, member kinds and every member schema | + +Union order is significant because it is meaning: TypeBox picks defaults and decodes by the first matching branch. +Annotations are dropped only on schema nodes; an object property that happens to be named `title` is data and counts. + +Only data schemas are allowed. One walker copies, validates and normalizes each schema against a **whitelist of +TypeBox keywords**: primitives, arrays and tuples, objects and records, unions and intersections, `$defs` and `$ref` +(use `$ref` for recursive types), constraints and annotations. Conditional and dependent schemas, `not`, +`definitions`, `contentSchema`, `unevaluated*`, `contains`, `propertyNames`, unknown keywords, executable or custom +TypeBox features, arbitrary `Unsafe` and refinements, accessors, symbols and cyclic objects are rejected at +declaration. Bytes are the one special leaf: `bytes({ min, max })` checks a `Uint8Array` and has the data marker +`{ type: "bytes", minLength?, maxLength? }`, so it hashes like any schema; TypeBox's checking still works on it. + +Schemas are deep-copied into a private, frozen history; your own schema objects are copied, not frozen. +`NORMAL_FORM_VERSION` is part of every digest, so a change of the normal form is a link protocol change. + +**Not a security boundary.** A fingerprint says "we were compiled against the same contract"; it proves nothing. A +hostile client can send any fingerprint it likes. Providers therefore always validate arguments with their own +schemas (chapter IV), whatever the fingerprint says. + +## Rules of thumb + +- Treat a published contract like a published API: every change is `evolve`, every `evolve` needs a decision about + old consumers. +- Keep mappings small and pure. A state mapping runs for every commit and every adapted binding; make it a map over + the value, not a query. +- Accept the versions your users still run (one or two back during a normal rollout), and delete entries when the + metrics show no old bindings. +- Change the input or the variants only with a new base, and only when updating every consumer is acceptable. +- Contract, document and job versions are separate and each is decided on its own. Pinned notes needed `NotesV2` + for the wire and `NoteList` version 2 with a `migrate` that adds `pinned: false` (chapter V), but no new + `ImportJob` version, because its outcome did not change. + +:::note | Built so far +This chapter's contract layer is built: `packages/chord/src/next/contracts/` (commit `7d2bb8ea8`, about 1,300 lines +with `remote`, `local`, `method`, `evolve`, `checkAccepts`, `Implementation`, the schema helpers, `fingerprintOf`, +`schemaNormalForm` and `NORMAL_FORM_VERSION`), with nine test files covering evolution, fingerprints, the hash, +local contracts, long chains, schemas, types and review findings. It stands alone: it hosts nothing, calls no +mapping and does not import the frozen runtime. Binding by version and fingerprint, adapted state and calls belong to +the objects and protocol layers, not built yet. +::: + +# On the Wire {#wire lede="What travels over a link, what a relay does with it, how big things are kept in check, what each hop costs, and how to see it in traces."} + +Most plugin code never thinks about the wire: it declares state, commits transactions and calls methods, and Chord +moves the bytes. You need this chapter when something is slow, when a value is large, when a link misbehaves, or when +you want to read a trace. It also explains why the earlier chapters insisted on small operations, shared structure and +no callbacks: each of those rules removes a cost on some hop. + +## What a state member sends + +Every exposed document, derive or readable is a **state member**. Over a hold it produces one of these: + +| Update | When | +| --- | --- | +| `snap` | on hold and on re-hold: the whole value, once | +| `ops` | per commit: the draft's recorded ops, no diff | +| `ops` from a diff | per change of a derive (and of an adapted binding, chapter XVI) | +| throttled `ops` | job progress | +| `reset` | a migration commit replaced the value (adapted for older consumers) | +| `error` | the member failed | + +**Commits are atomic for consumers.** On the same host, subscribers are notified once, after every document of the +commit is adopted. Remotely, one `state` message per hold carries a batch of member updates that is applied before +listeners run, and a commit that touches several holds on one link travels as **one** `state` message on that link. +A consumer never sees half a commit. + +## Messages + +A message is a **header plus opaque parts** (R1). There is one part per hold the message concerns; the header says +which hold each part belongs to and how large it is. + +| Direction | Kind | Parts | +| --- | --- | --- | +| consumer to provider | `hold`, `release`, `call`, `cancel` | `call` has one (the arguments) | +| provider to consumer | `bound` | one: the snapshot | +| | `state` | one per hold the commit touched on this link | +| | `result` | one | +| | `gone` | none | + +There are no request ids in application code. Hold ids and call ids exist only inside the protocol, per link. + +**One binary format on every link (Q4):** `u32 header length | header JSON | part bytes`, as one `Uint8Array`. A +WebSocket sends it as a binary frame, IPC passes it with `serialization: "advanced"`, a memory link delivers it +asynchronously. Encoding, framing, header checks, id resolution, flow control and size limits are one shared protocol +layer; link adapters only move bytes. Tests on memory links therefore run the production path. + +A part is encoded **once, at its origin**, with that hold's own Delta encoder (paths are interned per hold, end to +end between provider and consumer). A JSON part above a threshold (about 16 KiB) is compressed once at its origin, +whatever the link; byte segments stay raw; relays forward compressed parts as they are; the header names the codec. + +**The header is a TypeBox schema (R4)**, a union tagged by `kind`; the protocol types derive from it. Headers from +untrusted links (client WebSockets) are checked with the schema compiled once; trusted links (IPC between server and +workers) are not checked. **Unknown ids are ignored on both sides (Q1).** They are normal races: a `result` crossing a +`cancel`, a `call` crossing a `gone`, a state part crossing a `release`. The side that ended a hold or call settles +its own pending work: a consumer that receives `gone(h)` rejects its pending calls on `h`, and the order per +direction guarantees nothing else for `h` follows. Only a header that fails the schema is a protocol error, and it +closes that link. + +The link protocol has a version. Hosts with different protocol versions refuse each other; a web page reloads to get +the server's protocol. + +## Through a relay + +A tab never talks to a worker directly (chapter I). The server **relays**, and a relay reads and rewrites only +headers: it maps hold and call ids between its two links, stamps the principal on `hold` and `call` (a client's +principal field is always overwritten; only a trusted parent link may carry one), and groups parts per destination +link, so each link gets only its own holds' parts. It never parses a payload. + +:::sequence animate | The owner adds a note while Ann watches. The commit leaves the worker as one `state` message with two parts; the server splits it by destination without decoding either part. +actor owner "owner's tab" "hold h1" blue +actor ann "Ann's tab" "hold h2" muted +actor server "server" "relay" accent +actor worker "worker S1" "NoteStore, Notes" +owner -> server: call c7 Notes.add (1 part) +server -> server: map ids, stamp owner +server => worker: call c3 (part forwarded as is) +worker -> worker: method runs, self.tx commits +worker => server: state: parts for h1', h2' +server -> server: group parts per link +server --> owner: state (h1 part) +server --> ann: state (h2 part) +worker --> server: result c3 +server --> owner: result c7 +::: + +The `state` message reaches the owner's tab before the `result`, so when `await notes.add(…)` resolves, the new note +is already in `notes.value`. Ann's view is a different variant (reader) and gets its own part from its own derive; the +relay neither knows nor cares. + +## Flow control + +Each link has **one byte window**, counted from each message's `byteLength` (parts are already encoded bytes; nothing +is measured or stringified to count it). The window covers every message, first snapshots included. + +- **Snapshots are always admitted (R19)**, even when larger than the window; they use it up. Whether a hold + **overflows** is decided on the changes queued after the snapshot. +- **Under backpressure** a commit's ops are collected per hold and encoded into one part when credits return. The + consumer still gets the commit atomically. +- **After an overflow** the hold is dropped and established again once the previous snapshot's debt has drained + (credits back at the full window, Q3), and not more often than a minimum interval. So there is at most one snapshot + in flight per hold: no re-hold storm, and no stall for a snapshot larger than the window. +- **Calls** are throttled by call credits the provider grants (a generous window). Beyond them calls wait locally. + Nothing is refused for being "too many", and holds are not limited by count; authorization and input validation + protect them. +- **Byte-heavy traffic** such as terminals uses separate channels (chapter XIII), so a large state message never + holds up keystrokes. + +**Recovery:** after a link drops, the consumer re-holds every hold over the new link and receives snapshots; calls in +flight fail as on any disconnect, calls made meanwhile park. A relay whose window overflows releases the hold and +re-holds; the provider sends a fresh snapshot. After a routed link drops, Chord re-resolves the address (with the +per-address backoff) while holds on it exist, so a passive viewer recovers without anyone pressing anything. + +## Size limits + +Limits apply **per part** (R2, Q2), both to the compressed and the decoded size, and are checked from the sizes in the +header before anything is decoded; zlib's `maxOutputLength` enforces the decoded limit in the one decode function. + +| Situation | What fails | +| --- | --- | +| a part over the limit | only its own hold or call, with `too_large`; the other parts of the message apply | +| a snapshot over the limit | its hold ends `failed`, with no automatic re-hold | +| a WebSocket frame over `maxPayload` (e.g. 64 MiB) | the connection closes (code 1009); this is the transport's hard cap | + +Defaults for client WebSockets are 16 MiB compressed and 64 MiB decoded per part, configurable per link kind. The +decoded limit travels with relayed work: the server stamps the admitting link's limit on the header at admission +(never taken from the client), and the host that decodes enforces it. IPC therefore needs no limit of its own. + +Large data is never a large state member or a large argument. Files go through chunked transfers or the HTTP +component (chapter XIII); long lists are keyed documents or bounded windows (chapter VIII). + +## What each hop costs + +Chord v2's goal is as little compute and memory as possible while staying correct: **O(size of the change)** per +commit and per viewer, independent of the size of the value. A 500 KB value is touched as a whole only when a new +viewer takes its first snapshot. Delta, the value library under documents, was built for this, and its own primitives +are almost free: `diffRevisions` on shared subtrees takes 0.018 ms, `applyImmutable` of a small change 0.004 ms. + +| Hop | Work | +| --- | --- | +| commit | Delta's `prepare` (copies and checks only the values placed in this commit), adopt by pointer swap; storage writes the delta | +| reads in a transaction | the draft overlay, no copy; an unchanged draft writes nothing | +| subscribers on the same host | the same value and ops, by reference | +| derives | compute plus `diffRevisions`; unchanged parts stay the same objects (`mapShared`, chapter VIII) | +| send | one encoding per hold part at its origin, size taken from the encoded bytes | +| relay | header only: ids mapped, principal stamped, parts grouped; parts never parsed | +| receive | compiled header schema on untrusted links; arguments stripped and validated once at the provider (compiled schema); inputs normalized | +| apply | `applyImmutable`, structure shared | +| new hold | the value encoded once (unavoidable) | +| memory | one authoritative value per document per host; replicas and derived values share structure; one cached value per adapted binding | + +These are targets. The frozen kernel measured far worse, which is why the design is so strict here: + +:::stats +- 35 ms | one small change to a 500 KB value in the frozen kernel, no viewer +- +13 ms | per viewer on top (10 viewers: 130 ms) +- 17 / 33 / 101 ms | a tiny SQLite commit after 0 / 10 / 50 earlier deltas +::: + +Nearly all of that came from layers around Delta: whole-value validation on every access, whole-value freezing, +JSON-ness walks on every received frame, double stringification to measure a frame, snapshots cloned per viewer, and +a SQLite storage that replayed history, and schemas rebuilt for every check (a compiled TypeBox check takes +0.012 ms, an interpreted one 5.6 ms). The redesign removes each of them (chapters V, VIII and this one); the +before/after per hop is in `apps/p2/docs/p3-reviews/v2-cost-model.md`. + +:::review | Review: v2-cost-model.md is partly outdated +The measurements and the per-hop "after" targets in `apps/p2/docs/p3-reviews/v2-cost-model.md` are current. Its +closing section "Decisions this needs" is not: items 6 and 7 (validation by the caller's trust; contract descriptions +once per link and fingerprint) were replaced by R7 (validate for every caller) and R6 (no descriptions on the wire), +and §10 still describes the description fetch. +::: + +### Cost guards + +Chord reports **full-value walks, deep copies, encodes, schema validations and materializations** as ordinary +telemetry metrics, always on; with the NOOP adapter they cost an empty call where such work happens. Tests use the +in-memory adapter and assert them per hop. The reference case: one appended note to a 500 KB document with 10 viewers +does no walks, no copies, one encode per hold part at its origin and none at relays, and no validations. A +`vitest bench` suite measures the hops. + +### Ways to make it slow anyway + +| Mistake | Why it costs | Instead | +| --- | --- | --- | +| a derive that builds new objects for unchanged data | every commit diffs and sends O(value) | keep unchanged items identical: `mapShared`, `previous` (chapter VIII) | +| one long array that changes often (R24) | sharing is per path; each change copies one slot per element and visits each | keyed documents or a bounded window | +| bytes as base64 strings | inflated, parsed, compressed as text | `bytes()` leaves; chunked transfers for files | +| awaiting I/O inside `self.tx` | the object's line is blocked for every caller | do the I/O first, then a short transaction | +| storing registry entries or references in documents | stale after a reload, not data | look up the current value at use | +| opening natives at module level or in a facet | no owner: never closed, never restarted | `self.open` in an object (chapter IX) | +| callbacks as arguments | cannot cross a link, keep old code alive | registries, jobs, flows | +| one big object that everybody views | one large snapshot per viewer and per reconnect | split by viewer need; views of the same variant share one value | + +## Telemetry + +Chord reports traces, events and metrics through `@earendil-works/pi-telemetry`. Each host gets **one adapter**, +`createHost({ telemetry })`, NOOP by default; the adapter decides sampling, buffering, export and filtering. Only hosts +hold the adapter, to start root spans. Code sees spans: + +- `ctx.telemetry` is the current span (`Ctx` extends `Context`, so it is the same field Durable and pi-ai read). Every + span carries its adapter's metrics and propagation, so code with a `ctx` can record a metric or inject a carrier. +- `ctx.trace(name, attributes, fn)` runs `fn` in a child span with a child `ctx`. +- App values such as the principal stay out of Durable's context. + +**Across links**, `call` and `hold` carry the caller's trace carrier as a bounded header field (R5: limited key count +and string lengths); the provider runs the method under it, and each relay adds a `chord.relay` span. The call from the +sequence above, as a trace: + +```text +chord.call Notes.add [owner's tab] +└─ chord.relay [server] + └─ chord.method Notes.add [session:S1] principal=owner + · event chord.commit docs=2 ops=5 +``` + +Adapters trust typed input: no validation of malformed values and no try/catch around recording. Only carriers from +another process are checked (keys present, strings), and a bad one falls back to a new root. + +**Shared work** (one activation serving several `get`s, a worker start) is its own root span, linked to the trigger +that started it; each waiting span links to it. Routine runs, job runs, eviction, backoff retries and background +closes are root spans too, linked to their trigger where there is one. + +| Tier | What | +| --- | --- | +| spans | calls, methods, relays, holds, activations, opens, closes, loads, revocations, resolves, routine runs, job runs | +| events | commits inside methods, retries, parking | +| metrics | state messages, WebSocket bytes, commit durations, in-flight calls, active holds, open links, park durations, backpressure waits, the cost guards | + +Ids are attributes of spans, never of metrics. **Nothing is measured or walked for telemetry**; only sizes the code +already has are reported. Errors carry a fixed category and a message cut to 500 characters; cancelling is not an +error. Chord ships a typed telemetry schema, as pi-ai (`AI_TELEMETRY_SCHEMA`) and Durable +(`DURABLE_TELEMETRY_SCHEMA`) do. + +**Storing and reading:** `@earendil-works/chord/telemetry-sqlite` over `chord/sqlite` (Node and Cloudflare) stores one +row per settled span (events and links inline) and metrics aggregated per series per 10 s, flushes in batches every +second, keeps 7 days, and drops (and counts) beyond a bounded queue. The read interface `TelemetryStore` is shared: +`findTraces` (by name, attributes, errors, time window, with limit and cursor) returns complete traces, `getTrace` +one, `findMetrics` metrics. One renderer, `pi-telemetry/render`, prints traces as above; it prints links and never +follows them. The playground's trace viewer reads the same store. + +**Durable** already emits submit and phase spans, stores carriers on submissions and tasks (`TaskOptions.trace`), +and pi-ai emits `pi.ai.request` and `pi.ai.poll`. With the harness given the host's adapter (chapter XIV), an agent +tool that calls `Notes.add` through `chord(fn, context)` shows up as one tree from the model request down to the +commit. + +:::note | Built so far +Nothing of the protocol is built in v2: messages, relays, flow control and limits are layers L6 and L7, telemetry in +Chord is L11. Built in pi: `pi-telemetry` (propagation, links, metrics, the store interface, the renderer), +`chord/context` with `Context.telemetry`, pi-ai and Durable telemetry, and `chord/telemetry-sqlite`. Delta is built; +`mapShared` and bytes as atomic leaves in `diffRevisions` are still to do. The frozen kernel sends descriptions on the +wire and closes a link on an unknown id; both are replaced. +::: + +# Building p2: the Core {#p2-core lede="From a list of what the app must do to a set of owners: the base code every p3 plugin builds on."} + +The notes plugin used every Chord concept. A real app is the same concepts many times over. This chapter and the +next two assemble **p3**, the Chord v2 version of p2, from the parts the guide introduced. p3 is built next to p2 in +`apps/p3`, fully isolated with its own data directory, and **without migrating p2's data** (no import of +`catalog.sqlite`, `shares.sqlite` or the subagents table). Its plugins are derived from what the app must do, not +ported one to one from p2's plugin layout. + +:::review | Review: p3 is not designed yet +The design fixes the rules (`design.md` §25) and lists the p3 decisions still open (#17, #18, #20 to #23); the +per-domain design of p3 has not started. What these three chapters show is one construction that follows the rules, +derived from the functionality map in `apps/p2/docs/p3-functions/`. Object names are working names. Where a domain +has an open decision, the text says which. +::: + +## Start from functions, not from plugins + +The functionality map in `apps/p2/docs/p3-functions/` lists what p2 does, in plain words, with four facts per +function: the data and its size, how durable it is, who may see or do it, and where it must be available. It groups +the functions into 22 domains: + +| # | Domain | # | Domain | +| --- | --- | --- | --- | +| 1 | sessions | 12 | access | +| 2 | session workers | 13 | models, credentials, settings | +| 3 | conversations and agent | 14 | prompt resources | +| 4 | shell commands | 15 | status and presence | +| 5 | chat mode | 16 | search | +| 6 | agents (subagents, one-shots, peer messages) | 17 | remote access and forwards | +| 7 | code mode | 18 | logs and notices | +| 8 | session files | 19 | dev and reload | +| 9 | environments and execution | 20 | web shell | +| 10 | workspace | 21 | TUI shell | +| 11 | terminals | 22 | transport | + +The four facts answer the four questions every Chord object needs, in order: + +:::flow wide animate | From a function in the map to code. Each fact in the map answers one question Chord needs answered. +row "The map says" +box data "data" "what, how big" +box dur "durability" "ephemeral · until restart · persisted · transcript-bound" +box who "who" "owner · guest read · guest write · worker" +box where "where" "server · worker · web · TUI · environment" +row "Chord needs" accent +box owner "the owner" "smallest object it belongs to" accent +box store "where it is kept" "Durable, persisted or in-memory document, tab" +box variant "variants" "and revocation" blue +box host "host and code" "placement; core or plugin" +arrow data -> owner +arrow dur -> store +arrow who -> variant +arrow where -> host +note "Durability decides storage: transcript-bound data is Durable's; persisted app data is a persisted document with a stable identity; until-restart data is an in-memory document or a native; tab-scoped UI state stays in the client." +::: + +**One owner per thing, the smallest object it belongs to.** A guest's grant belongs to the access owner, not to the +session; a draft belongs to one (conversation, author) pair, not to the chat plugin as a whole. Ownership nests, and +rules set at creation are written in the creating commit (Durable's `createConversation({ init })`). + +**Core or plugin.** Core owners are what the app always needs and everything builds on; they are always present, so +other code may depend on them directly. Plugin owners may be missing; anything that depends on an optional feature +does so through a registry it owns itself (chapter XIII). The risk to avoid is a **core god object**: each core owner +owns only what belongs to it. + +## The ownership map + +The result of that procedure for p3, as a starting point: + +| Domain | Owner object(s) | Host | Code | +| --- | --- | --- | --- | +| access | `Access` (grants, devices, share links), `authenticate` | server | core | +| sessions | `Catalog` (session records), lifecycle jobs | server | core | +| session workers | the supervisor (`route(Session)` + idle stop) | server | core (app code) | +| session location | `SessionLocation { session }` | worker | core | +| conversations and agent | `Harness`, `ConversationStore`, `Conversation` views | worker | core | +| status and presence | `SessionStatus` (server copy), reporters on workers | both | core | +| shell commands | `ShellJob` in the chat plugin | worker | plugin | +| chat mode | chat plugin | worker | plugin | +| agents | subagents plugin (Durable child conversations) | worker | plugin | +| code mode | code mode plugin (sandbox native) | worker | plugin | +| session files | `SessionFiles { session }` (proposed, #17) | server | core or plugin, open | +| environments | envs plugin (config, SSH pools #20) | server and worker | plugin | +| workspace | workspace plugin | worker | plugin | +| terminals | terminal host process (#23), terminals plugin | separate process | plugin | +| models, credentials | models plugin, login job + flow | server | plugin | +| settings | `Settings` (single writer, #21) | server | core | +| prompt resources | prompt plugin (watchers in the session's environment) | worker | plugin | +| search | search plugin (index on the server, feed on workers) | both | plugin | +| remote access, forwards | radius plugin, ports plugin | server | plugin | +| logs and notices | `Notices` (one channel), log capture | server | core | +| dev and reload | the reload control, `reload_plugins` tool | server | plugin | +| web and TUI shells | UI facets over the objects above | clients | shells | + +The rest of this chapter builds the core rows. Chapters XIX and XX build the plugins. + +## The base code + +```text +apps/p3/ + src/ + hosts.ts Server, Session host kinds + principal.ts P, Revocation, principalKey, revokedBy, can() + server/main.ts createHost, load, route(Session), WebSocket and Radius entry points, supervisor + worker/main.ts createHost with the start data, connect to the parent + core/ the core facets: Access, Catalog, SessionStatus, Settings, Notices (server); + SessionInfo, SessionLocation, Harness, ConversationStore (worker) + plugins/ chat, subagents, codemode, envs, workspace, terminals, models, login, + prompt, search, ports, radius, files, dev +``` + +`hosts.ts`, `principal.ts`, the `SessionInfo` start data and the `Harness` object are the ones the guide already +built for Studio (chapters I, III, IX and XIV). Base code restarts the hosts when it changes; everything in +`plugins/` reloads. + +## Who is calling: authentication + +Chord stamps principals; the app produces them. p3's server has four ways in, each turned into a principal by +`authenticate(req)` when a WebSocket connects (chapter III): + +| Way in | Principal | +| --- | --- | +| loopback without a cookie (TUI, CLI, first page load) | `{ kind: "owner", access: "local" }` | +| loopback owner browser cookie (a per-port secret file) | `{ kind: "owner", access: "local" }` | +| a device cookie (HMAC-signed, per port, 30 days), obtained with a one-time sign-in token | `{ kind: "owner", access: "device", device }` | +| a guest cookie from a share link | `{ kind: "guest", session, mode, grant, revision }` | +| the Radius relay channel | `{ kind: "owner", access: "relay" }` | + +The checks that are not Chord's stay in this function: origin checks and Fetch Metadata (a browser request from +another site gets no principal, except a top-level navigation), proxy trust, the sign-in token rules (10 minutes, +single use, at most 32 outstanding, memory only), cookie expiry and revocation lists. A failed check closes the socket +before Chord sees it. + +**One permission matrix.** Every function in the map names who may use it, and p2 checks those rules in several +places. p3 writes them once, in `principal.ts`, and every factory and `authorize` uses it: + +```ts +// src/principal.ts +type Action = "read" | "send" | "upload" | "stop" | "workspace" | "terminal" | "manage"; + +export function can(p: P, action: Action, session?: string): boolean { + if (p.kind === "owner" || p.kind === "server") return true; + if (p.kind === "worker") return action !== "manage" && p.session === session; + if (p.kind !== "guest" || p.session !== session) return false; + if (action === "terminal" || action === "manage") return false; // terminals: owner only + if (p.mode === "read") return action === "read"; // no workspace for guest read + return true; // guest write: send, upload, stop, workspace +} +``` + +Factories pick variants from it (`can(p, "send", session) ? "writer" : "reader"`), so the matrix is visible in one +file and the members a caller lacks are simply not there. + +## Access: grants, devices, share links + +`Access` is the core owner of everything that lets someone in: share links per session and mode, guest records, the +device list and the signing key. It lives on the server as the chapter IV pattern: a `local` store that persists the +documents (only stable identities may), and a `remote` view for the owner's settings screen: + +```ts +// src/core/access.ts (server, in the core server facet) +const Links = defineState("access.links", { schema: map(ShareLink), version: 1, persist: true, + initial: () => ({}), checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64 }); +const Guests = defineState("access.guests", { … }); // same shape: map(Guest), persisted +const Devices = defineState("access.devices", { … }); // map(Device), persisted + +host.serve(AccessStore, (_input, self) => { // local: may persist, acts for the host + const links = self.state(Links), guests = self.state(Guests), devices = self.state(Devices); + return { + links, guests, devices, + removeGuest: async ({ grant }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(guests))[grant].blocked = true; }, ctx); + host.revoke({ grant }); // live access ends now + }, + resetLink: async ({ session, mode }, ctx) => { + const blocked = await self.tx((tx) => rotateLink(tx, links, guests, session, mode), ctx); + for (const grant of blocked) host.revoke({ grant }); + }, + signOut: async ({ device }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(devices))[device].revoked = true; }, ctx); + host.revoke({ device }); + }, + … // shareLink, join, signOutAll + }; +}); + +host.serve(Access, (_input, principal, self) => { // remote view: owner only + if (!can(principal, "manage")) throw forbidden(); + const store = self.get(AccessStore, {}); + return { + variant: "owner", + guests: self.derive("guests", [store.guests], ([g]) => publicGuests(g)), // names, modes; no tokens + removeGuest: (args, ctx) => store.removeGuest(args, ctx), + resetLink: (args, ctx) => store.resetLink(args, ctx), + signOut: (args, ctx) => store.signOut(args, ctx), + }; +}); +``` + +`authenticate` is base code on the same host and checks `AccessStore`'s current documents on every new connection +(it holds a reference to the store, taken with `server.root` at start). Live access and lasting denial stay separate, as chapter VII explained: + +| Event | Lasting effect (data) | Live effect | +| --- | --- | --- | +| remove a guest | guest record blocked | `revoke({ grant })` | +| reset a share link | new token; its guests' grants blocked | `revoke({ grant })` for each | +| archive a session | grants of that session void while archived | `revoke({ session })` | +| restore a session | grants valid again; revision bumped | none: guests reconnect | +| sign out a device | device revoked until its expiry | `revoke({ device })` | +| sign out all devices | signing key rotated, tokens cleared | `revoke({ device })` for each | + +A guest who rejoins with the same browser keeps its identity; a removed guest cannot mint a new one, because the +block is in `Guests` and `authenticate` refuses it. Names are unverified, cleaned, at most 40 characters, unique per +session ("Ann (2)"). + +## The catalog + +`Catalog` owns the session records: id, title (and whether it was edited), location, model default, pinned, +archived, timestamps. It lives on the server, persisted, and it **never starts a worker**: listing sessions, showing +their status or their unread marks reads server-side copies only. + +```ts +export const Catalog = remote("p3.catalog", { + version: 1, + input: obj({}), + placement: () => Server(), + variants: { + owner: { + state: { sessions: obj({ items: arr(SessionRow) }) }, + methods: { + create: method(CreateArgs, str()), rename: method(RenameArgs), pin: method(PinArgs), + archive: method(obj({ id: str() })), restore: method(obj({ id: str() })), + }, + }, + guest: { state: { sessions: obj({ items: arr(SessionRow) }) } }, // only the guest's own session, if not archived + }, +}); +``` + +Behind it sits a persisted `local` `CatalogStore`; the guest variant is a derive over it, as in chapter IV. The lifecycle operations (archive, clone, fork, +relocate, delete) are chapter XIX; here only the rule that keeps the catalog small: **the catalog does not know which +plugins exist.** Archiving a session writes `archived: true` and revokes the session; every plugin that keeps +per-session things reconciles from `catalog.sessions` with a routine (chapter X), as the notes plugin did. The ports +plugin closes forwards of archived sessions that way, also after a crash. + +## On the worker: start data, location, harness + +Three core objects exist on every worker from its start: + +- **`SessionInfo`** (chapter IX): the start message as `constant(init)`, with the session id, data directory and the + location the server chose at start. +- **`SessionLocation { session }`** (working name), lifetime host: the current cwd and environment, with a location + revision. Relocation changes it while the worker runs (chapter XIX); natives that depend on the location follow it + with a routine and a generation (chapter IX), and conversations that follow the session's cwd are listed in the core + per-conversation policy document. +- **`Harness`** (chapter XIV): the Durable harness for the session's store, one SQLite database per session, opened + under an exclusive lock keyed by the real data directory and session id. It refuses to start when the saved root + model is not configured, and resumes interrupted Durable work on start. + +`ConversationStore` and the `Conversation` views in front of it are chapter XIX. + +## Workers: start, keep alive, stop + +Starting, supervising and stopping workers is app code (chapter I); Chord only routes and counts. p3's supervisor is +`server.route(Session, { authorize, resolve })` plus a small server object that keeps per-session worker state: + +:::steps | A worker's life, as the supervisor sees it ++++ Authorize, then start +A tab gets `Notes { S1 }`. `authorize` checks `can(p, "read", "S1")` and that the session exists and is not archived +(an archived session refuses, except an index rebuild that starts the worker without resuming work). `resolve` forks +`worker/main.ts` with the start data from the catalog, bounded by a 60 s readiness timeout. ++++ A start failure sticks +If the worker fails to start (an unconfigured environment, a missing model), the error is kept per session and +every `get` fails at once with it, until the owner presses Retry. That clears the error and the related notices. +Guests see that the session is unavailable, not the error text. ++++ Counting +The core facet on the worker reports `host.activity` (`{ references, calls, jobs, lingering }`) and the harness's +running Durable task count to the supervisor object, from a routine whenever either changes. The call carries +`ctx.incarnation`, so a report from an older worker incarnation is ignored (chapter X). ++++ Idle stop +When everything has been zero for five minutes (checked every second, as in p2), the supervisor stops the worker: +the worker closes its host (routines stop, natives close, storage flushes) and exits. Durable work stays persisted +and resumes on the next start. Detached work that must keep the worker alive (a shell command whose caller left) is +a job, so it counts. ++++ Crash +If the worker exits on its own, the link drops. The supervisor records the exit code and a redacted stderr tail +(at most 4,000 characters) for that incarnation and posts one notice. References that still exist make Chord call +`resolve` again with backoff; a passive viewer recovers without a click. ++++ Shutdown +On server shutdown the supervisor stops every worker, with a forced exit after three seconds. Durable work remains +persisted. +::: + +## Status without a worker + +The session list shows per session whether it is running, busy (which conversations), its task count, usage and cost, +and an error. None of that may start a worker. The pattern is chapter X's pub/sub: a reporter on each worker (a +routine over the harness's state) calls `SessionStatus.report(summary)` on the server; `SessionStatus` keeps the latest +value per session and ignores calls whose `ctx.incarnation` is older than the newest it has seen; clients read +`SessionStatus`, which lives on the server. Children lists, unread marks, search index rows and artifact lists use the +same pattern (chapter XIX). + +## Settings and notices + +**Settings** (default model and thinking, run settings, compaction, queue modes, code mode switch, owner name) have +**one writer**: a core `Settings` object on the server owns the file, validates writes and serializes them; every +other process follows its state through a reference instead of watching and rewriting the file itself. p2 watches +settings, models and auth in every process and writes them from two, which is decision #21. + +**Notices** are one channel: one core `Notices` object on the server, written by everything that fails or finishes +(worker crashes, reload results, model errors, login outcomes, grants), keeping one latest notice per source, at most +50. The owner's shell reads one list; dismissing is a method. p2 has five notice channels for the same thing. + +:::review | Review: settings and file formats (#21, #22) +Single writer for settings, models and auth (#21) and versioning on-disk formats such as `envs.json` like documents +(#22) are open p3 decisions in `design.md` §26. The construction above assumes "one owner on the server, others +follow its state" for #21. +::: + +:::note | Built so far +Nothing of p3 exists. The functionality map is complete (`apps/p2/docs/p3-functions/`, five area files plus the +condensed `README.md`), and the design's app rules (`design.md` §25) are decided. p3 starts after the layered rebuild +and the playground's first real proof. +::: + +# Sessions and Conversations {#p2-sessions lede="The session lifecycle, status without a worker, chat, shell commands, subagents and search, each mapped to an owner and the Chord parts it uses."} + +This chapter builds the domains a person touches most: the session list and its operations, the conversation view +and composer, shell commands, chat mode, subagents, messages between sessions, search and unread marks. Each section +names the owner, the host, and the Chord parts it uses; code appears only where it shows something new. + +## The session lifecycle + +The catalog (chapter XVIII) owns the session records. Its operations, by the Chord part each needs: + +| Operation | Owner and form | Notes | +| --- | --- | --- | +| create | `Catalog.create`, a method | absolute directory, `~` expanded, must be a directory; on an environment, resolved there through the envs plugin; refuses an unknown model; model and thinking default to the last used | +| rename, pin | methods on `Catalog` | title 1 to 120 characters, whitespace collapsed; an explicit title is never replaced automatically | +| automatic title | a routine on the search feed | the first user message, line breaks to spaces, 100 characters | +| agent rename | an agent tool on the worker calling `Catalog` | root conversation only | +| archive, restore | methods; revocation; reconciling routines | below | +| clone, fork, relocate | jobs on the catalog store | outlive the call, cancellable, one outcome | +| delete | a job; decision #18 | below | + +### Archive and restore + +Archiving must stop the worker whatever it is doing, end guest access, and close the session's port forwards, while +keeping its storage. The catalog does none of the plugin-specific parts: + +:::steps | Archiving S1 ++++ One write +`Catalog.archive({ id: "S1" })` sets `archived: true` in the session record, in one transaction. ++++ Revoke the session +The catalog calls `host.revoke({ session: "S1" })`. Every guest principal of S1 has `{ session: "S1" }` in its +`revokedBy`, so their objects end and their links close on the server and on the worker (chapter VII). New +connections are refused by `authenticate`, because grants of an archived session are void. ++++ Stop the worker +The supervisor reconciles from `catalog.sessions` with a routine: an archived session's worker is stopped, and +`authorize` refuses to start it again. ++++ Plugins reconcile +The ports plugin's routine on `catalog.sessions` closes forwards of archived sessions. The search plugin marks the +session's rows archived. If the server crashed between the write and these steps, the routines run again with the +current value at the next start: nothing is a notification that can be lost. ++++ Restore +`Catalog.restore` clears the flag. Grants are valid again, so guests reconnect and authenticate as before; forwards +stay closed. +::: + +### Clone, fork, relocate + +Each is work that outlives its call, can be cancelled and has one outcome, so each is a **job** (chapter XI) on the +catalog's local store, keyed by the owner's request id. They differ in what they pause: + +- **Clone** copies the session's store, uploads, downloads and artifacts under a new id, with the source worker + paused (refused while work runs) and title "(copy)"; directories `0700`, files `0600`, no links, no overwrite; a + partial copy is removed on failure. +- **Fork** before a user message copies the active context as of the entry before it into a new session (title + "(fork)"), with the message waiting in the new session's editor. It needs two workers: the source reads history, + the new one is seeded empty. A failure archives the copy. +- **Relocate** (cwd, environment or both): refused while the session is busy (naming the running subagents) or when + a subagent has its own cwd and the move crosses machines. The job fences the worker (new admissions wait), updates + the record, and moves the running worker in place by updating `SessionLocation`: natives that follow the location + switch with a generation (chapter IX), and conversations that follow the session's cwd are rewritten in one Durable + commit before work resumes. After 60 s it restarts the worker instead; on failure it releases the fence. + +:::review | Review: jobs as objects (#9) +The design defers whether a long relocation or a login is a record inside its owner (watched through `running`) or an +object of its own (`Relocation { id }`). The construction above uses records. Decide when login is ported +(`design.md` §15, §26). +::: + +### Delete + +p2 has no session delete. p3 needs one that removes the session everywhere: every document tagged with the session +(`store.retireByTag(SessionTag, id)`, chapter V), the worker's databases (the Durable store), the session's files, the +search rows, read marks, children copies, forwards, guest records, and finally the catalog entry. The design commits to +this outcome (`design.md` §25); how plugins take part is open. + +:::review | Review: session delete (#18) +Options in `design.md` §26: per-session objects plus `SessionTag` plus one session directory (delete is retire by tag +and remove a directory, plugins need do nothing), or owner hooks contributed through a catalog registry. Also open: +whether the search index is per session (deleted with it) or global with a hook. Who holds the store for +`retireByTag` is not specified either. +::: + +## Status, children, unread: reports to the server + +The session list must show status, subagents and unread marks for sessions whose workers are not running, and must +not start them. All three follow chapter X's pub/sub pattern: a routine on the worker calls a server object with the +current summary, the server object keeps the latest value per session (persisted where it must survive a restart), +ignores calls from an older `ctx.incarnation`, and clients read the server object. + +| Summary | Source on the worker | Kept on the server | +| --- | --- | --- | +| status: running, error, task count, busy conversations, usage and cost | the harness's task graph and usage | `SessionStatus`, in memory: a stopped worker is "not running" | +| children: name, model, thinking, tools, cwd, last entry, archived | the subagents plugin's child list | `Children`, persisted | +| latest root answer | the search feed | `Unread`: compared with the owner's read mark, persisted, shared by all owner devices | + +Guests see their own session's status without the error text. The owner's read mark is written when the root +conversation is visible, and a new session starts as read. + +## Conversations + +The harness (chapter XIV) owns the Durable store. The UI needs, per conversation: a live window of entries, older +pages, usage numbers, the working indicator, the inbox (queued inputs), and the actions. Two core objects provide them: + +- **`ConversationStore { conversation }`** (local, on the worker): opens the Durable watch as its native, reaching + the harness through the core-private `HarnessRuntime` registry entry, and exposes a **window** as a derive: the + newest entries (at most 100 or 512 KiB, at least 20) plus the conversation's documents. The window's start stays + put while it is within 1.5 times the limits, so new entries arrive as small appends instead of a shifted list + (chapter VIII's window pattern). +- **`Conversation { session, conversation }`** (remote, placement `Session`): the per-principal view. Variants by + `can()`: `owner` (everything, including model, thinking and handoff), `writer` (guest write: send, steer, stop, + compact, shell), `reader` (the window only). + +```ts +host.serve(Conversation, ({ session, conversation }, principal, self) => { + const store = self.get(ConversationStore, { conversation }); + const policy = self.get(ConversationPolicy, { conversation }); // core: readOnly, followsSessionCwd + const author = authorOf(principal); // structured: kind, id, display name + if (!can(principal, "send", session)) return { variant: "reader", view: store.window, page: store.page }; + return { + variant: can(principal, "manage") ? "owner" : "writer", + view: store.window, + page: store.page, // older history: returns data + send: ({ text, attachments, mode, requestId }, ctx) => { + if (policy.value.readOnly) throw forbidden("read-only conversation"); + return store.submit({ type: "input", content: text, attachments, author, whenBusy: mode, requestId }, ctx); + }, + stop: (_args, ctx) => store.stop(author, ctx), // allowed even when read-only + … + }; +}); +``` + +What each function becomes: + +| Function | How | +| --- | --- | +| send, retried send | Durable `submit` with the author's `requestId`: a retry with the same id is admitted once (Durable dedupes, chapter XIV) | +| steer or queue while busy | `whenBusy: "steer"` or a follow-up; the inbox is `docs["pi.inbox"]` in the window | +| attribution | the view derives the author from the principal; the entry stores it structured, the model sees a speaker block; nothing parses text | +| wait for prompt resources | the first input of a fresh worker waits until the prompt plugin's resources are loaded (a registry readable the store follows) | +| stop, dequeue | withdraw queued inputs, abort the run; the caller's texts return to its editor, other authors' to their **restored drafts** | +| restored drafts | a keyed document per (conversation, author), tagged `SessionTag`, persisted in the store; claimed once, first tab wins | +| read-only conversations | the core per-conversation policy document, written at `createConversation({ init })` by whoever creates the conversation; send, dequeue, configure, compact and shell refuse; stop is allowed | +| history pages | a method returning a page (100 entries, 1 MB, 2,000 scanned); compaction and reset markers always included | +| model and thinking | owner variant only; validated against the models plugin's catalog; the root conversation's model is also reported as the session default | +| compaction, handoff | Durable operations through the store | + +**Drafts in the composer** are UI state, not Chord state: the web shell keeps one draft in `sessionStorage`, the TUI +keeps one per conversation across UI replacements, and pending sends keep their request id until acknowledged +("Sending when connected"). Only restored drafts, which the worker produces, are documents. + +## Shell commands + +`!cmd` runs a command in the conversation's cwd on its environment and records the result as an entry; `!!cmd` does +the same but keeps the output out of the model's context. One command at a time per conversation; the live tail is +at most 50 KB or 2,000 lines, sampled every 200 ms; the full output spills to a file; the result entry is placed at the +run's next boundary while the agent is busy and never starts a run. This is the design's job example almost as is: + +```ts +const ShellJob = defineJob("chat.shell", { + version: 1, + args: obj({ command: str({ max: 100_000 }), include: bool(), author: Author }), + progress: obj({ output: str() }), + result: ShellResult, + concurrency: 1, // one per conversation +}); + +const results = self.state(ShellResults); // keyed by run id: dedupe after completion +const shell = self.job(ShellJob, { + run: async ({ command, include, author }, ctx) => { + const out = new OutputBuffer({ maxBytes: 50_000, maxLines: 2000, retain: "tail" }); + const env = await environmentFor(ctx); // the conversation's environment, chapter XX + const r = await env.exec(command, { cwd: env.cwd, signal: ctx.signal, spill: spillPath(ctx.jobId) }, + (chunk) => { out.push(chunk); ctx.progress({ output: out.snapshot().text }); }); + const data = { command, output: out.snapshot().text, exit: r.code, cancelled: ctx.signal.aborted }; + await conversation.submit( + { type: "write", entry: shellEntry(data, author, include), requestId: `shell:${ctx.jobId}` }, ctx); + return data; + }, + finish: async (done, tx) => { + (await tx.draft(results, done.id)).outcome = done.kind === "done" ? done.result : { error: done.error }; + }, +}); +``` + +The request id `shell:${ctx.jobId}` makes the entry idempotent: an attempt interrupted after the write cannot write +it twice. Cancel is `shell.abort(id)`, allowed for the command's author and the owner. A retried `runShell` with the +same request id returns the running job (dedupe of unsettled runs), or the stored result (the owner's own dedupe after +completion, chapter XI). p2's "cancel when nobody watched for 30 s" is dropped: a job keeps its owner alive and does +not depend on viewers. + +## Chat mode + +Chat mode is a per-conversation switch that everyone in the session sees and only the owner changes; forks and new +subagents start with it off. Because it must fork and rewind with the conversation, it is **transcript-bound**: a +Durable conversation document, not a Chord document. Messages without `@pi` become display-only Durable `write` +entries (author, text, attachments, reply quote); a message with `@pi` becomes an input with a digest of the chat +since the last mention (at most 100 messages, 20 files, 4 images), recording which entries it passed on. The reply +quote is derived on the worker from the entry id the client names: the client never supplies author or quote text. + +## Subagents and one-shot agents + +Subagents are Durable child conversations owned by tasks, as in Durable's own background-subagent example: one +`subagent` tool spawns, steers, waits for, stops and lists them; each child is owned by a background anchor task, and +each message is delivered by a reporter task that posts the answer back to the parent once, with request ids so a +restart sends nothing twice. Chord adds only three things: + +- the **child list** for the sidebar is reported to the server (`Children`, above), so it shows without a worker; +- a child that follows the session's cwd, or a read-only one-shot agent created by a code-mode script, is marked in + the core **per-conversation policy document** when it is created, so chat and relocation enforce it without knowing + the subagents plugin (`design.md` §25 porting guideline); +- archive and restore of a child from the UI are methods on the subagents plugin's view, refused for one-shots. + +Subagents cannot spawn subagents; thinking levels are clamped to what the model supports, in one helper shared with +the session's model choice. + +## Messages between sessions + +The root agent can list other sessions and send one an attributed message. The tool runs on the sender's worker and +calls a server object through `chord(fn, context)` (chapter XIV): + +```ts +const peerSend = defineTool({ + name: "session_send", + execute: (args, api, context) => chord(async (ctx) => { + const peers = await ctx.get(Peers, {}); // server, worker variant + await peers.send({ to: args.session, text: args.text, mode: args.mode, requestId: `peer:${api.taskId}` }, ctx); + return { output: [{ type: "text", text: "sent" }] }; + }, context), +}); +``` + +`Peers` checks that the sender is a root conversation of a non-archived session, refuses sending to itself and +archived recipients, then gets the recipient's `Conversation` (which starts its worker) and submits with the +sender as a structured author and the same request id (the tool task's id, stable across restarts), so a retried +tool call delivers once. The sender's +incarnation is `ctx.incarnation`; a call from a replaced worker is refused. Only owner-initiated turns may use the +tool; guests never. + +## Search + +The search plugin indexes user and assistant messages of root and subagent conversations (no shell output, no +Durable internal inputs) for title, cwd and message search across sessions, active or archived. + +:::flow wide animate | The search feed. The worker pushes pages from a cursor; the server owns the index and never starts a worker to answer a query. +row "Worker S1" blue +box commit "Durable commit" "in any conversation" +box wake "wake" "a routine on the harness's commit signal" +box page "read from cursor" "200 entries per page" +row "Server" accent +box index "SearchIndex" "native: SQLite FTS" accent +box cursor "cursor per conversation" "moves after the whole range" +box query "query" "terms, scope, at most 41 hits" +arrow commit -> wake +arrow wake -> page +arrow page -> index "report(page)" +arrow index -> cursor +note "Reports carry ctx.incarnation. Indexing is idempotent: a page sent twice changes nothing." +::: + +The index is a native owned by a host-lifetime object on the server; cursors live with the index, so after a crash +the worker asks for the cursor and continues. Rebuilding one session's index drops its rows and rescans, starting +an archived session's worker without resuming its work. Queries are methods returning data: at most 41 hits, +snippets of 300 characters, queries up to 500 characters. The same feed sets automatic titles, moves "updated at" +forward and supplies the latest root answer for unread marks. + +## Presence + +Viewers per session are `host.references(Conversation)` on the worker, counted per client (chapter VIII), and +reported with the status. Connected guests (name and mode) are the references whose principal is a guest. Presence +is not keep-alive: the supervisor uses `host.activity`, which also counts calls and jobs. + +:::note | Built so far +None of this exists in p3. The parts it relies on are Durable (built: submit with request ids, steer and queue, +child conversations, background tasks) and the Chord layers L3 to L9 (not built). +::: + +# Machines, Files, Models {#p2-machines lede="Environments, workspace, terminals, session files, ports and Radius, models and logins, prompt resources, logs, reload and the shells: the rest of p3."} + +The last domains are the ones that touch the outside world: other machines over SSH, the file system and git, +terminals, the relay, model providers and their credentials. Most of them are native objects, and most of them are +places where p2 has the same thing several times (four SSH connections to one environment, five notice channels, +three copies of the environment config parser). The rule that fixes that is the one from chapter XVIII: one owner per +thing. + +## Environments + +The **envs plugin** owns the environment configuration: name, host, user, port, identity file, shell and its +variables, plus a managed ed25519 key per environment and its own `known_hosts` (kept apart from the user's, aliased +per environment name). + +| Function | Owner and form | +| --- | --- | +| list, add, edit, delete (refused while a session, archived too, uses it) | `Envs` on the server: a persisted document behind an owner-only view | +| validation (no leading `-`, no whitespace or control characters in host and user) | the view's methods, before the write | +| a hand-edited `envs.json` is picked up; an invalid file keeps the last valid set and shows an error | a native file watcher (poll fallback) in the envs store; one writer (#21) | +| environment revision: a hash of the config, part of its identity | a derive | +| connection test (connect, deploy the daemon, platform, latency, warnings) | a job; the result per revision in a document | +| unknown or changed host key: fingerprints, trust after confirmation, forget | a flow (chapter XII) on the server | +| browse directories (`~`, relative paths, at most 10k scanned, 1,000 shown) | a method with a **call-scoped** SSH session (chapter IX) or the pooled connection | +| a changed environment restarts its sessions' workers once idle | a routine on the envs store and the status reports | + +**Execution in the agent.** The worker's harness asks `HarnessEnvironments` for an environment per target (chapter +XIV). The envs plugin contributes it from its worker facet: local execution at the session's cwd, or a remote one over +SSH through the `pi-env` daemon, deployed per platform and connecting at the first operation. A remote session never +falls back to local tools. The environment follows `SessionLocation`: after a move, the old connection closes once +the move has committed and watchers reopen at the new location. + +```ts +// envs/worker.ts +export default defineFacet((host, self) => { + const location = self.get(SessionLocation, { session }); + const pool = self.get(EnvPool, {}); // one owner of SSH connections (#20) + self.contribute(HarnessEnvironments, (target) => + target.env === "local" ? localEnv(target.cwd) : pool.environment(target.env, target.cwd)); +}); +``` + +:::review | Review: connection pools (#20) +SSH connections to one environment exist four times in p2 (worker environment, browse connection, test connection, +terminal host). `design.md` §26 asks for one owner with a pool or deliberate per-use connections. `EnvPool` above is +the first option: a native object that closes idle connections after 10 minutes and invalidates them on a config or +key change. Reloading the plugin that owns the pool kills running remote commands (chapter XV), so it should change +rarely. +::: + +## Workspace and file index + +The **workspace plugin** serves the viewed conversation's working tree (a subagent's own cwd included) on the worker: +list a folder (at most 2,000 entries, `.git` hidden), read a file (1 MiB, binary detected), git status, diff, log and +single commits, literal text search, and all paths for name filtering. Every query is a **method returning data**, not +state: results are bounded and requested by the UI. Only change notifications are state: a counter in a document, +bumped by a native watcher of the tree, `HEAD`, the index and config (debounced 100 ms); viewers reload what they show +when it changes. + +Two rules live in one place each instead of three: + +- **Confinement:** path rules, a canonical root, no `.git`, the app's private data directory denied (also excluded + from search). One helper used by listing, reading, uploads and the file index. +- **Git hardening:** no optional locks, no fsmonitor, filters disabled, no prompts or pager, no system config, 10 s + timeout, output caps. + +Downloads (25 MiB, byte ranges for media) go through the HTTP component (chapter XIII) with a route on the server that +reads chunks from the worker; uploads into the project use the chunked protocol of chapter XIII (1 MiB chunks, temp +file then rename, never overwrite, writes serialized). Guest read gets no workspace: the view has no variant for it. + +The **file index** for `@` completion belongs to the same owner, so completion consumes the index instead of scanning +on its own: built from the environment's listing (else git or a walk, at most 50k paths), rebuilt at most once per +second while files change, discarded after a move. Review comments drafted on files and diffs are browser state until +sent; sending them is an ordinary `Conversation.send`. + +## Terminals + +Terminals must survive server and worker restarts, so their PTYs live in a **detached terminal host process**, one per +data directory, with its own process lock; it exits after 10 minutes with no terminals and no clients. The terminals +plugin on the server is the owner the UI talks to: list (title, cwd, environment, size, exit status), open at the +session's cwd and environment (refused for archived sessions), attach, close (asking first if a foreground job runs). +Terminals are **owner only**. + +Bytes do not go over object links. Attaching opens a **channel** (chapter XIII) with a single-use token (30 s): JSON +control on the object, bytes on the channel, with its own acknowledged window (256 KiB, acknowledgements per 64 KiB; a +viewer that falls behind gets a fresh screen snapshot; an 8 MiB backlog disconnects). Attaching gets the current +screen from a headless emulator with 10k lines of scrollback. + +:::review | Review: the terminal host process (#23) +The terminal host runs its own code. Is it part of the revision, reloaded or restarted with it (and running terminals +end), or deliberately outside it with a "host runs older code" notice as in p2? Open in `design.md` §26. +::: + +## Session files + +Attachments (25 MiB, at most 20 per message), sent-file snapshots made by the agent's `send_file` tool, versioned +artifacts (HTML, SVG, Markdown, text; 2 MiB; each update a new immutable version) and shell spill files are all +per-session bytes with caps, immutable ids and range reads; p2 has two upload routes, two caps and two directory +layouts for them. The proposal is one owner: + +```ts +export const SessionFiles = remote("p3.session-files", { + version: 1, + input: obj({ session: str() }), + placement: () => Server(), // readable without a worker + variants: { writer: { … }, reader: { … }, worker: { … } }, +}); +``` + +`SessionFiles { session }` on the server holds the index (per kind, readable without a worker), the chunked upload +and read protocol, the caps per kind, copying on clone and fork, and deletion with the session. Attachments are copied +into a remote environment before an input is admitted, with no fallback on failure. Artifact previews run in a +sandboxed frame with its own CSP and no app identity; ordinary file previews allow only safe types. + +:::review | Review: session files (#17) +Proposed in `design.md` §26, not decided. Open sub-points: one preview limit (p2 has 256 KiB for sent files and +512 KiB for artifacts), what a fork copies of the artifact list, and how attachments are copied into remote +environments. Related: #22, whether on-disk layouts such as downloads and uploads get versions like documents. +::: + +## Ports and Radius + +The **radius plugin** keeps the outbound relay connection, a native in a host-lifetime object on the server: sign-in +by device code from the CLI (credential in a private file), token refresh, a heartbeat in the link adapter (20 s ping, +60 s timeout; TCP does not notice a sleeping laptop), backoff from 1 to 30 s, a disable switch. The relay delivers two +kinds of channels: + +```ts +// radius/server.ts (the entry point: one line, as in chapter I) +radius.onChannel((ws) => host.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } })); +radius.onForward((channel) => proxyToForward(channel, host.registry(RadiusForwards).value)); +``` + +Object links become owner principals with `access: "relay"`: owner, but treated as remote (no localhost-only +powers). p2 had to fake a loopback request with a secret header for this; with `accept` that is gone. + +Forward channels go to `localhost:` (Host, Origin and Location rewritten; 502 when nothing listens; anonymous +visitors only to public forwards). Which forwards exist is not radius's business: it owns a registry, +`radius.forwards`, and the **ports plugin** contributes its forwards (at most 50, random ids, a public toggle that +needs confirmation, an agent tool `ports` for top-level owner-initiated conversations). Without the ports plugin the +table is empty and radius still works: an optional dependency is a registry owned by the dependent (chapter XIII). The +ports plugin closes forwards of archived sessions with a routine on `catalog.sessions` (chapter XIX). + +:::review | Review: may a plugin call `accept`? +`design.md` §3 shows the Radius line in the server's start code and calls entry points "app or adapter code". If +radius is a plugin, its facet needs the host's `accept`, which no other part of this guide gives a plugin. The +alternative is that the start code keeps the entry point and gets channels from the radius object. Not specified. +::: + +## Models and credentials + +The **models plugin** owns the model runtime (pi-ai's models with custom providers and the model cache) in a +host-lifetime object, and the credentials. It contributes the runtime to `HarnessModels` (chapter XIV), so after a +models reload the next request uses the new runtime while a running stream finishes on the old one. **Credentials +never go to clients**: the views expose providers, login methods and which type is signed in, never tokens. Every +secret is registered for redaction in one place, the log capture. + +Views: a model catalog for pickers (names, ids, context windows, thinking levels; at most 300 matches in the web), +logout (refused during a login), refresh a provider's model list. The default model and the cycling scope are +settings (chapter XVIII). + +### Login + +A login is the guide's flow example (chapter XII): a **job** on the server that runs pi-ai's login, and a flow that +shows its steps to the starting owner. + +```ts +const LoginJob = defineJob("models.login", { + version: 1, args: obj({ provider: str(), type: AuthType }), progress: LoginState, result: obj({ ok: bool() }), + concurrency: 1, // one login at a time on the server +}); + +const login = self.job(LoginJob, { + run: async ({ provider, type }, ctx) => { + await runtime.value.login(provider, type, { + signal: ctx.signal, // 10 minute deadline, cancel, leaving + prompt: (prompt) => steps.ask(prompt, ctx), // text, secret, select, manual_code + notify: (event) => steps.notice(event, ctx), // info, auth_url, device_code, progress + }); + return { ok: true }; + }, + finish: async (done, tx) => { /* record the outcome; refresh the model list */ }, +}); +``` + +`prompt` is a promise-based callback inside the work; the resolver stays in the job (`steps.ask` records the step in +the flow state and waits for `answer`), and nothing crosses an object boundary. On a relayed connection the device +code method is selected automatically. The flow is visible only to the owner principal that started it; others are +refused. Leaving ends it: when the starting owner's references are all gone (presence, not an unmount), a routine +aborts the job. Codes, URLs and typed secrets are redacted from logs. + +## Prompt resources + +`AGENTS.md`, skills and templates are read in the session's environment, watched (poll fallback, last valid state), +and expanded on the worker before sending. The **prompt plugin** owns them per session on the worker, contributes the +system prompt section and skills to `HarnessExtensions` as readable contributions (a change recomposes), and exposes a +readiness readable: the first input of a fresh worker waits until it is ready (chapter XIX). After a move the watchers +reopen at the new location. + +## Code mode + +Code mode is a plugin (chapter XIV): its `pi-codemode` sandbox is a native; scripts call tools as nested Durable calls +and replay after a restart; the tool entry looks up the current sandbox when it starts. A reload seals and drains the +old sandbox for at most 30 s. The on/off switch is a setting. + +## Logs and notices + +**Log capture** tees stdout and stderr of the server, workers and detached hosts without changing their behaviour, +redacts known secrets and credential patterns, and keeps bounded rings (at most 128 sources, 1,000 lines or 256 KiB +each). Detached hosts publish redacted snapshots to a private file. Logs are read without waking a stopped worker. +**Notices** are the one channel from chapter XVIII; dismissal stores only notice ids. + +## Dev and reload + +The dev plugin is chapter XV's reload control for p3: a source watcher marks plugins dirty and never applies anything; +the `reload_plugins` agent tool and a control in the owner's shell reload the changed plugins, or restart all hosts for +base code (through the supervisor, reopening the sessions that were running). UI bundles are published only after a +successful backend change; a web page detects a new manifest and reloads with its drafts and navigation state saved; +the TUI swaps its sandboxed UI code with a snapshot of its state. A local Unix socket (`0700`) accepts development +commands without a network admin port. + +## The shells + +The web and TUI shells are UI facets over the objects above. Two rules matter for Chord: + +- **UI state survives navigation, reload and reconnect on purpose.** Composer drafts, pending sends with their request + ids, reading positions and review drafts are kept in tab storage or the TUI's replacement snapshot. A reconnect + re-holds and gets snapshots; nothing is lost that was not in flight. +- **Leaving is never derived from an unmount.** A plugin reload remounts components (chapter XV). Cancelling a login, + closing an overlay or giving up a lease is an explicit action or a dropped reference. + +The TUI launcher keeps downloaded UI code in a permission-dropped worker with no access to files, network or +credentials; local side effects (copy, upload, download, open a URL, an external editor) need a real user gesture or a +confirmation. That is app code around Chord, not part of it. + +## Transport + +The last domain, transport, is chapter XVII: one message format, per-part limits (16 MiB compressed and 64 MiB decoded +on client WebSockets, instead of p2's 1 MiB input cap and 32 pending calls), one byte window per link, re-holds with +snapshots after a reconnect, and a protocol version that makes an outdated web page reload. + +:::note | Built so far +None of p3 exists. pi-ai's login API, pi-codemode and the `pi-env` daemon exist and are used by p2 today; the HTTP +component and channels are deferred until after the kernel's first real proof (chapter XIII). +::: + +# Other Ways to Do It {#compared lede="Cordis and DeepSeek Harness attack the same problem, plugins that come and go while an agent runs, inside one process. What that buys, and what it costs."} + +Chord v2 is not the only answer to "replace plugin code while the app keeps running". The closest relative is +**Cordis**, a TypeScript plugin framework by Shigma that has been the plugin kernel of the Koishi chatbot framework +since 2019, and its most visible user, **DeepSeek Harness** (`dsh`), DeepSeek's open-source agent harness, whose +motto is "everything is a plugin". This chapter compares them with Chord v2 on the questions the guide cared about: +who owns side effects, how plugins depend on each other, what a reload keeps, what a crash keeps, and how far the model +reaches beyond one process. + +:::note | Sources and status +DeepSeek Harness at commit `687f282` (release 0.2.1-rc.2), which vendors Cordis as `@deepseek-ai/cordis` 4.0.5 (Cordis +RC7 with local changes, `vendor/README.md`): its Cordis tutorial (`docs/cordis-tutorial/`), `docs/architecture.md`, +`docs/subsystems/persistence.md`, `docs/subsystems/core.md` and `packages/boot/hmr/README.md`; the Cordis paper "A +Programming Paradigm for Spatiotemporal Composability" (arXiv 2608.25512). Cordis and dsh are built and in use; Chord v2 +is the design this guide describes. The comparison is between a working system and a design. +::: + +## Cordis in one example + +A Cordis plugin is a function (or an object with `apply`, or a `Service` subclass) that receives a **context**. It may +declare the services it needs with `inject`, and everything it does to the world goes through the context: + +```ts +export const name = "greeter"; +export const inject = ["tools"]; // activate only while a `tools` service exists + +export function apply(ctx: Context) { + ctx.tools.register(defineTool({ name: "greet", /* … */ })); // a registration is an effect: undone on unload + ctx.on("session/start", (s) => log(s.id)); // an effect-bound listener + ctx.effect(() => { // any other side effect, with its disposer + const timer = setInterval(work, 200); + return () => clearInterval(timer); + }); +} +``` + +Each plugin instance is a **fiber** with states `PENDING → LOADING → ACTIVE → UNLOADING → DISPOSED` (or `FAILED`). +Every effect, registration, listener, child plugin and provided service is recorded on the fiber, and unloading runs +the disposers. The paper calls this **temporal composability**: a component can be removed at any time and reverses its +own effects. Its counterpart, **spatial composability**, is the dependency side: `inject` is live, so a plugin starts +when its services appear, and unloads (and later reloads) when a service it depends on goes away or is replaced. The +paper's claim is that independently written components can therefore be combined and replaced without disturbing each +other. The next sections check how far that holds. + +**Services** are named capabilities on the context (`ctx.provide("x", value)` or `class X extends Service`). Names are +runtime strings in one flat namespace per application; TypeScript declaration merging makes `ctx.x` typecheck, but +nothing checks the service's shape at runtime, and optional dependencies are probed with `ctx.get("x")`. Scopes come +from the context tree: `ctx.extend()` derives a child context, `ctx.isolate(name)` gives one subtree its own binding of +a service, so two parts of an app can use two different `shell` providers. + +**Hot reload** follows from the two halves: the reload plugin unloads the old fiber (unwinding all its effects) and +loads the new module with the same config. Nothing held in the old instance's closures or fields survives; a plugin +that needs state across reloads must keep it in a service or on disk. Config edits are diffed by entry id and +mount, unmount or reconfigure only what changed; config fields declared volatile update in place without a remount. + +## DeepSeek Harness on Cordis + +dsh applies "everything is a plugin" to an agent: the model adapters (`ctx.llm`), the tool registry (`ctx.tools`), the +session log (`ctx.sessions`), the agents (`ctx.agents`) and the agent loop itself (`ctx.agentLoop`) are plugins and +services. A running dsh is a plugin tree composed at boot from a **profile** (`web`, `headless`, `sdk`, `acp`) that +stacks **bundles** of Cordis config rows, with user patches on top. There is no privileged core: replacing the agent +loop is a config patch. + +| Concern | How dsh does it | +| --- | --- | +| sessions | a durable, append-only event log per session (JSONL backend, single writer, `flush` as the durability barrier); model history is derived from the events | +| crash mid-turn | the log keeps everything appended; on resume the agent loop closes the open turn by appending missing tool errors and a synthetic `turn/end { reason: "interrupted" }` | +| tools | `ctx.tools.register(defineTool(…))`, an effect of the registering plugin; calls go through pre/execute/post policy waterfalls | +| approval | closed by default; `allowed-once` is the only grant; plugin installation needs full access or per-call approval | +| reload | a source watcher evicts changed modules; Loader entries running affected plugins stop and start again with the new code; a failed replacement is an activation failure, its previous code no longer runs | +| processes | one plugin tree in the Host process; the browser client and Electron's renderer talk to it over RPC streams | +| distribution | a Plugin Manager installs and enables bundles per profile; plugin markets are optional indexes | +| subagents | local subagents with their own durable sessions, and external ones | + +## What the guarantees are in practice + +Reading the Cordis source vendored in dsh and dsh's own documentation, the guarantees are narrower than the paper's +framing. An earlier review for p2 came to the same result (`apps/p2/docs/p3-reviews/dsh-compare.md`). + +- **Reversal is cooperative and unordered.** `_unload` runs all of a fiber's top-level disposers concurrently + (`Promise.all` in `vendor/cordis/src/fiber.ts`); order holds only inside one effect, and the primer tells authors to + "keep the related work in one effect" when teardown order matters. A disposer that throws is logged and skipped. A + disposer that never settles stalls the teardown; dsh's jobs service works around this by force-failing a job whose + cancel throws and warning that "the work may be orphaned" (`packages/jobs/jobs-local/README.md`). Reversal is as + good as every disposer, and nothing checks that a plugin registered all its effects. +- **Behaviour depends on registration order, and a reload changes it.** `emit`, `waterfall`, `serial` and `bail` + listeners run in registration order (`docs/cordis-primer.md`). Registering appends the listener (`push` in + `vendor/cordis/src/events.ts`, unless `prepend`), so when a plugin reloads, its listeners move to the end of the + list. A tool-policy listener that ran first runs last after its own plugin reloads; a waterfall that short-circuits + now sees a different decision first. +- **Dependencies are strings, and every provider reload restarts its dependents.** A fiber's activation key is the + concatenation of its injected providers' fiber ids (`_refresh` in `fiber.ts`), so replacing a provider unloads and + reloads every plugin that injects it, with their in-memory state. Service APIs are "pre-stable; update every + consumer" (`AGENTS.md`): there is no versioned contract between a provider and its consumers. +- **Reloading a core service ends live work.** Unloading the agent provider "stops and drains every live handle it + made" (`packages/core/agent/README.md`), so running agents stop; reloading the session persistence backend under + live sessions invalidates their write handles, and "writes fail loudly until the sessions restart" + (`.agents/notes/implemented/architecture/2026-08-27-handle-based-session-persistence.md`). +- **Registrations leave a gap.** A reloading plugin's tools and other registry entries are removed with its old fiber + and added again by the new one; anything that looks them up in between does not find them. + +None of this is a bug in the narrow sense: each point is documented, and dsh is built around it. But "components +interleave without disturbing each other" holds only for plugins that own nothing other plugins depend on, register +every effect, order nothing by registration, and hold no state that must outlive a reload. + +## Side by side + +| | Cordis and dsh | Chord v2 | +| --- | --- | --- | +| unit | a plugin instance (fiber) in one context tree | an object per (type, input, principal), placed on a host (chapters II, III) | +| side effects | recorded on the fiber with `ctx.effect` and friends; on unload the disposers run concurrently, each as good as its author made it | declared through `self` (documents, natives, routines, jobs, contributions); Chord activates and undoes them (chapters V, IX to XI) | +| dependencies | service names as strings in one flat namespace, via `inject`; a provider reload restarts every dependent | typed contracts via `get(Type, input)`; a reference parks and recovers through restarts (chapter VI); optional dependencies are registries (chapter XIII) | +| ordering | listeners run in registration order, which changes when a plugin reloads | registries compose with a declared `compose` function; nothing depends on load order (chapter XIII) | +| where code runs | one process; remote UIs over RPC to the Host | a tree of hosts (server, workers, tabs); objects reachable across links, relayed by the server (chapters I, XVII) | +| who is calling | not a framework concept | principals stamped by links, variants per caller, revocation (chapters III, VII) | +| state | the plugin's own business; dsh's session log is a service | documents written in transactions, persisted, versioned and migrated (chapter V) | +| reload | unload the old instance, load the new one: in-memory state and running work of that plugin end | pure objects swap their code in place with their documents and running work kept; only objects with natives restart (chapter XV) | +| a running agent turn during reload | entries of changed plugins stop; unloading the agent provider stops its live agents | the harness is base code; Durable pins tools per task phase, so a running call finishes on old code and the next phase uses the new (chapter XIV) | +| crash recovery of a turn | the turn is closed as `interrupted` | pi-durable continues the turn's tasks from their checkpoints; replay-safe tools rerun (chapter XIV) | +| contracts and versions | TypeScript types only, pre-stable APIs, every consumer updated together; config validated with a schema | contract versions with fingerprints and provider-side mappings for old consumers (chapter XVI) | +| status | in production in Koishi since 2019; dsh released | a design with the contract layer built | + +## What the comparison says + +**The core idea is the same.** "Factories declare, Chord activates" (chapter II) and Cordis' reversible effects are +two forms of one rule: a plugin may only change the world through its runtime, so the runtime knows what to undo. +Both reject the module-level timer and the stored callback for the same reason. + +**The difference is what survives a reload.** Cordis reloads at plugin granularity: the old instance unwinds and the +new one starts from nothing, together with every plugin that depends on it. That is simple, but a plugin's in-memory +state and its dependents' state and work in flight end with it, and correctness rests on every disposer and on +registration order. Chord v2 keeps state out of closures on purpose (documents, references, jobs as +records) so that it can swap the code of a pure object while its data and running work stay, and restarts only what +holds OS handles. That is more machinery: transactions, a store, versions on every document and contract. + +**The other difference is reach.** Cordis composes plugins inside one process, and dsh adds RPC to its own clients. +Chord v2 makes the process boundary part of the model: objects live on the host their placement names, callers are +principals, and a reload rolls out across the server, the workers and every tab. An app like p2, with a process per +session and guests in other browsers, needs that; a single-process agent does not. + +:::review | Review: what a reload does to a running tool call in dsh +The sources say that Loader entries running affected plugins stop and that unloading the agent provider stops its live +agents (`packages/boot/hmr/README.md`, `docs/subsystems/core.md`). They do not say what happens to a tool call in +flight when only the tool's plugin reloads; the comparison above does not claim either way. +::: + +# What's Next {.epilogue lede="Where Chord v2 stands, what is still undecided, and where the written sources disagree with each other."} + +The guide built Studio from one object to a p2-sized app. Every piece used the same few rules: one owner per thing, +factories that only declare, state that changes only in transactions, long-lived behaviour in declared forms, and +mappings and migrations owned by the provider. Those rules are what let Mario press Reload on his phone in the +preface while the agent keeps running. This last chapter says how much of that exists and what is still open. + +## Status + +:::stats +- ~10k | lines in the frozen first kernel (steps 1 to 6, 443 tests), kept as a reference only +- 1 of 11 | layers of the rebuild started: L1 contracts, built and reviewed +- 6–7k | target size of Chord v2 without tests +::: + +- **The design is decided** except the points marked below. `packages/chord/docs/next/design.md` is the target; + `packages/chord/docs/next/decisions.md` is the log and wins where they differ. +- **The rebuild** goes layer by layer, each with a one-page spec the owner approves, a minimal build with counting and + garbage-collection tests, the cost-guard metrics, and a cleanup loop: L1 data and contracts; L2 readable and + derive; L3 documents and store; L4 objects; L5 references and `ctx`; L6 protocol and links; L7 relay, routes, + revocation; L8 routines and jobs; L9 registries; L10 reload and rollout; L11 telemetry in Chord. +- **Built:** the contract layer (`src/next/contracts/`, commit `7d2bb8ea8`, chapter XVI). In pi: Delta, + `pi-telemetry`, `chord/context`, `chord/telemetry-sqlite`, pi-ai and Durable telemetry, Durable's required + `checkpointWhen`. +- **Next, in order** (`apps/p2/docs/p3-handoff.md`): decide Q12 and Q15, which the L1 spec needs; write the spec for + the rest of L1 (values, inputs and identity, argument checks, principals, errors; draft in + `packages/chord/docs/next/layer-1.md`); then Q11, Q13, Q14 and the cuts C3 to C17 before the layers they affect. +- **In pi:** `mapShared` and bytes as atomic leaves in Delta's `diffRevisions`. +- **Then** the playground web shell (React, esbuild) with the reload control and a preview at `p3.mariozechner.at`, + and the first real proof: the harness, one tool with an approval pause, a code mode reload. Then the HTTP and + channel components, then p3, domain by domain. + +## Deferred decisions + +Kept with stable numbers in `design.md` §26, each waiting for a concrete case: + +| # | Topic | When | Chapter | +| --- | --- | --- | --- | +| 9 | jobs as objects: a record inside the owner, or an object addressed by its id | when login is ported | XI, XIX | +| 11 | where a flow lives: a keyed document of its owner, or an object `LoginFlow { id }` | when login is ported | XII | +| 14 | project plugins: per-directory plugins with trust prompts and snapshots | own pass over `plugin-loading.md` | XV | +| 16 | a Durable Object profile | when a DO target becomes concrete | | +| 17 | session files: one `SessionFiles { session }` owner on the server (proposed) | p3 design | XX | +| 18 | session delete: retire by tag plus a session directory, or owner hooks | p3 design | XIX | +| 20 | connection pools: one owner of SSH connections per environment, or per-use connections | p3 design | XX | +| 21 | config files with a single writer: one owner on the server, others follow | p3 design | XVIII | +| 22 | file formats (`envs.json`, downloads, uploads) as versioned interfaces | p3 design | XX | +| 23 | the terminal host process: inside the revision or deliberately outside | p3 design | XX | + +## Open from review round 3 + +Decisions (`apps/p2/docs/p3-reviews/v2-review3-summary.md`; Q1 to Q10 are decided): + +| # | Problem | Proposal | +| --- | --- | --- | +| Q11 | a singleton that does not exist yet: `undefined` or `initial()`? | singletons read `initial()`; `undefined` only for absent keyed documents | +| Q12 | stripping arguments (R7) against "nothing is copied" | copy on change: a new object only when the argument had extra fields; never mutate the caller's value (proposed in the L1 draft) | +| Q13 | `ctx.incarnation` on relayed calls | relays stamp the origin host's incarnation, as they stamp the principal | +| Q14 | a writable holding functions exposed as a state member | only members with a JSON schema in the contract are exposable, checked at serve | +| Q15 | arguments of `local` methods: validated or not | apply R7 to `local` too, or keep `local` as the one exception | + +Cuts the owner may choose (each removes a primitive this guide teaches): + +| # | Cut | What it costs | +| --- | --- | --- | +| C3 | shrink jobs to a runner over documents (no versions, `accepts`, `blocked`, `persist`), or move jobs out of the kernel | job versioning, or `self.job` for core-only users | +| C4 | routines: `on` and `loop` only | `every`'s due time across reloads | +| C5 | merge `lifetime: "host"` objects into the facet | a plugin's always-on natives share one failure domain | +| C6 | remove `broken()` for opens and `retryOn` | nothing; reload and Retry still work | +| C7 | remove provider-side `linger` | ten lines of app code | +| C8 | remove `state: false` (method-only references) | a method-only consumer pays one snapshot | +| C9 | remove `previous` from derives | derives that used it move to `mapShared` | +| C10 | one `cell(initial)` for `constant` and `writable` | nothing | +| C11 | remove `ctx.trace` (use `traceIn(ctx, …)`) | nothing | +| C12 | remove call credits (the byte window covers calls) | nothing | +| C13 | two timeouts (`closeTimeout`, `readyTimeout`) instead of eight | per-thing overrides | +| C14 | remove the `handle.change` and `handle.retire` shorthands | nothing | +| C15 | remove `inVariant` and `expectVariant` (keep `RefIn`) | nothing | +| C16 | remove facet-owned mutable documents and their transaction exception | atomic writes across an object and its facet move to one owner | +| C17 | no harness-specific start-barrier exception | factories before the barrier cannot depend on the harness | + +## Where the sources disagree + +This guide follows the decision log where it is newer than the design, and the design where it is newer than +everything else. The places that need a fix in the sources: + +:::review | Review: drift in the written sources +- `design.md` §26 lists only the deferred decisions; it omits the open review-3 items Q11 to Q15 and the cuts C3 to + C17. +- `design.md` §3 has a garbled sentence, "Values are method arguments are stripped". +- `apps/p2/docs/p3-handoff.md` §3 is stale: it says all R items are decided and, two lines later, that R2 to R5 and + R7 to R26 are open, and "resume R6 next". +- `kernel.md` and `step-*.md` describe the frozen code: host values, a development freeze, consumer-side linger, + JSONL storage, descriptions on the wire, and a link closed on an unknown id. None of these is the target. +- `chord-next-walkthrough.md` lags the log: validation by trust (now R7, every caller), dev mode and freeze (removed + by R9), descriptions once per link (gone with R6), an IPC frame-count window (now one byte window, Q4), job outcome + `{ ok }` (now `{ kind }`), harness activation as the start barrier (R17), registries "newer refused at load" (Q8), + and hooks using `host.root` (now `chord(fn, context)`, Q6). +- `apps/p2/docs/questions.md` uses `self.resource` and `replace` (both rejected) and `host.root` in hooks. +- `apps/p2/docs/p3-reviews/v2-cost-model.md`: the measurements are current, its "Decisions this needs" section is + not (chapter XVII). +- L1's `Implementation` shape differs from the factory return value used in the design (chapter XVI). +::: + +# API Reference {.appendix #api lede="Every name the guide uses, grouped by where you meet it. Working names; only the contract layer is built."} + +## Hosts + +| Call | Meaning | Chapter | +| --- | --- | --- | +| `createHost({ address?, principal?, storage?, principalKey?, revokedBy?, telemetry?, onError?, timeouts })` | a host; with an address it is addressable and its own code acts as `principal`; without, a client | I | +| `host.load([{ name, facet }])` | load facets by plugin name; base code passes its core facet the same way | I | +| `host.connect(link, { principal })` | the link to the parent (at most one) | I | +| `host.accept(link, { principal })` | a link from a client; every request on it acts as `principal` | I | +| `host.route(Kind, { authorize, resolve })` | how to reach hosts of a kind; `resolve` returns `{ link, principal }` | I | +| `host.revoke(r)` | end matching objects and links here and below | VII | +| `host.root(fn, { principal?, signal?, telemetry? })` | a call scope with a `ctx` for code outside objects | VI | +| `host.get(Type, input, { signal, state?, parkTimeout? })`, `host.hold(…)` | a reference from outside objects; `release()` ends it | VI | +| `host.serve(Type, factory, { lifetime?, linger? })` | serve an object type (in a facet) | II | +| `host.registry(Registry)` | the composed value of a registry, a readable | XIII | +| `host.references(Type)` | who holds references to objects of a type on this host (presence) | VIII | +| `host.activity` | `{ references, calls, jobs, lingering }`, for keep-alive | I | +| `host.close()` | end the host: objects, natives, links | I | + +## Inside a factory: `self` + +| Member | Meaning | Chapter | +| --- | --- | --- | +| `self.state(Definition)` | a document handle | V | +| `self.tx(async (tx) => …, ctx)` | a transaction on the object's line | V | +| `self.derive(name, [sources], fn)` | a derived readable | VIII | +| `self.get(Type, input)` | a reference held for the object's life | VI | +| `self.open(open, close, { closeTimeout? })` | a native; `.value` once open | IX | +| `self.routines.on(readable, fn)`, `.every(interval, fn)`, `.loop(fn)`, `.custom(…)` | standing behaviour | X | +| `self.job(Job, { run, finish? })` | owned work that outlives its call | XI | +| `self.contribute(Registry, value)` | a contribution (facets and `lifetime: "host"` objects only) | XIII | +| `self.writable(initial)` | a host-local readable of any in-process value, with `set` | XIII | +| `self.retryOn(readable)` | a trigger that retries a broken activation | VI | + +**Document handles:** `value`, `get(key)` (keyed), `keys`, `change(fn, ctx)` / `change(key, fn, ctx)` and `retire(ctx)` / `retire(key, ctx)` (shorthands for +one-document transactions). **Transactions:** `tx.draft(handle, key?, { tags? })`, `tx.retire(handle, key?)`, +`tx.start(job, args, { key? })`. **Jobs:** `job.start(args, ctx, { key? })`, `job.abort(id)`, `job.running`, +`run.outcome(ctx)`. + +## References and `ctx` + +| Member | Meaning | +| --- | --- | +| `ref.` | a state member (readable: `.value`, subscribe) or a method | +| `ref.state`, `ref.onState(fn)` | `loading`, `live`, `failed` (cause, `retryAt`), `released` | +| `ref.retry()` | retry a failed reference now | +| `ref.release()` | end a reference taken outside objects | +| `ref.members` | the variant's members | +| `ctx.signal`, `ctx.principal`, `ctx.incarnation`, `ctx.get` | cancellation, caller, calling host's incarnation, call-scoped references | +| `ctx.telemetry`, `ctx.trace(name, attributes, fn)` | the current span; a child span with a child `ctx` | +| `ctx.jobId`, `ctx.progress(value)` | in a job's `run` | + +## Definitions + +| Name | What it declares | Chapter | +| --- | --- | --- | +| `hostKind(name, input?)` | a kind of host and its address | I | +| `remote(name, { version, input, placement, variants })` | an object type reachable across links | II | +| `local(name, { version, input, members })` | an object type on its own host | IV | +| `method(args?, result?)` | a method in a base declaration | II | +| `evolve(prev, (e) => …)`, `checkAccepts(contract, accepts)` | the next contract version; the provider's mappings (**built**) | XVI | +| `Implementation` | the type of a provider for the current version (**built**) | XVI | +| `str num int bool nul lit oneOf union obj arr map opt bytes unknown`, `Static` | schema helpers (**built**) | II | +| `defineState(name, { schema, version, initial, checkpointWhen, persist?, keyed?, ordered?, migrate? })` | a document | V | +| `defineTag(name)` | a tag; `store.retireByTag(tag, value)` | V | +| `defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })` | a job type | XI | +| `defineRegistry(name, { version, compose, accepts? })` | an extension point | XIII | +| `bindFacet

()` | base code's `defineFacet`, with factories' principals typed as the app's `P` (working name) | III | +| `defineFacet((host, self) => …)` | a plugin's setup for one kind of host; imported from base code (`@studio/core`) | II, III | +| `constant(value)` | a readable that never changes (start data) | IX | +| `broken(error)` | a failure that is not retried by the backoff | VI | +| `inVariant(ref, …)`, `expectVariant(ref, …)`, `RefIn<…>` | narrowing a reference by variant | III | +| `mapShared(fn)` | an array map that keeps unchanged items identical (planned in Delta) | VIII | +| `OutputBuffer`, `Progress` | bounded output and throttled progress (`@earendil-works/chord/output`) | XI | +| `memoryLinkPair()`, `ipcLink(child)`, `webSocketLink(ws)` | link adapters | I | +| `DocumentStorage`, `registerDocumentStorageConformance` | the storage port and its shared test suite | V | + +# Timeouts and Defaults {.appendix #defaults} + +| Timeout or default | Bounds | Default | Override | +| --- | --- | --- | --- | +| `closeTimeout` | a native's `close`, then reported as a possible leak | 30 s | per native | +| call drain | an ending object's own call handlers | = `closeTimeout` | per type | +| `readyTimeout` | activation; an awaited `get` | 60 s | per `get` | +| park deadline | parked calls (link drop, loading, rollout) | 60 s | per `get` (`parkTimeout`); per type | +| routine handover | a new routine waiting for the old handler on reload | 10 s | per routine | +| `finish` retry | backoff of a throwing job `finish` | 1 s to 60 s | per job | +| rollout install | one host installing a revision | 60 s | none | +| linger | an object kept alive after its last reference | per type | per type | +| backoff | retries of transient failures, per address | 1 s doubling to 60 s | none | +| eviction | keyed persisted documents idle in memory | 10 minutes | | +| client WebSocket limits | per part | 16 MiB compressed, 64 MiB decoded | per link kind | +| compression | JSON parts compressed at their origin | above about 16 KiB | | +| worker keep-alive (p2, app code) | idle time before the supervisor stops a worker | 5 minutes | app | + +# Versions {.appendix #versions} + +| Surface | Version | On change | Chapter | +| --- | --- | --- | --- | +| object type | `version` plus fingerprint | `evolve` and `checkAccepts`; otherwise `contract_mismatch` | XVI | +| document | `version` plus schema fingerprint | pure `migrate` on access; no bump fails `broken` | V | +| job | `version` | `accepts` on outcomes and `running`; `blocked` without | XI | +| registry | `version` | `accepts` on contributions; unmatched ones left out and reported | XIII | +| native | none | restarts on reload | IX | +| Durable tasks and documents | Durable's `version` | Durable's `migrate` | XIV | +| link protocol | protocol version | refused; hosts update together | XVII | +| UI components, Chord and app APIs | none yet | exposed as they are | | + +# Errors and States {.appendix #errors} + +`ChordError` has a stable `code` and a finer `reason`; on the wire it is `{ code, reason?, message, details? }` +(specified in `layer-1.md`, not built). + +| Code | Meaning | +| --- | --- | +| `invalid` | input, arguments or placement rejected (`invalid/input`, `invalid/args`, `invalid/placement`) | +| `contract_mismatch` | the consumer's version and fingerprint are neither current nor accepted | +| `forbidden` | the caller may not do this | +| `gone` | the object or hold ended | +| `not_ready` | not activated in time | +| `released` | the reference was released | +| `too_large` | a part exceeded its size limit | +| `unavailable` | could not be reached before the deadline (`unavailable/locked` and others) | +| `aborted` | cancelled by a signal | + +| Type | Values | +| --- | --- | +| `FailureCause` | `{ kind: "transient"; error }` (retried with backoff) or `{ kind: "broken"; error }` (retried only by a trigger) | +| reference state | `loading`, `live`, `failed` (cause, `retryAt`), `released` | +| job outcome | `{ kind: "done"; result }` or `{ kind: "failed"; error: "threw" \| "aborted" \| "lost" }` | +| job phase | `running`, `finishing`, `blocked` | +| rollout status of a provider | `updating` (calls park), rejected (fail at once), same revision (`contract_mismatch`) | +| host install report | `installed`, `installed_failed`, `rejected` | + +# Rejected Designs {.appendix #rejected} + +From `design.md` §31, grouped. Each was considered and is not coming back without a new reason. + +| Area | Rejected | +| --- | --- | +| reload | drain-based handover; object restart for state changes; a factory trial; a migrate dry run; a consumer contract check during staging; reload votes; per-plugin rollback within a host; keeping natives across reloads by name; `restart_required`; host values and a development freeze | +| state | version fences without transactions; volatile documents; per-group lines; optimistic transactions; `tx.get`; a JSONL document storage; validating state and documents on commits or reads; copying values on same-host calls; op-rewriting projections and a view algebra; Durable's Session as the object store | +| objects and references | resources replaced in place (`replace`, `deps`, stale handles); revoking proxies on `.value`; consumer-side linger; detached references counting toward presence; an `optional` flag on references; `opaque` members; callbacks as method arguments; generic call or request ids; a stream type in Chord | +| registries | registries created at runtime; entries resolved through mutable indirection; generic registry keys with a `conflict` flag; numeric middleware priorities; registry composition as a staging check | +| principals and placement | principal field patterns and re-stamping for revocation; `storeKey`; placement as data naming an input field; host-name strings in app code; "magic" input-field matching for delete | +| wire | an outbox for transient delivery errors; cross-relay reset coordination; count limits that refuse calls or holds; contracts taken from the running app | +| process and tooling | a topology layer above the host API; loader variants A and B; Chord ending the process on a storage failure; `defaultProjectTrust: "always"`; ceremony (`native/` directories, `defineNative`, lints); weakening type checking | + +# Glossary {.appendix #glossary} + +**Adapted binding:** a consumer bound at an older accepted contract version, served through the provider's mappings. +**Base code:** the app's own code, imported statically; a change restarts the hosts. **Client:** a tab or TUI host, +not addressable. **Contract:** an object type's declaration (name, input, placement, members, version). +**Contribution:** a value a plugin adds to a registry. **`ctx`:** what every call receives. **Derive:** a value +computed from readables. **Document:** a named JSON value an object owns. **Facet:** a plugin's code for one kind of +host. **Factory:** the synchronous function that declares an object. **Fingerprint:** a hash of a contract's or +schema's normal form. **Flow:** input into running work, kept as state by its owner. **Host:** one participant of the +app, with an identity and an incarnation. **Incarnation:** the id a host gets on every start. **Input:** an object's +key among objects of its type; with the type and the caller's principal key it identifies the object. **Job:** owned work that outlives its call. **Link:** an ordered connection between +two hosts. **Local type:** an object type reachable only on its own host, without principals. **Native:** an OS-level +value opened with `self.open`. **Park:** a call waiting for a reference to become live again. **Placement:** the +function from input to host address. **Plugin:** a directory with a `package.json`, reloadable. **Principal:** who +calls, stamped by the link. **Principal key:** the string that identifies a principal in object identities. **Pure +object:** an object without natives; swapped in place on reload. **Readable:** a value plus its ordered changes. +**Reference:** what `get` returns. **Registry:** an owner's extension point. **Relay:** a host forwarding messages +between two links, rewriting only headers. **Reload:** swapping a plugin's code in running hosts. **Restart:** ending a +process. **Revision:** the complete code set the hosts should run. **Revocation:** ending live access for principals +matched by `revokedBy`. **Routine:** standing behaviour of an object. **Server:** the addressable root host. **Shared +store:** a local object that holds data once behind per-principal views. **Tag:** a typed label on documents, used to +retire them together. **Transaction:** an atomic set of document writes on the object's line. **Variant:** the set of +members a principal gets. **View:** a per-principal remote object in front of a shared store. **Worker:** a host the +server starts for one session. + +# Sources {.appendix #sources} + +| Source | Used for | Weight | +| --- | --- | --- | +| `packages/chord/docs/next/decisions.md` | every decision, review rounds 2 and 3 | wins where newer | +| `packages/chord/docs/next/design.md` | the target design, all chapters | authoritative otherwise | +| `packages/chord/src/next/contracts/` and its `README.md` | chapter XVI | built code | +| `packages/chord/docs/next/layer-1.md` | errors, inputs, argument checks | draft spec | +| `apps/p2/docs/p3-reviews/v2-review3-summary.md` | open Q11 to Q15, cuts C3 to C17 | current | +| `apps/p2/docs/p3-reviews/v2-cost-model.md` | measurements, per-hop costs | current except its last section | +| `apps/p2/docs/p3-functions/` | chapters XVIII to XX | current | +| `apps/p2/docs/p3-handoff.md` | status and order of work | §3 partly stale | +| `packages/durable/README.md` | the harness, submit, tasks, subagents | current | +| `chord-next-walkthrough.md`, `kernel.md`, `step-*.md`, `apps/p2/docs/questions.md` | background only | lag the decisions (see What's Next) | +| DeepSeek Harness at `687f282` (its Cordis tutorial, architecture, persistence, core and HMR docs), the Cordis paper (arXiv 2608.25512) | "Other Ways to Do It" | external, built systems | diff --git a/packages/chord/docs/next/explainer/index.html b/packages/chord/docs/next/explainer/index.html new file mode 100644 index 000000000..e13c03572 --- /dev/null +++ b/packages/chord/docs/next/explainer/index.html @@ -0,0 +1,5409 @@ + + +Chord v2: a guide to building apps that reload + + + + +

+
+
A guide · Chord v2 target design · 2026-10-20
+

Chord v2
A Guide

+

How to build an app like p2 from nothing on Chord v2 and pi-durable, one object at a time, so that new plugin code takes over while the agent keeps running

+
11layers in the rebuild, L1 to L11
~1,300lines built so far (L1 contracts)
6–7ktarget size of Chord v2, without tests
0.004 msDelta's apply per small change
+

This guide teaches the designed API of Chord v2 (packages/chord/src/next, export @earendil-works/chord/next) +from packages/chord/docs/next/design.md and the newer decision log packages/chord/docs/next/decisions.md, which wins +where the two differ. Most of it is not built yet: a first kernel of about 10k lines is frozen as a reference and +Chord v2 is being rebuilt layer by layer; only the L1 contract layer exists (src/next/contracts/, commit +7d2bb8ea8). Code in this guide is therefore the target, not something you can run today. Every chapter says what is +built. API names are working names. Notes marked review are points that are undecided, or where the sources +disagree; they name the files. The reader is assumed to know pi-durable; Chord is explained from the start.

+
+
+ + +
+

Before we begin

+

It is late in the evening. Mario is on his phone, connected to his laptop at home through the Radius relay. In session +S1 the agent is twenty minutes into a refactoring; right now it is running a test suite through bash. A colleague, +Ann, follows the session read-only from her own browser. Mario notices that the notes panel sorts notes the wrong way +round. He fixes one line in the notes plugin on the laptop and presses Reload.

+

What should happen is easy to say. The notes panel on both screens shows the new order. The test run keeps running. The +agent's turn continues as if nothing happened. Ann's view does not flicker into an error. Nothing that was written is +lost, and no process restarts.

+

The obvious ways to get there fail. Restarting the processes loses the turn in flight, the open terminal and every +connection; pi-durable would resume the turn, but the test run and the connections are gone. Hot-swapping modules +in place, as front-end tooling does, leaves the old code alive in closures, timers, event listeners and sockets that +nobody knows about: the old sort function keeps running inside a callback someone stored, a file watcher opened by the +old code is never closed, and two versions of the plugin now write the same data without any order.

+

Chord v2 is the runtime that makes the first outcome normal. It does so by being strict about one thing: every piece +of state, every open OS handle and every piece of running work has exactly one owner, an object, and objects only +declare what they need. Chord itself activates, connects, reloads and ends them. Because Chord knows every timer, +watcher, transaction and reference an object has, it can swap the object's code while its data stays, restart only +what touches the operating system, and route calls between browser tabs, the server and per-session worker processes +without the plugin author writing any of that.

+

This guide teaches Chord v2 by building an app called Studio: a small version of p2 (Pi Mobile), with a server, +one worker process per session, browser tabs as clients, and an agent from pi-durable on each worker. The first +chapters grow one plugin, @studio/notes, until it uses every concept; the last three chapters assemble the rest of a +p2-like app from the same parts.

+
Review: what exists today

Nothing in this guide runs yet except the contract layer. packages/chord/src/next/contracts/ (L1 part 1: remote, +local, method, evolve, checkAccepts, the schema helpers) is built and reviewed. The rest of L1 (inputs, +argument checks, principals, errors) is a draft spec (packages/chord/docs/next/layer-1.md). The frozen kernel in +src/next/*.ts implements an older version of the design (host values, a development freeze, JSONL storage, +descriptions on the wire); do not read it as the target. Build on the design, not on the frozen code.

+
+

Ten things to know

+
    +
  1. An app is a tree of hosts. One server, any number of workers it starts on demand, any number of clients (tabs, +TUIs) that connect to it. Clients reach workers only through the server, which relays.
    2. +
  2. Objects are the unit of everything. An object type is a contract (name, input, placement, members, version); an +object is one instance per input and caller. The first get creates it, later gets share it, the last release ends +it.
    3. +
  3. Factories declare, Chord activates. A factory is synchronous: it declares documents, references, derived +values, OS handles, routines and jobs through self, and returns the members. Chord opens, binds and starts them, +and undoes them in reverse.
    4. +
  4. Links stamp who is calling. A caller's data never says who it is; the link it arrived on does. Each caller gets +its own object and the variant of members it may use.
    5. +
  5. Shared data lives once, behind views. A local shared store holds the data; per-caller remote views expose +what each caller may see. Only stable identities persist documents.
    6. +
  6. State changes only in transactions. Documents are JSON values written in self.tx; commits are atomic for +every viewer and travel as small operations, never as whole values.
    7. +
  7. Failures look like disconnects. A reference survives link drops, restarts and failures: calls park or fail, +values stay readable, and Chord retries by itself. There are no request ids.
    8. +
  8. Everything long-lived has a declared form. Standing behaviour is a routine, work that outlives a call is a job, +input from a person is a flow, behaviour from other plugins comes through a registry. Nothing else may hold a timer +or a callback.
    9. +
  9. Reload swaps pure objects in place and restarts native ones. Objects without OS handles get the new code with +their data and running work intact; objects with OS handles close and reopen them. A broken change leaves the whole +host on its old code.
    10. +
  10. pi-durable runs inside a core object. The agent harness is a host-lifetime object on each worker; models, +environments and tools reach it through registries, and tools reach Chord through chord(fn, context).
    11. +
+

Words used throughout

+

A host is one participant of the app: a process, a browser tab or a Durable Object, identified by a logical +identity and an incarnation that changes on every start. A link connects two hosts. The server is the one +addressable root host; a worker is a host the server starts for one session; a client is a tab or TUI, not +addressable. A plugin is a directory with a package.json; its code for one kind of host is a facet. Base +code is the app's own code (Chord calls its facet the core facet); it restarts the hosts when it changes, while +plugins reload. An object type (or contract) describes objects; an object is one instance; its input is +its key among objects of that type, and its placement computes from the input on which host it lives. A factory builds an +object; self is what a factory declares through. A principal is who calls, stamped by the link; a +variant is the set of members a principal gets. A reference is what get returns. A readable is a value +plus its ordered changes. A document is a named JSON value an object owns, written in a transaction. A +derive is a value computed from readables. A native is an OS-level value (a socket, a watcher, a process) +opened with self.open; an object with natives is a native object, one without is a pure object. A +routine is standing behaviour, a job is owned work that outlives its call, a flow is input into running +work. A registry is an owner's extension point, filled with contributions. ctx is what every call +receives: cancellation, caller, call-scoped references. A reload swaps a plugin's code inside running hosts; a +restart ends a process; a revision names the complete code set the hosts run.

+
Reading the code in this guide

Imports are shown once per file and then omitted. obj, str, arr and friends are Chord's schema helpers (they +return ordinary TypeBox schemas). Helper functions that belong to the app (authenticate, watchDir, uuidv7) are +named and not shown. Where the design gives no name, the guide says it uses a working name.

+
+
+ +
I

The Shape of an App

Which processes a Chord app has, how they find each other, and the start code that connects them.

+

Before any plugin exists, Studio needs processes and connections between them. In Chord each participant is a host, +and the app's own start code creates every host, connects them and decides how workers start. Chord deliberately does +none of that itself: a laptop app forks child processes, a test runs everything in one process, a Cloudflare +deployment routes through the edge. This chapter writes Studio's start code: server.ts, worker.ts, the browser +entry, and a test setup.

+

Three roles

+

A Node app on Chord has one shape:

+
CLIENTS: NOT ADDRESSABLESERVER: EXACTLY ONEWORKERS: ONE PER SESSION, ON DEMANDbrowser tabowner, localbrowser tabguest AnnTUIownerserverServer()workerSession({ id: "S1" })workerSession({ id: "S2" })workerSession({ id: "S3" })WebSocketWebSocketWebSocketIPCIPCA worker for S3 does not exist until something asks for an object that lives there.
Figure 1. The three roles. Clients connect to the one server; the server starts workers and relays between them.
+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
RoleAddressableWho it is comes fromLifecycle
clientnothe authentication of its linkconnects by itself
serveryes, exactly oneits own declarationruns
workeryes, many, by a typed addressstamped by the server, its parentstarted on demand by the app
+

The server holds what all sessions share (the session list, access, settings) and starts workers. A worker +runs one session: its agent, its files, its terminal connections. Clients are browser tabs and TUIs. Nobody can +send a request to a client, so a client can only serve objects that nobody outside it needs; chapter IV names them +local types.

+

Things that look like hosts but are not: a code mode sandbox (a worker thread inside a worker, owned by an object), a +remote environment's daemon (it speaks its own protocol, owned by an object), the TUI's sandbox behind its launcher. +A detached process such as a terminal host could become another kind of child of the server.

+

Host kinds and addresses

+

Every addressable host has a typed address. The app declares its host kinds once, in base code:

+
// studio/src/hosts.ts
+import { hostKind, obj, str } from "@earendil-works/chord/next";
+
+export const Server = hostKind("server");
+export const Session = hostKind("session", obj({ id: str() }));
+
+Server();                  // the address of the server
+Session({ id: "S1" });     // the address of the worker of session S1
+

There are no host-name strings anywhere in app code. Chord serializes addresses internally; your code only calls the +kinds. An object type names its host with a placement function from its input to an address (chapter II), so the +type checker knows that Session({ id: session }) needs a session field.

+ +

A link is an ordered, two-way channel of messages between two hosts; it only delivers. Chord ships three: a memory +pair for tests, IPC for a Node child process forked with serialization: "advanced", and WebSocket. All three carry +the same binary message format and the same flow control (chapter XVII). Recovery policy belongs to the link adapter: +IPC needs no heartbeat, a WebSocket through a relay needs one, because TCP does not notice that a laptop went to sleep.

+

Each link has a direction. A host has at most one link to its parent (connect) and any number of links from +below (accept for clients, route for workers). Requests arriving on a link act as the principal the link was given; +chapter III explains why that matters.

+

The server

+
// studio/src/server.ts
+import { createHost, ipcLink, webSocketLink } from "@earendil-works/chord/next";
+import { fork } from "node:child_process";
+import { WebSocketServer } from "ws";
+import { Server, Session } from "./hosts.ts";
+import { coreServerFacet } from "./core/server.ts";
+import type { P } from "./principal.ts";
+
+const server = createHost<P>({
+  address: Server(),
+  principal: { kind: "server" },                 // who this host's own code acts as
+  storage: await openServerStorage(dataDir),     // persisted documents (chapter V); memory if omitted
+  onError: (e) => { if (e.kind === "storage_failed") process.exit(75); else log(e); },
+});
+
+await server.load([
+  { name: "@studio/core", facet: coreServerFacet(config) },   // base code, under a fixed name
+  ...(await loadPlugins("server")),                           // [{ name, facet }] from the plugin loader
+]);
+
+const wss = new WebSocketServer({ server: http, path: "/chord", maxPayload: 64 * 1024 * 1024 });
+wss.on("connection", (ws, req) => {
+  const principal = authenticate(req);           // app code: cookie, loopback, device token
+  if (!principal) return ws.close(4401);
+  server.accept(webSocketLink(ws), { principal });
+});
+
+server.route(Session, {
+  authorize: (p, { id }) =>
+    p.kind === "owner" || p.kind === "server" || (p.kind === "guest" && p.session === id),
+  resolve: async ({ id }, { signal }) => {
+    const principal: P = { kind: "worker", session: id };
+    const child = fork("worker.ts", [], { serialization: "advanced", signal });
+    child.send({ id, principal, init: await initFor(id, signal) });   // the app's own start message
+    return { link: ipcLink(child), principal };                       // Chord stamps `principal` on it
+  },
+});
+

Each call has one job:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CallMeaning
createHost<P>({ address?, principal?, ...options })with an address: an addressable host whose own code acts as principal; without: a client. P is the app's principal type
load([{ name, facet }])the facets this host runs, each with its plugin's package.json name; duplicates and invalid names are refused. Base code passes its core facet the same way
accept(link, { principal })a link from a client; every request on it acts as principal
route(Kind, { authorize, resolve })how to reach hosts of a kind. authorize(principal, address) runs before anything starts; resolve(address, { signal }) returns { link, principal }
connect(link, { principal })the link to the parent, at most one, replaced after a reconnect; principal says who the parent is
root(fn, { principal?, signal? })a call scope for code outside objects (a CLI command): gives fn a ctx (chapter VI)
+

Chord has no listen. Entry points differ too much: a loopback WebSocket server, a tunnel, the Radius relay +delivering channels, a Durable Object edge, a memory link in a test. Each entry point authenticates and calls +accept. Radius, for example, is one more line:

+
radius.onChannel((ws) =>
+  server.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } }));
+

The worker

+
// studio/src/worker.ts
+const { id, principal, init } = await startMessage(process);       // app code: the first IPC message
+const worker = createHost<P>({
+  address: Session({ id }),
+  principal,                                                       // { kind: "worker", session: id }
+  storage: await openSessionStorage(init.dataDir, id),             // one storage per session
+});
+await worker.load([
+  { name: "@studio/core", facet: coreWorkerFacet(init) },          // serves the start data (chapter IX)
+  ...(await loadPlugins("worker")),
+]);
+worker.connect(ipcLink(process), { principal: { kind: "server" } });
+

A browser tab

+
// studio/web/main.tsx
+const client = createHost<P>();                                    // no address: a client
+await client.load(await loadPlugins("web"));
+client.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } });
+const notes = await client.get(Notes, { session: "S1" });          // tab → server → worker S1
+

The first get, step by step

+

The tab asks for the notes of S1. No worker for S1 runs yet.

+
browser tabownerserverroute(Session)app codeworker S1placement: Session({ id: "S1" })hold Notes { session: "S1" }not mine: route(Session)authorize(owner, { id: "S1" })trueresolve({ id: "S1" })fork + start messagecreateHost, load, connect{ link, principal: worker S1 }hold (stamped: owner)placement check, run factorybound: snapshotbound: snapshot
Figure 2. The first get for an object on a worker that is not running. The server authorizes, the app starts the worker, Chord stamps the link.
+

resolve runs once per address while its link lives. When the link drops (the worker crashed or the supervisor +stopped it) and references to that worker still exist, Chord calls resolve again by itself, with a per-address +backoff, so a tab that only watches recovers without a click. A worker start that already produced a link finishes +even if the request that triggered it went away.

+

Incarnations make restarts visible. Chord mints a fresh id (a uuidv7) on every host start and sends it in the link +handshake; the other side sees a restart as a new incarnation of the same identity, and references to it re-hold +(chapter VI).

+

Tests: everything in one process

+

The same code runs in a test with memory links. resolve builds the worker host in memory and returns one end of the +pair:

+
server.route(Session, {
+  authorize: () => true,
+  resolve: async ({ id }) => {
+    const [up, down] = memoryLinkPair();
+    const principal: P = { kind: "worker", session: id };
+    const worker = createHost<P>({ address: Session({ id }), principal });
+    await worker.load([{ name: "@studio/core", facet: coreWorkerFacet({ id, dataDir }) }, notesWorker]);
+    worker.connect(up, { principal: { kind: "server" } });
+    return { link: down, principal };
+  },
+});
+

Memory links deliver the same encoded messages asynchronously, so a test runs the production protocol path.

+

What stays app code

+

Starting processes, the start message, authentication, supervising and stopping idle workers are app code. Chord +provides one readable for the last one: host.activity, with { references, calls, jobs, lingering } (viewers, calls +in flight, unsettled jobs, objects kept warm after their last reference). Studio's core reports it to the server +together with the agent's running tasks, and the server's supervisor stops a worker when everything has been zero for +five minutes, as p2 does. Chord counts; it has no idle policy.

+

Host options and timeouts

+

createHost takes the options introduced later in this guide: storage (V), principalKey and revokedBy (III, +VII), telemetry (XVII) and onError. onError receives background errors and { kind: "storage_failed" }; every +app must handle the latter, because Chord never ends the process. Studio exits with its own code so its supervisor +restarts it.

+

Timeouts are host defaults with per-thing overrides; the full table is in the appendix. The two you meet first: +readyTimeout (60 s: how long an activation and an awaited get wait) and the park deadline (60 s: how long a call +waits through a link drop or a restart before failing unavailable).

+
Built so far

Nothing in this chapter is built in Chord v2 yet: hosts, links and routes are layers L6 and L7. The frozen kernel has +working versions with the same shape (createHost, connect, accept, route, load), which the rebuild +simplifies.

+
+
+ +
II

A First Object

A plugin, a contract, a factory, and a browser tab that calls it on another process.

+

Studio's first feature is session notes: a list of short notes per session, shown in a panel, that anyone allowed can +add to. Several places need "the notes of S1": the owner's tab, a guest's tab, later a download route and an agent +tool. If each built its own copy, the copies would drift apart and nobody would know when to clean them up. In Chord +there is one object for "the notes of S1", everyone asks for it by its type and input, and Chord routes them all to it.

+

The plugin's files

+

A plugin is a directory with a package.json. Its name (npm style, globally unique and stable) is the plugin's +identity: Chord finds its stored documents, reports its errors and orders its contributions by it, never by a +directory name or load order.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FileRuns onHolds
package.jsonread by the loader"name": "@studio/notes"
contract.tsevery host that imports itobject types: names, shapes, method signatures; no running code
worker.tsevery session workerthe worker facet
server.tsthe serverthe server facet (later: the download route)
web.tsxbrowser tabsthe web facet: the panel
+

The contract

+

An object type is a contract: a name, an input (the object's key among objects of this type), a placement +(on which host it lives, computed from the input), the members (state you can read and methods you can call) and a version.

+
// notes/contract.ts
+import { arr, int, method, obj, remote, str, type Static } from "@earendil-works/chord/next";
+import { Session } from "@studio/core/hosts";
+
+export const Note = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int() });
+export type Note = Static<typeof Note>;
+
+export const Notes = remote("notes", {
+  version: 1,
+  input: obj({ session: str() }),                          // the key: { session: "S1" }
+  placement: ({ session }) => Session({ id: session }),    // input → the host's address
+  variants: {
+    editor: {
+      state: { notes: obj({ items: arr(Note) }) },
+      methods: {
+        add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()),   // returns the new id
+        remove: method(obj({ id: str() })),                              // no result
+      },
+    },
+  },
+});
+

remote means objects of this type can be reached from any host, across links (chapter IV introduces local). The +schema helpers (obj, str, int, arr, map, opt, lit, oneOf, union, bool, num, nul, bytes, +unknown) return ordinary TypeBox schemas; short options such as { min, max } bound lengths or values. Schemas must +describe data: no callbacks, no class instances, no custom codecs.

+

method() has four forms: method() (no arguments, no result), method(args), method(args, result) and +method(undefined, result).

+

A contract contains no code that runs, so every host can import it. When you declare it, Chord materializes its +description and computes a fingerprint, a hash of its input, variants and members. Two hosts compare contracts by +version and fingerprint (chapter XVI). Placement is not part of the fingerprint.

+

Placement

+

The input says which object; placement says where it lives. The caller runs the placement function on the input and sends the resulting address with its request; the server +routes by it. The provider runs the function too and refuses an object that is not its own with invalid/placement. +So a worker for S2 never builds "Notes for S1", even if a caller has an outdated rule. Because placement is a typed +function, a Session placement on a type whose input has no session field does not compile.

+

Inputs are normalized before anything else: defaults are applied, unknown fields removed, and a canonical key +computed. { session: "S1" } and { session: "S1", extra: 1 } are the same object.

+

The facet and the factory

+

A facet is a plugin's setup for one kind of host. It runs once when the plugin loads on that host and tells the +host which object types it serves:

+
// notes/worker.ts
+import { defineState } from "@earendil-works/chord/next";
+import { defineFacet } from "@studio/core";          // Studio's defineFacet: principals typed as Studio's P (chapter III)
+import { Note, Notes } from "./contract.ts";
+
+const NoteList = defineState("notes.list", {      // a document (chapter V); in memory for now
+  schema: obj({ items: arr(Note) }),
+  version: 1,
+  initial: () => ({ items: [] }),
+  checkpointWhen: () => true,
+});
+
+export default defineFacet((host, self) => {
+  host.serve(Notes, ({ session }, principal, self) => {
+    const list = self.state(NoteList);            // THE note list of this object
+    return {
+      variant: "editor",
+      notes: list,                                // a document handle is a readable: expose it as state
+      add: ({ text }, ctx) =>
+        self.tx(async (tx) => {
+          const id = uuidv7();
+          (await tx.draft(list)).items.push({ id, text, by: "someone", at: Date.now() });
+          return id;
+        }, ctx),
+      remove: ({ id }, ctx) =>
+        list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx),
+    };
+  });
+});
+

Read host.serve(Notes, factory) as "this host builds Notes objects like this". The factory receives the +normalized input, the caller's principal (chapter III) and self, through which it declares what the object needs. +It returns the members of one variant, tagged with variant.

+

Factories declare; Chord activates. The factory is synchronous and has no side effects. It does not await, open +files or start timers. self.state(NoteList) does not load anything: it declares that the object has this document +and returns a handle that stays valid for the object's whole life. Chord then performs the activation: it loads the +document, binds references, opens OS handles, starts routines, and only then lets calls in. The rule exists for +reload: a reload runs the factory again, and anything the factory did itself would happen twice.

+

Calling it

+
// notes/web.tsx
+export function NotesPanel({ session }: { session: string }) {
+  const notes = useRef(Notes, { session });                 // app-side hook over host.get / release
+  const items = useReadable(notes?.notes)?.items ?? [];
+  return (
+    <section>
+      {items.map((n) => <p key={n.id}>{n.text}</p>)}
+      <AddBox onAdd={(text) => notes?.add({ text })} />
+    </section>
+  );
+}
+

Underneath the hooks is the reference API:

+
const notes = await client.get(Notes, { session: "S1" });   // live and hydrated once awaited
+notes.notes.value.items;                                    // the current list
+notes.notes.subscribe((v) => render(v.items));              // every committed change
+const id = await notes.add({ text: "check the flaky test" });
+notes.release();                                            // the object may end now
+

An awaited reference is live and hydrated: every state member has a value. State members are readables: +.value is the current value, subscribe is called after every commit. The first get for { session: "S1" } runs +the factory; every later get for the same input and caller shares that object. The object ends when the last +reference is released (a type may keep it a little longer with linger).

+
browser tabserverworker S1Notes { S1 }call add { text }relay (header only)strip + validate argsself.tx: push note, commitresult: idstate: ops of the commitresultstate, then result
Figure 3. A call to add. Arguments are checked once at the provider; the commit reaches the viewer as one small change.
+

Two rules for members

+

Methods return data, never objects. A method that wants to point you to another object returns its address +({ session: "S2" }), and you get it yourself. That keeps every reference owned by someone who releases it.

+

No callbacks as arguments. A callback carries code, and everything it closes over, from one object into another. +Suppose an inbox object passed onFile to another object, which stored it. After a reload of the inbox's plugin, the +other object still calls the old onFile; after the inbox ends, it calls into an object whose file watcher is +already closed. Chord forbids this by construction: members are data (JSON plus bytes). Behaviour from elsewhere goes +through registries (chapter XIII), multi-step work through jobs (XI), input from a person through flows (XII).

+

What is wrong with this version

+

This first version works for one person. It has two problems, which the next two chapters fix. Everyone gets the +editor variant, including a read-only guest. And, less obvious, the owner and a guest do not see the same notes: as +chapter III shows, a remote object exists per caller, so each caller's Notes object has its own NoteList.

+
Built so far

remote, method, the schema helpers, Static and fingerprints are built (src/next/contracts/). Serving, get, +references and documents are layers L3 to L5.

+
+
+ +
III

Who Is Calling

Principals: how Chord knows who calls, why a caller cannot lie about it, and how the answer picks the object and its members.

+

The owner may add and remove notes. Ann, a read-only guest, may only read them and should not even see who wrote +what. The object must therefore know who is calling, and the caller must not be able to claim to be someone else. In +Chord the answer is a principal, a small JSON value the app defines and the links stamp.

+

The app's principal type

+

Chord defines no principal kinds; the app does. Every variant has a kind. Studio uses p2's set:

+
// studio/src/principal.ts
+export type P =
+  | { kind: "owner"; access: "local" | "relay" | "device"; device?: string }
+  | { kind: "guest"; session: string; mode: "read" | "write"; grant: string; revision: number }
+  | { kind: "server" }
+  | { kind: "worker"; session: string };
+ + + + + + + + + + + + + + + + + + + + + + + +
kindWho
ownerthe person running the app, on loopback, through the relay, or on a paired device
guestsomeone invited into one session, with a mode and the grant (and its revision) that let them in
serverthe server's own code calling a worker
workera session worker calling the server
+ +

A principal never comes from a request's data. The server authenticates a client once, when its WebSocket connects, +and stamps the result on the whole link (server.accept(link, { principal })). The server tells Chord who each worker +is when it resolves it (return { link, principal }). When the server relays a client's request to a worker, Chord +writes the client's principal into the forwarded request; a principal field coming from a client is always +overwritten, and only a trusted parent link may carry one.

+

Where principals come from is app code. Studio's sharing feature creates a share link per session and mode; a guest +who opens it gets a guest record (grant g1) and a cookie; authenticate(req) turns the cookie into +{ kind: "guest", session: "S1", mode: "read", grant: "g1", revision: 1 } on every new connection. Chord never sees +cookies or grants, only the finished principal.

+

The caller is part of the identity

+

A remote object exists once per (type, input, principal key). The principal key is a string computed from +the principal; by default its canonical JSON (sorted keys). The app may pass its own principalKey to createHost, +and must pass the same function on every host, or identities split silently.

+
const principalKey = (p: P) =>
+  p.kind === "owner" ? `owner:${p.access}:${p.device ?? ""}`
+  : p.kind === "guest" ? `guest:${p.grant}:r${p.revision}`
+  : p.kind === "worker" ? `worker:${p.session}`
+  : p.kind;
+

The design wants the key to include how the client is connected and the grant's revision, so that two access paths +never share objects and a changed grant never reuses the old grant's object:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CallerPrincipal keyNotes { S1 } object
owner, tab A, localowner:local:1
owner, tab B, localowner:local:1 (shared with tab A)
owner through the relayowner:relay:2
guest g1, revision 1guest:g1:r13
guest g1 after a change to write mode, revision 2guest:g1:r24 (new)
+

Variants

+

The contract lists named sets of members; the factory receives the principal and returns the variant that principal +may use:

+
// notes/contract.ts (version 1, final form for now)
+variants: {
+  editor: {
+    state: { notes: obj({ items: arr(Note) }) },
+    methods: { add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), remove: method(obj({ id: str() })) },
+  },
+  reader: {
+    state: { notes: obj({ items: arr(PublicNote) }) },     // no `by`: authors are hidden
+  },
+},
+
// notes/worker.ts
+host.serve(Notes, ({ session }, principal, self) => {
+  if (canWrite(principal, session)) return { variant: "editor", notes: …, add: …, remove: … };
+  return { variant: "reader", notes: … };
+});
+

A reader has no add at all. Chord refuses a call to a member that is not in the caller's variant, so add needs no +check of its own. Three rules keep variants compatible: a member name has the same kind (state or method) in every +variant that has it; a method has the same arguments and result in every variant; only state schemas may differ per +variant (the reader's notes have no authors). remote(...) enforces them at the type level, and at runtime for +contracts that arrive as data.

+

How a facet knows P

+

P is Studio's type, not Chord's, and every host is created with it (createHost<P>). A plugin's facets are written +and compiled separately, then handed to host.load as values, so the type checker cannot infer P from the host into +the factory's principal. Studio's base code therefore exports its own defineFacet, already bound to P, and every +facet of every plugin imports that one instead of Chord's:

+
// studio/src/facet.ts (base code, exported as @studio/core)
+import { bindFacet } from "@earendil-works/chord/next";
+import type { P } from "./principal.ts";
+
+export const defineFacet = bindFacet<P>();         // a facet's principals are Studio's P
+export type { P, Revocation } from "./principal.ts";
+
// notes/worker.ts (server.ts, web.tsx and tui.ts import the same defineFacet)
+import { defineFacet } from "@studio/core";
+
+export default defineFacet((host, self) => {
+  host.serve(Notes, ({ session }, principal, self) => {   // principal: P
+    if (principal.kind === "guest" && principal.mode === "read") return { variant: "reader", notes: … };
+    …
+  });
+});
+

The import is cheap: @studio/core is a host package plugins already import at runtime (the contract imports Session +from it), P is a type-only import, and bindFacet is a one-line function. The host checks the pairing: load +accepts only facets typed for its own P, so a facet compiled against another app's principal type, or one built with +Chord's untyped defineFacet, does not compile into Studio's start code. Client facets (web.tsx, tui.ts) use the +same defineFacet; they rarely see a principal, because clients serve only local types.

+

None of this is checked at runtime, and none of it needs to be: the principal a factory receives is the one the link +stamped.

+
Review: binding facets to the app's principal type (open)

The design shows defineFacet((host, self) => …) without saying how P reaches the factory. The frozen kernel takes +it as an explicit type parameter, defineFacet<P>(…), defaulting to { kind: string } +(packages/chord/src/next/facet.ts), and the playground writes defineFacet<AppPrincipal>(…). The pre-bound +defineFacet above (bindFacet is a working name) is the owner's preferred direction, recorded as an open point in +packages/chord/docs/next/layer-1.md §4.

+
+

Narrowing on the caller's side

+

Owner and guest run the same web.tsx, and which variant they get is known only at runtime. The types force every +caller to narrow before using variant members:

+
const notes = await client.get(Notes, { session: "S1" });
+notes.add({ text: "hi" });                         // type error: `add` exists only in "editor"
+if (inVariant(notes, "editor")) await notes.add({ text: "hi" });
+expectVariant(notes, "editor");                    // or assert (throws)
+function Editor({ notes }: { notes: RefIn<typeof Notes, "editor"> }) { … }
+

The protection is not the type checker; it is Chord refusing the call at the provider. A client that ignores the +types gets the same error.

+

Admission before a worker starts

+

Principals also decide whether a caller may reach a host at all. route(Session, { authorize }) runs before Chord +resolves an address, so a guest of S1 asking for anything in S2 is refused at the server, and no worker starts for +them. Finer authority lives in the per-principal view's methods; there is no asynchronous "may this caller activate +this object" guard. An object whose address carries authority (say, a login flow by id) may exist; what a caller can +see and do is decided by the variant its factory picks.

+

Inside methods

+

ctx.principal is the caller of the current call. In Studio's notes the editor's add uses it to record the +author:

+
add: ({ text }, ctx) => store.add({ text, by: authorOf(ctx.principal) }, ctx),
+

Where principals act, in summary:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
PurposeWhere
identity: which object a caller getsprincipal key
variant: which membersthe factory
admission: whether a host is reachableroute.authorize
attribution and fine checksctx.principal in methods
revocationrevokedBy (chapter VII)
+

The bug from chapter II

+

Now the bug is visible. The owner's Notes { S1 } and Ann's Notes { S1 } are two objects (rows 1 and 3 above), and +each declared its own NoteList. The owner's notes are not in Ann's list. Per-caller objects must not hold the data; +they must be views over something that exists once. That is the next chapter.

+
+ +
IV

One Store, Many Views

Local types, and the pattern every Chord plugin uses: shared data once in a store, per-caller views in front of it.

+

Every caller has its own Notes object, which is what makes per-caller variants possible. But data must exist once. +The fix is the most common pattern in Chord: a shared store, a local object that holds the data and does the +heavy work once, behind thin per-principal views, the remote objects callers get.

+

local types

+
// notes/contract.ts
+export const NoteStore = local("notes.store", {
+  version: 1,
+  input: obj({ session: str() }),
+  members: {
+    notes: obj({ items: arr(Note) }),                          // state: a schema
+    public: obj({ items: arr(PublicNote) }),
+    add: method(obj({ text: str(), by: str() }), str()),       // methods: method(...)
+    remove: method(obj({ id: str() })),
+  },
+});
+

A local contract has flat members and no variants. The differences:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
remotelocal
reachable fromany host, across linksits own host only
identity(type, input, principal key)(type, input)
principals, variantsyesno
factory(input, principal, self)(input, self)
argumentsstripped to the provider's schema and validated, for every callersee the review note below
values on the same hostpassed by reference, never copiedpassed by reference, never copied
ctx.principalthe callernone
may persist documentsno (chapter V)yes
+

A local object has no principal: it is trusted code on its own host, acting for the host. So attribution ("who wrote +this note") is passed as data (by), by the view that knows who called. Clients serve only local types, because +nobody can address a client.

+

The pattern

+
// notes/worker.ts
+export default defineFacet((host, self) => {
+  host.serve(NoteStore, ({ session }, self) => {
+    const list = self.state(NoteList);
+    const pub = self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) }));
+    return {
+      notes: list,
+      public: pub,                                   // computed once, for every reader
+      add: ({ text, by }, ctx) =>
+        self.tx(async (tx) => {
+          const id = uuidv7();
+          (await tx.draft(list)).items.push({ id, text, by, at: Date.now() });
+          return id;
+        }, ctx),
+      remove: ({ id }, ctx) => list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx),
+    };
+  });
+
+  host.serve(Notes, ({ session }, principal, self) => {
+    const store = self.get(NoteStore, { session });   // a reference, declared: held for the view's life
+    if (canWrite(principal, session)) return {
+      variant: "editor",
+      notes: store.notes,                             // the store's readable, exposed as-is
+      add: ({ text }, ctx) => store.add({ text, by: authorOf(principal) }, ctx),
+      remove: (args, ctx) => store.remove(args, ctx),
+    };
+    return { variant: "reader", notes: store.public };
+  });
+});
+
PER-PRINCIPAL VIEWS (REMOTE)SHARED STORE (LOCAL)Notes { S1 }owner · editorNotes { S1 }guest Ann · readerNotes { S1 }guest Bob · readerNoteStore { S1 }notes, public (a derive)self.getself.getself.getAuthority is the view's: the reader variant has no add. The store does not check; it is trusted and acts for the host.
Figure 4. One store, three views. The redaction runs once in the store; both readers read the same value.
+

self.get(NoteStore, { session }) in a factory declares a reference that the view holds for its whole life; Chord +binds it during activation and waits until the store is live (chapter VI). The view exposes the store's readables +directly. Owner, Ann and Bob now give three Notes objects, one NoteStore, one note list and one redacted list.

+

Any heavy or shared part belongs in a store like this: a pi-durable watch, a file watcher, a sandbox, a search index. +Base code uses the same pattern with the same primitives; nothing about it is special to core.

+

Derived values, briefly

+

self.derive(name, sources, fn) computes a value from readables and recomputes it when a source changes; viewers get +only the difference. The reader's list above is one. Chapter VIII covers derives in depth, including the one footgun +that matters: stripAuthor must return the same object for an unchanged note, or every commit costs the size of the +whole list. The tool for that is mapShared.

+

Arguments are checked at the provider

+

Trusted code is not type-checked code: plugins are bundled with esbuild without type checks, and agents write +JavaScript plugins. So the provider checks arguments itself (decision R7): every method's arguments are stripped to +the provider's own schema (unknown fields removed) and validated with it, whoever the caller is, and never with a +schema a peer sent. Schemas are compiled lazily on first use and cached per schema object. Bytes are checked as leaves +(instanceof Uint8Array plus an optional length bound), never walked. Stripping copies only when a field was removed; +the caller's value is never mutated (proposal Q12, in the L1 draft). Results and state are not validated: they come +from the provider.

+

On the same host nothing is copied: arguments and results are passed by reference. Values are immutable by +convention (Delta's rule, as in Durable): nobody mutates a value it received, and Chord neither copies nor freezes +values to enforce it. There is no development mode that would freeze them; one behaviour everywhere (R9).

+
Review: are local arguments validated? (Q15, open)

Design §5's table says local arguments are "neither validated nor copied"; §3 and R7 say arguments are validated +"for every caller". Review round 3 raised this as Q15 (apply R7 to local too, or keep local as the one exception), +and layer-1.md lists it as open. Until it is decided, write local methods so that they do not depend on either +answer: trust the shape, but never trust content (lengths, ids) that came from a remote caller without the view +checking it. chord-next-walkthrough.md concept 6 still describes the older rule ("validated only when the caller is +not a host principal"), which R7 replaced.

+
+

Choosing between the two

+

Use a remote type for what callers address: views, things with per-caller authority. Use a local type for what +exists once per input on one host: stores, caches, owners of OS handles, and the core objects a host provides +(chapter IX). If a type is reached across a link, it is remote; if a type persists documents, it is local (or a +host-lifetime object, chapter V).

+
+ +
V

Writing It Down

Documents and transactions: how an object stores data, changes it atomically, survives restarts and migrates it.

+

Notes must be stored, changed safely while others read them, sent to viewers as small changes, and survive a worker +restart. An object's data lives in documents: named JSON values the object owns, changed only in +transactions. The design is a port of pi-durable's store, so most rules will look familiar; this chapter states +them for Chord.

+

Defining a document

+
// notes/worker.ts
+const NoteList = defineState("notes.list", {
+  schema: obj({ items: arr(Note) }),
+  version: 1,
+  initial: () => ({ items: [] }),
+  persist: true,                                                   // stored; default: memory only
+  checkpointWhen: (_value, _ops, info) => info.deltasSinceBase > 64,
+});
+
+const Draft = defineState("notes.draft", {                         // one unfinished note per author
+  schema: obj({ text: str() }),
+  version: 1,
+  initial: () => ({ text: "" }),
+  keyed: true,                                                     // many per object, by key
+  persist: true,
+  checkpointWhen: () => true,                                      // small: a full base per write
+});
+

checkpointWhen is required on every definition, with no default (R14, as in Durable since pi 5c1736f82): it says +when a commit stores a complete base instead of a delta, which bounds how much is replayed on load. Small documents +checkpoint on every write; growing ones after a number of deltas.

+

A document's root is always a JSON object (lists live in a field). Documents hold JSON only; bytes are a value and wire +type, not a document type.

+

Singleton and keyed

+
host.serve(NoteStore, ({ session }, self) => {
+  const list = self.state(NoteList);      // THE note list of this object
+  const drafts = self.state(Draft);       // all drafts of this object
+  …
+});
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
singleton (no keyed)keyed
how manyone per objectmany per object, by a short key
loadedat activation, kept in memoryon demand
readlist.value (synchronous)await drafts.get(key): the value or undefined
list(it is one value)for await (const key of drafts.keys()) pages keys
expose as stateyes: the handle is a readableno: expose a method or a singleton index
+

There is no live key list. A collection shown live is a singleton (an object holding a list or a map); a large +collection is a method returning pages.

+

Transactions

+

All writes go through self.tx(fn, ctx). Inside, await tx.draft(handle, key?, { tags? }) returns a draft of the +current document, or a draft over initial() if it is absent. Drafts record operations (assignments, push, +Object.assign); asking for the same document twice in one transaction returns the same draft. tx.retire(handle, key?) removes a document. There is no tx.get: reading is drafting, and a draft is an overlay over the current +value, so nothing is copied. A draft you do not change commits nothing, so reading never stores an absent document.

+
saveDraft: ({ author, text }, ctx) =>
+  self.tx(async (tx) => { (await tx.draft(drafts, author)).text = text; }, ctx),
+

Shorthands for one document: handle.change(fn, ctx) (keyed: handle.change(key, fn, ctx)) (loads the document, then runs the synchronous fn on +its draft) and handle.retire(ctx) (keyed: handle.retire(key, ctx)).

+

Four patterns

+

Two decisions (R12 and R13, both as in Durable) shape how transactions are written. Drafts are revoked when the +transaction settles, so a transaction returns at most plain values, never something taken from a draft. And +there is no existence check inside a transaction.

+
// 1. Return data: write in the transaction, read after the commit.
+await self.tx(async (tx) => { (await tx.draft(list)).items.push(note); }, ctx);
+return list.value.items.length;                    // the committed revision, or a newer one
+
+// 2. Remove if present: retire of an absent document is a no-op.
+await drafts.retire(author, ctx);
+
+// 3. Create with tags if missing: the draft creates it, only if it changes.
+await self.tx(async (tx) => {
+  (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text;
+}, ctx);
+
+// 4. Branch on existence: read first, outside the transaction.
+if ((await drafts.get(author)) === undefined) { … }
+

Durable's rule

+

Inside the callback, only await tx.*. No other objects, no network, no models, no processes, no second self.tx, no +reads past tx. Breaking it is not an error but a hang: the object's line is held while the callback waits. Fetch +first; decide and write inside.

+
// Wrong: a fetch inside the transaction holds the object's line.
+await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title: await fetchTitle(url) }); }, ctx);
+// Right: fetch, then write.
+const title = await fetchTitle(url, ctx.signal);
+await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title }); }, ctx);
+

What a commit does

+
INSIDE SELF.TXAFTER THE WRITEwait for the lineper objectrun the callbackdrafts record opsprepareDelta checks placed valuesstorage batchdeltas or a baseadoptpointer swapnotify locallyonce, with the opssendone batch per linkrevoke draftstouching one throwsstoredAn error or an abort before the storage write discards everything. Storage commits are serialized per host.
Figure 5. The commit sequence, Durable's. Nothing is copied, frozen or revalidated; viewers get one batch per commit.
+

Lines are per object, never across objects; reads outside a transaction do not take the line. Subscribers on the same +host get the new value and the commit's operations by reference. Remote viewers get the commit's operations, never the +document again; chapter XVII shows how one commit touching several holds still arrives as one message.

+

Tags and deleting by tag

+

Tags are typed tokens set when a document is created:

+
export const SessionTag = defineTag<string>("studio.session");     // base code, shared by every plugin
+await self.tx(async (tx) => { (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; }, ctx);
+// when the session is deleted
+await store.retireByTag(SessionTag, "S1");                          // every document tagged S1
+

The design names the operation store.retireByTag(tag, value) without saying who holds the store; session delete is +one of the app decisions still open (#18, chapter XIX).

+

Who may persist

+

persist: true is allowed only in local objects and lifetime: "host" objects: identities without a principal. A +remote object's identity contains the principal key, which changes with the access path and the grant revision. Its +stored documents would scatter across access paths and be orphaned after a reconnect. The rule is enforced at the type +level (self.state in a remote factory accepts only non-persisted definitions) and at declaration ("persist needs a +stable identity: put the documents into a local shared store"). This is the second reason for the shared store.

+

What survives a host teardown (a crash, a restart) is exactly what is persisted. In-memory documents live as long as +their object: they survive reloads and native restarts of a live object, and go when it ends. Everything else +(natives, derives, routines, references) is rebuilt on activation.

+

Facet documents

+

A facet has its own identity (its plugin's name) and may hold documents too, for plugin-wide data such as +preferences. Objects the facet serves may read and write their facet's documents, and one transaction may span the +object's own documents and its facet's (lock order: own line, then the facet's). The facet never touches a served +object's documents.

+

Ownership across plugins

+

No plugin reads or writes another plugin's documents, Chord or Durable. Data crosses plugins only through state and +methods. Data that core enforces lives in a format core owns and is handed to core through its API when the data is +created (as with Durable's createConversation({ init })), never written into core's documents.

+

Versions and migration

+

A change of shape or meaning raises version and adds a pure migrate(old, from):

+
const NoteList = defineState("notes.list", {
+  schema: obj({ items: arr(NoteV2) }),           // NoteV2 adds `pinned`
+  version: 2,
+  migrate: (old, from) => ({ items: old.items.map((n) => ({ ...n, pinned: false })) }),
+  initial: () => ({ items: [] }),
+  persist: true,
+  checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64,
+});
+

Documents migrate on access by the new definition: singletons during activation (hydration counts as access), keyed +ones on their next access. The migrated base is persisted with the next commit; right after a bump, a read-only +transaction writes it.

+

Chord does not walk values against schemas. Instead, each definition computes a schema fingerprint once +(descriptions and titles ignored), and every stored record keeps the fingerprint it was written with. On access Chord +compares them: the same version with a different fingerprint fails the object broken ("schema changed without a +version bump"). So every structural change needs a bump, an additive one with a trivial migrate. The one exception +to "no value checks" (R8): the result of migrate and of initial() is validated against the new schema once, when +produced. A failed check, a throwing migrate, or old code meeting a newer document refuses the access, keeps the +stored data and fails the owning object. A changed meaning without a schema change cannot be detected; that is on +you.

+

Storage

+

Behind the store is a DocumentStorage, passed as createHost({ storage }); memory by default. The server has one +shared storage, each worker one for its session.

+
interface DocumentStorage {
+  commit(writes: readonly DocumentWrite[]): Promise<Seq>;            // create/change/retire, atomic
+  find(address: DocumentAddress): Promise<DocumentRecord | undefined>;
+  load(id: string): Promise<StoredDocument | undefined>;             // version, value, deltasSinceBase
+  scan(query: { owner: string; definition: string }, limit: number, cursor?: string):
+    Promise<{ keys: string[]; next?: string }>;
+  byTag(tag: string, value: string, limit: number, cursor?: string): Promise<{ ids: string[]; next?: string }>;
+  mintId(): Promise<string>;
+}
+

Chord ships a memory storage and a SQLite storage (over @earendil-works/chord/sqlite, the driver layer that moved +from Durable; Node and Durable Object drivers). Storages check only metadata (incarnation, version, sequence, address +occupancy, fingerprint) and write the supplied delta or base; they never rebuild or validate content on a commit. +Chord documents never commit together with Durable transcripts: Chord has its own storage, not an adapter over +Durable's. Your own storage can prove itself with registerDocumentStorageConformance(runner, name, withStorage) from +@earendil-works/chord/testing.

+

Keyed persisted documents that nobody reads, writes or watches for 10 minutes leave memory and reload on the next +access. A storage error ends the store and closes the host; onError receives { kind: "storage_failed" } and the +app decides (chapter I).

+

Kinds of state

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
KindForAPI
documentdata the object ownsself.state, self.tx
derivecomputed valuesself.derive (chapter VIII)
source readableexternal data: Durable views, files, logsvalue plus ordered changes; feeds derives
paged methodlarge or queryable collectionsa method returning pages (chat history)
plugin databasefull-text search, large blobsa native opened by its owner (chapter IX)
+

App state lives in Chord documents. Only transcript-bound data, which must fork and rewind with a conversation, lives +in Durable.

+
Review: does an absent singleton read undefined or initial()? (Q11, open)

Design §9 says "whether a document exists is read outside the transaction (handle.value / handle.get give +undefined for an absent one)" and also "a singleton that does not exist yet reads initial() through its handle; +retiring resets it to initial()". Review round 3 proposes: singletons always read initial(), only absent keyed +documents read undefined, and singleton existence needs no API. This guide follows the proposal (list.value is +never undefined); it is not decided.

+
+
Built so far

Documents, transactions and storages are layer L3, a port of Durable's session store; not started. The frozen kernel +has a working version with a JSONL storage, which the design rejected.

+
+
+ +
VI

Holding On

References, the object lifecycle and ctx: what a caller sees while objects start, fail, restart and move, and why nobody writes retry code.

+

A tab shows the notes of S1 for an hour. During that hour the laptop sleeps, the worker is stopped for idleness and +started again, the notes plugin reloads twice and the store's storage hiccups once. The panel should show the notes the +whole time, say "reconnecting" when it cannot reach them, and recover by itself. In Chord this is what a +reference does; plugin code contains no retry loops and no request ids.

+

Three ways to hold

+ + + + + + + + + + + + + + + + + + + + + + + +
WhereAPIReleased
a factoryself.get(Type, input)when the object ends
a callawait ctx.get(Type, input, { state? })when the call ends
outside objectsawait host.get(Type, input, { signal, state?, parkTimeout? })ref.release()
+

Use self.get for what an object needs its whole life (the view's store), ctx.get for what one call needs, and +host.get only in UI code, tests and entry points. { state: false } binds for methods only, without receiving state. +A reference keeps its object alive while held.

+

The object lifecycle

+
(get) → activating → live → ending → ended
+             └→ failed (backoff) → activating …
+

Activating: Chord binds the declared references and waits until they are live and hydrated, loads singleton +documents, opens natives (one after another, in declaration order), computes derives and starts routines. The object +goes live when all of that succeeded. Calls wait behind its gate until then.

+

Ending: routines stop, the object's calls and jobs are aborted as their rules say, natives close in reverse order, +references are released. An object with lifetime: "referenced" (the default) ends when its last reference goes, +after the type's linger if it has one. A lifetime: "host" object (only for types without a required input) lives +as long as the host. A running, finishing or blocked job counts as a reference (chapter XI).

+

ref.state

+

Every reference reports what is going on as a tagged value; listeners get each new one:

+
type RefState =
+  | { kind: "loading" }                                                // first hold, reconnecting, restarting
+  | { kind: "live" }
+  | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // undefined: no automatic retry
+  | { kind: "released" };
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SituationCallsValues
link droppark until reconnectedreadable, loading
object loading or restartingpark until livereadable, loading
rollout: provider ahead or installingpark until installedreadable
object failedreject at once with the causefailed, with cause and retryAt
+

Values stay readable through all of this: the panel keeps showing the last notes. Parked calls share one deadline (60 +s by default, parkTimeout per get), then fail unavailable. Otherwise a call ends only through its caller's +signal; there is no per-call deadline. A call that was running on an instance that ends fails with an error and is +not resent: Chord cannot know whether it took effect.

+

Errors look like disconnects. A consumer handles a failed call exactly as a dropped link: show the error, let the +user retry. Work that must not run twice carries its own idempotency key as data (Durable's requestId, a job's +key).

+
The worker restarts while a tab watches the notes

Watching

The tab holds Notes { S1 }. ref.state is live; notes.notes.value has 40 notes.

+

The worker goes away

The supervisor stops worker S1 (or it crashes). The server's IPC link to it drops. The server tells the tab that its +hold is gone; the tab's reference goes loading. The 40 notes stay readable.

+

A click meanwhile

Ann's colleague clicks Add. The call parks on the reference; nothing is sent yet. The deadline is 60 s.

+

Chord resolves again

Holds on S1 still exist, so Chord calls route(Session).resolve({ id: "S1" }) again, with the per-address backoff. +The app forks a new worker: a new incarnation of Session({ id: "S1" }).

+

Re-hold

The server re-holds every hold on S1 over the new link. The new worker activates NoteStore { S1 } (its notes load +from storage) and the views; it sends fresh snapshots. The tab's reference goes live.

+

The parked call runs

The parked add is sent now and succeeds. The panel shows 41 notes. Nobody wrote recovery code.

+
+

Failure and retry

+

Any broken part fails the whole object: a half-working object could break rules that rely on its documents +changing together. Chord tags the cause by where it failed:

+
type FailureCause =
+  | { kind: "transient"; error: ChordError }   // an open threw, a dependency was unreachable: automatic backoff
+  | { kind: "broken"; error: ChordError };     // factory, derive, hydration or migrate threw: retried only by a trigger
+

Transient causes get a backoff per address: 1 s, doubling to 60 s; concurrent gets share one attempt; success +resets it. Broken causes would fail the same way again (a code bug, invalid data), so they are retried only by a +trigger: a plugin reload, ref.retry() (a Retry button), or a readable the factory declared with +self.retryOn(...readables). Only an opener may declare its own failure broken, with throw broken(error), for +example for invalid configuration. A failed reference stays held; on success it goes loading, then live.

+
host.serve(EnvConnection, ({ env }, self) => {
+  const config = self.get(EnvConfig, {});                 // a facet or registry value, outlives the attempt
+  self.retryOn(config.envs);                              // a config change retries a broken open
+  const ssh = self.open(async ({ signal }) => {
+    const c = config.envs.value[env];
+    if (!c) throw broken(new Error(`unknown environment ${env}`));
+    return connect(c, signal);
+  }, (s) => s.close());
+  …
+});
+

retryOn sources must live independently of the attempt (facet documents, registry values); a member of one of the +object's own references is a type error, since it goes away with the attempt.

+

Waiting for dependencies

+

An object whose references are not live yet stays loading and goes live the moment they are, bounded by +readyTimeout (60 s). The decision for the failure case (R18) is what makes recovery automatic:

+
tabNotes { S1 }NoteStore { S1 }holdself.get (activation)open fails: disk not mountedfailed (transient)failed: transient, cause insidethe view keeps its reference; its state is the retry triggerbackoff retry: open succeedsliveretried at once (backoff cancelled)live
Figure 6. Dependency recovery. The view fails transient with the store's cause, keeps its reference, and is retried the moment the store goes live.
+

A dependency that fails, or does not become live in time, fails the dependent transient with the dependency's +cause inside, never broken: the cause is the dependency's. The dependent keeps its references while failed, and Chord +uses their state as a retry trigger. Factory authors write self.get and know none of this. After an object is live, +a dependency leaving live does not change it: its calls that use the dependency get the dependency's error, or park +while it loads.

+

Cycles are rejected (Q5). If A's activation gets B and B's gets A, neither would ever lose its last reference. +Every object instance has a uuidv7 instance id; on one host Chord follows the waiting edges of activating objects, and +across hosts a hold made during an activation carries the chain of instance ids. A cycle fails the activation +broken, naming it ("A → B → A"), and keeps no references.

+

No optional references

+

get has no optional flag. When a dependency's plugin may be missing, turn the dependency around: the dependent +owns a registry and the other plugin contributes to it. Radius uses forwards from the ports plugin, but ports is +optional, so radius owns radius.forwards and ports contributes its forwards; without ports the table is empty and +radius works (chapter XIII).

+

ctx

+

Every call gets a ctx as its last argument:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldMeaning
signalfires when the caller cancelled or the object ended; pass it to everything you wait on
principalthe caller (remote types only)
incarnationthe calling host's incarnation, for calls that arrived over a link (chapter X)
getreferences released when the call ends
telemetry, trace(name, attributes, fn)the current span; a child span with a child ctx (chapter XVII)
jobId, progressin a job's run (chapter XI)
+
exportText: async ({}, ctx) => {
+  const catalog = await ctx.get(Catalog, {});                     // released when the call ends
+  const title = catalog.sessions.value[session]?.title ?? session;
+  const res = await fetch(renderUrl, { method: "POST", body: render(title, list.value), signal: ctx.signal });
+  return await res.text();
+},
+

There are no code versions and no request ids in ctx. Durable's Context is the same module (chord/context): +ctx extends it, so passing ctx into pi-durable needs no conversion. Code outside objects gets a ctx from +host.root(fn, { principal?, signal? }); inside Durable tools it uses chord(fn, context) (chapter XIV).

+
Built so far

References, ctx and the lifecycle are layers L4 and L5. ChordError and FailureCause are specified in the L1 +draft (layer-1.md), with codes invalid, contract_mismatch, forbidden, gone, not_ready, released, +too_large, unavailable and aborted, and a finer reason such as invalid/args.

+
+
+ +
VII

Taking Access Away

Revocation: how removing a guest ends their objects on every host at once, and why reconnecting does not bring access back.

+

The owner removes Ann from S1. Her tab still has a WebSocket open and a Notes { S1 } view on worker S1. Both must end +now, on the server and on the worker, and her tab's automatic reconnect must not restore anything. Chord provides the +mechanism; the app decides who is blocked and remembers it.

+

Labels, not predicates

+

The app declares, once, which revocation labels each principal carries. Revoking means revoking one label; +everything carrying it is hit.

+
// studio/src/principal.ts
+export type Revocation = { grant: string } | { device: string } | { session: string };
+
+export const revokedBy = (p: P): Revocation[] =>
+  p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }]
+  : p.kind === "owner" && p.access === "device" && p.device ? [{ device: p.device }]
+  : [];
+
+const server = createHost<P, Revocation>({ address: Server(), principal: { kind: "server" }, revokedBy });
+// every worker is created with the same revokedBy
+ + + + + + + + + + + + + + + + + + + + + + + +
PrincipalLabels
guest Ann (grant g1) in S1{ grant: "g1" }, { session: "S1" }
guest Bob (grant g2) in S1{ grant: "g2" }, { session: "S1" }
owner on the paired phone{ device: "phone" }
owner, localnone
+ + + + + + + + + + + + + + + + + + + +
CallHits
server.revoke({ grant: "g1" }): remove AnnAnn
server.revoke({ session: "S1" }): archive S1Ann and Bob
server.revoke({ device: "phone" }): sign out the phonethe phone
+

A label is plain data because revocation is an event that must travel from the server to the workers; a predicate +function could not be sent.

+

What revoke does

+

host.revoke(r) ends every object and closes every link whose principal p has r in revokedBy(p) (compared by +canonical JSON; never the host's own principal), and passes r to its children, which apply their own revokedBy.

+
owner tabserverworker S1Ann's tabguest g1Sharing.remove({ grant: "g1" })app: mark g1 blocked (document)revoke({ grant: "g1" })close linkrevoke { grant: "g1" }end Notes { S1 } as g1reconnectauthenticate: g1 blockedrefused (4401)
Figure 7. Removing guest Ann. The label travels down; her objects end on every host; her reconnect is refused by the app's own record.
+

On worker S1, for revoke({ grant: "g1" }):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ObjectPrincipalHit
Notes { S1 }owner, localno
Notes { S1 }guest g1yes: ends
Notes { S1 }guest g2no
NoteStore { S1 }none (local)no
+

Ending Ann's view releases its reference to the store, which keeps serving the others.

+

Chord persists nothing here

+

revoke cuts live access only. Lasting denial lives in the app's data (the sharing plugin's guest records), which +authenticate checks on every new connection. After a server restart nothing live exists to revoke; every client +authenticates again and blocked grants are refused there.

+

No re-stamping. A changed grant (Ann moves from read to write) is a revocation of the old grant. Her link closes, +her tab reconnects and authenticates with the new grant revision, which gives a new principal key, and her references +re-hold by themselves, now as an editor. Reconnecting never restores revoked authority.

+

Revocation beats jobs (R11). Normally a running job keeps its object alive after the last reference is released +(a closed tab does not abort an import). Revoking a principal ends its objects at once regardless; their jobs get the +abort signal and the normal close timeout.

+

Work in shared stores

+

Revocation ends objects whose principal carries the label. A shared store has no principal, so work it runs on +behalf of someone (a question waiting for Ann's answer, a shell command a guest started) is not ended by revoke. +Such a store keeps the grant as data with the work and reconciles from the grant owner's live state with a routine +(chapter X):

+
// in a shared store that runs work for guests
+const sharing = self.get(Grants, {});                              // the sharing plugin's live grants
+self.routines.on(sharing.live, async (live, ctx) => {
+  for (const run of shell.running.value)
+    if (run.args.grant && !live[run.args.grant]) shell.abort(run.id);   // idempotent
+});
+

There is no revocation hook: a hook can be missed (a crash between revoke and hook), a routine over current state +cannot. The window between the revocation and the routine's run is accepted (R10); a step started in it still runs.

+

Who does what

+ + + + + + + + + + + + + + + + + + + +
ChordThe app
stamps principals on linkswhat a grant is; share links, cookies, authenticate
keeps one object per principal keydefines Revocation and revokedBy
on revoke(r): ends matching objects and links, passes r downdecides when to revoke; stores who is blocked
+
+ +
VIII

Computed Values

Derives, the one footgun that makes them expensive, mapShared, and presence: who is looking.

+

The reader's notes without authors, a note count, the number of people viewing: values computed from other values. +Storing copies would go stale; recomputing them on every read would be wasteful and could not be watched. A +derive is computed once per change and sent to viewers as a difference.

+

self.derive

+
const count = self.derive("count", [list], ([l]) => l.items.length);
+const summary = self.derive("summary", [list, store.pinned], ([l, pinned], previous) => ({
+  total: l.items.length,
+  pinned: pinned.length,
+}));
+

self.derive(name, sources, (values, previous) => value). Sources can be documents, readables, other derives and +state members of references. The value is recomputed when a source changes and diffed into operations for the wire. +Derived values are adopted before ordinary observers run and share their source commit's wire batch, so a viewer +never sees a note list and a count from different commits.

+

previous is the derive's last value, kept across factory re-runs (reloads) by name. It is a cache, empty after a +host teardown, so a derive must be correct without it. A derive that throws fails its whole owner broken.

+

How the difference is found

+

Chord compares the new value with the previous one using Delta's diffRevisions. Parts that are the same objects +as last time cost one === each. Parts that are new objects must be compared by content.

+

The footgun. A derive that builds new objects for unchanged data makes every commit cost the size of the whole +value, per view. The reader list from chapter IV, written naively:

+
self.derive("public", [list], ([l]) => ({ items: l.items.map((n) => ({ id: n.id, text: n.text, at: n.at })) }));
+// one note added to 2,000: 2,001 new objects, every one compared by content
+

mapShared(f) keeps one result per source element: f runs only for new or changed elements, unchanged ones +return their previous result object.

+
import { mapShared } from "@earendil-works/chord/delta";
+
+const stripAuthor = mapShared((n: Note): PublicNote => ({ id: n.id, text: n.text, at: n.at }));
+self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) }));
+// one note added: one new object, 2,000 identical references; the diff emits one insert
+
Review: mapShared is planned, not available

Design §12 introduces mapShared as coming "from @earendil-works/chord/delta", but nothing in packages/chord/src +defines it yet, and Delta does not yet treat bytes as atomic leaves in diffRevisions. Design §28 lists both as the +step "Delta in pi", before the layered rebuild. Until then, keep a Map from source element to result yourself, which +is what mapShared will do.

+
+

Cookbook

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CaseHow
select a field, look up by key, a small aggregatecompute it; the value is small
filter or map a listmapShared over the elements
redactionpreferably separate documents per visibility; otherwise mapShared, computed once in the shared store, never per view
join two sourcesmapShared plus the looked-up source's revision
a window over a large listtake a section of the source elements without rebuilding them; cache per-element measurements by identity; keep the window's start stable so changes arrive as appends
a large view combined from several sourcesre-path the sources' operations, as Durable's ConversationView does; application code never interprets ops
+

Container width (R24): structural sharing is O(change) along a path, not across a container's width. Changing one +item of a flat array copies the array's slots, and mapShared visits every element once (one identity comparison +each). Large collections are keyed documents or bounded windows. A benchmark of long arrays is to set the documented +limit.

+

Readables that are not documents

+

Three small readables round out the toolkit:

+
    +
  • constant(value): never changes. Start data uses it (chapter IX).
    • +
  • self.writable(initial): a host-local readable of any in-process value (functions included), compared by identity, +with set. It never goes on the wire; it exists for registry contributions that change (chapter XIII).
    • +
  • Source readables from elsewhere: a Durable view or document watch is already a Chord readable, because Durable and +Chord v2 share the readable type (chord/src/state.ts), so it can feed a derive or be exposed as state directly.
    • +
+
Review: exposing a writable (Q14, open)

A writable may hold functions; nothing yet stops a factory from returning one as a state member, which would reach +the wire encoder. Review round 3 proposes that only members with a JSON schema in the contract are exposable, checked +when the object is served. Undecided.

+
+

Presence: who is looking

+

host.references(Type) is a readable of who holds references to objects of a type on this host: input, principal and +client. Client holds carry an opaque origin id, so two tabs are two clients.

+
host.serve(NoteStore, ({ session }, self) => {
+  const viewers = self.derive("viewers", [host.references(Notes)], ([refs]) =>
+    refs.filter((r) => r.input.session === session).length);
+  return { …, viewers };
+});
+

A link drop is a departure: accurate and simple. Smoothing (a phone switching networks makes the count flicker) is the +UI's job, for example a debounced count. Counts per person are derived by the app from principals and origins. There +is no consumer-side linger: a closed tab closes its link and keeps nothing. The provider's linger keeps a +just-closed object warm, and that also keeps the worker alive through host.activity.

+

"Leaving" is never derived from an unmount. A UI reload unmounts and remounts components; a component unmounted by +a reload has not left. Leaving is explicit navigation, or presence (a dropped reference). This replaces p2's presence, +viewer and login-abandon leases with one mechanism.

+
Built so far

Readables and derives are layer L2 (a readable is today's replicatedState, derives reuse today's derived cells over +it). Not started in the rebuild.

+
+
+ +
IX

Things the OS Gives You

Start data, host-lifetime objects and natives: how a plugin opens a file watcher or a socket so that it is closed exactly once, also across reloads.

+

Studio's notes get an inbox: a text file dropped into the session's inbox/ directory becomes a note. That needs a +file watcher, a native: an OS-level value (a file handle, a socket, a process, a database connection) that cannot +be JSON, cannot be copied, must be opened, and must be closed exactly once. It also needs the session's data +directory, which the plugin does not know. This chapter covers both.

+

Start data is an object

+

A worker knows things before any plugin loads: its session id, the data directory, init data from the server's start +message. There is no special mechanism for them. Start data is an ordinary local object with lifetime: "host" and +constant state, served by base code's core facet:

+
// studio/src/core/contract.ts
+export const Init = obj({ id: str(), dataDir: str(), cwd: str(), env: opt(str()) });
+export const SessionInfo = local("studio.session-info", {
+  version: 1,
+  input: obj({}),
+  members: { init: Init },
+});
+
+// studio/src/core/worker.ts
+export const coreWorkerFacet = (init: Static<typeof Init>) => defineFacet((host) => {
+  host.serve(SessionInfo, () => ({ init: constant(init) }), { lifetime: "host" });
+  // … the Harness and other core objects (chapter XIV)
+});
+

Any plugin on the worker declares a reference and reads it after activation: in an opener, a derive, a routine or +a method, never synchronously in the factory.

+
const info = self.get(SessionInfo, {});
+// later: info.init.value.id, info.init.value.dataDir
+

Paths are derived by the plugin from generic start data (join(dataDir, id, "notes-inbox")), never provided per +plugin. Platform bindings work the same way: on a Durable Object, a core object's self.open returns the binding +(state, env) and its close does nothing.

+

self.open

+
const watcher = self.open(
+  async ({ signal }) => watchDir(path, { signal }),       // open: during activation
+  (w, reason) => w.close(),                               // close: when the object ends or restarts
+  { closeTimeout: 10_000 },                               // optional; host default 30 s
+);
+watcher.value;                                            // the watcher; reading it before activation throws not_ready
+

self.open returns a handle with .value (type Native<T>), not a proxy. The rules:

+
    +
  • Natives live only in objects. Not at module level, not in the facet. A module-level watcher would survive a +reload with nobody to close it.
    • +
  • Order: natives open one after another in declaration order (a later opener may read an earlier handle's +.value) and close in reverse. The object goes live after all opens succeeded.
    • +
  • The value never changes during an attempt. No replacement, no versions, no stale-handle logic. To change a live +native, the object restarts.
    • +
  • A failed open makes the object failed: transient with backoff by default, broken if the opener says so +(throw broken(error)). A partly failed activation closes the natives already open, with reason end; the next +attempt opens all of them anew.
    • +
  • close gets a reason: { kind: "reload" } (the plugin reloads and this native object restarts), { kind: "stop" } (the host shuts down) or { kind: "end" } (the last reference ended, or the type was removed). Each +close is bounded by closeTimeout, then reported as a possible leak while the rest continues. A throwing close +does not stop the others; host.close() collects the errors, background ends report to onError.
    • +
  • Behaviour around a native goes into routines (chapter X). Listeners on natives outside routines are forbidden.
    • +
+

An object with at least one self.open is a native object; one without is a pure object. The difference +decides what a reload does (chapter XV): pure objects are swapped in place, native objects restart.

+

The inbox

+

The inbox must run as long as the worker, whether anyone looks at the notes or not. So it is a lifetime: "host" +object. The facet cannot self.get a type it serves, and the inbox has no meaningful input, so it is an object +without input that reads its session from the start data:

+
// notes/contract.ts
+export const NoteInbox = local("notes.inbox", { version: 1, input: obj({}), members: {} });
+
+// notes/worker.ts
+host.serve(NoteInbox, (_input, self) => {
+  const info = self.get(SessionInfo, {});
+  const watcher = self.open(async ({ signal }) => {
+    const { id, dataDir } = info.init.value;                       // read in the opener: after activation
+    return watchDir(join(dataDir, id, "notes-inbox"), { signal });
+  }, (w) => w.close());
+
+  self.routines.loop("import", async (ctx) => {                    // chapter X
+    const store = await ctx.get(NoteStore, { session: info.init.value.id });
+    for await (const file of watcher.value.files(ctx.signal)) {
+      await store.add({ text: await readFile(file, "utf8"), by: "inbox" }, ctx);
+      await rm(file);
+    }
+  });
+  return {};
+}, { lifetime: "host" });
+

Note ctx.get(NoteStore, …) in the routine rather than self.get in the factory. The factory runs before the +session id is known (start data is read after activation), so a host-lifetime object without input gets +session-scoped objects with ctx.get in a routine or method (R22). A get whose input is computed after activation is +to be added only if a real case needs it.

+

Natives with work in flight

+

A restart is sequential (R16): close the old native, then open the new one. When the old one still has work running, +use one of two patterns:

+
    +
  • Drain in the background. close detaches the old resource and returns at once; it disposes of it when its work +finishes, or after closeTimeout at most. Chord opens the new one immediately. Code mode's sandbox does this: a +script running during a reload finishes on the old sandbox (for up to 30 s, Q7; a longer one is aborted and its turn +fails), new scripts start on the new one.
    • +
  • Swap through a registry. The resource is a registry contribution (chapter XIII). Readers always look up the +current one at use; a reader that must wait for one uses a cancellable "wait until present" with the park deadline.
    • +
+

Chord aborts only the object's own calls and jobs on close. Other work that uses the native (Durable tool calls on a +sandbox) is the owner's business: close may drain instead of tearing down. What an old native does after its owner +restarts is also the owner's close: a model runtime keeps working until collected, a closed SSH connection fails its +in-flight exec.

+

Natives that follow a relocation

+

A session can move to another directory. The inbox watcher must follow. Restarting the object on every move would be +heavy-handed; instead the native is a stable manager opened once, and a routine on the session location switches its +internal resource and bumps a generation, so late events from the old location are dropped:

+
const watcher = self.open(async ({ signal }) => new SwitchableWatcher({ signal }), (w) => w.close());
+self.routines.on(location.cwd, (cwd) => watcher.value.switchTo(join(cwd, ".notes-inbox")));
+

Call-scoped natives

+

A native that lives exactly as long as one call (a temporary SSH session for a directory listing, a file handle for a +download) is opened and closed inside the method with try/finally. There is no ctx.open:

+
list: async ({ path }, ctx) => {
+  const session = await ssh.open(config, ctx.signal);
+  try { return await session.readDir(path, ctx.signal); }
+  finally { session.close(); }
+},
+

Core vs. plugin

+

SessionInfo comes from base code, NoteInbox from a plugin; both use the same primitives. Where code lives decides +which it is. Base code (the app's src/, imported statically by the start code under a fixed name such as +@studio/core) is always there, not switchable, and a change restarts the hosts. A plugin (its own directory, +found by listing, named by its package.json) is switchable and reloads. Core is what the app always needs and +everything builds on: the harness, the session list, access, the session location. Optional features are plugins; +code mode is a plugin.

+
Built so far

Natives and host-lifetime objects are layer L4. constant is decided (R23), not built.

+
+
+ +
X

Standing Behaviour

Routines, reconciling instead of notifying, and how two owners keep a rule between them without a shared transaction.

+

Notes older than 30 days should go every hour. Inbox files should be imported as they arrive. Notes of an archived +session should become read-only. Each of these is behaviour that stands, not a call someone makes. Written as bare +setIntervals and event listeners, it would keep running old code after a reload and touch state from nowhere. Chord +has one declared form for it: the routine.

+

Four kinds

+
self.routines.every("cleanup", HOUR, (ctx) => dropOlderThan(30 * DAY, ctx));   // due time survives reloads
+self.routines.on(settings, async (value, ctx) => applyLimit(value, ctx));        // a readable or event source
+self.routines.loop("import", async (ctx) => {                                     // runs until the object ends
+  while (!ctx.signal.aborted) await importNext(ctx);
+});
+self.routines.custom("ticker", ({ signal, run }) => {                             // anything else
+  const timer = setInterval(() => run(tick), 1000);
+  return () => clearInterval(timer);                                              // cleanup undoes the setup
+});
+

Routines start when the object goes live, stop when it ends, and restart with the new code on every reload of their +plugin. Each run gets its own ctx; errors are caught and reported, never uncaught. Routines are named (on +optionally by its source) so Chord recognizes them across reloads; that is how every keeps its due time.

+

Serialized per routine

+

The new instance of a routine starts only after the old handler in flight has settled; only that routine waits, for at +most the routine handover timeout (10 s), after which it starts anyway and the stuck handler is reported.

+

The reason, as a trace: a supervisor routine reconciles pending worker restarts. The old handler is midway through +restart(W1) when the plugin reloads. Without serialization the new routine starts at once, sees W1 still pending, and +restarts it a second time.

+

Reconcile, don't track

+

A routine reconciles from the current value: it makes the world match what it sees now. It writes no "in +progress" flags, and it starts a job for long work (chapter XI). A handler that only cares about the latest value +compares with the current value itself; there is no overlap option. Subscribing to the object's own documents is +gapless: every commit is seen.

+

No bare timers or callbacks that touch state outside routines, jobs or calls. That is what lets Chord stop all of an +object's behaviour when it ends or reloads.

+

Rules between two owners

+

Every thing has exactly one owner, the smallest object it belongs to. Some rules span two: "notes of an archived +session are read-only" involves the catalog (which knows archiving) and NoteStore (which holds notes). Two tempting +designs are wrong:

+
    +
  • A cross-owner transaction (archive and lock the notes atomically) would make the catalog know every plugin and +couple their storage.
    • +
  • A notification (catalog calls notes.archive(session)) can be lost: the worker is not running, the link drops, +the process crashes between the archive and the call.
    • +
+

The design's rule: the owner of the dependent data enforces the rule by reconciling from the other owner's state +with an idempotent routine.

+
// NoteStore: an archived session's notes are read-only
+const flags = self.state(NoteFlags);                         // { archived: boolean }, NoteStore's own document
+const catalog = self.get(Catalog, {});                       // the server's session list: another owner
+self.routines.on(catalog.sessions, async (sessions, ctx) => {
+  const archived = sessions[session]?.archived ?? false;
+  if (flags.value.archived !== archived) await flags.change((d) => { d.archived = archived; }, ctx);
+});
+// add and remove refuse while flags.value.archived
+

After a crash the routine runs again with the current value. The catalog does not know the notes plugin exists. The +same pattern closes a forward of an archived session in the ports plugin, aborts a job whose grant was revoked +(chapter VII), and keeps any invariant between plugins.

+

Reading without starting a worker

+

The session list in the sidebar shows each session's note count. Getting NoteStore { S1 } for every session would +start every worker. The answer is plain publishing between two objects of one plugin, no Chord mechanism: the worker +side reports a summary to a server-side object by a method call from a routine, and clients read the server object.

+
// notes/contract.ts
+export const NoteCounts = remote("notes.counts", {
+  version: 1, input: obj({}), placement: () => Server(),
+  variants: {
+    owner: { state: { counts: map(int()) } },
+    worker: { methods: { report: method(obj({ session: str(), count: int() })) } },
+  },
+});
+
+// notes/server.ts
+const Counts = defineState("notes.counts", {
+  schema: obj({ bySession: map(obj({ count: int(), incarnation: str() })) }),
+  version: 1, initial: () => ({ bySession: {} }), persist: true,     // survives a server restart
+  checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64,
+});
+
+host.serve(NoteCountStore, (_input, self) => {                        // local, lifetime "host"
+  const counts = self.state(Counts);
+  return {
+    counts: self.derive("counts", [counts], ([c]) => mapValues(c.bySession, (e) => e.count)),
+    report: ({ session, count, incarnation }, ctx) =>
+      counts.change((d) => {
+        const last = d.bySession[session];
+        if (last && last.incarnation > incarnation) return;               // uuidv7: an older worker, ignore
+        d.bySession[session] = { count, incarnation };
+      }, ctx),
+  };
+}, { lifetime: "host" });
+
+host.serve(NoteCounts, (_input, principal, self) => {
+  const store = self.get(NoteCountStore, {});
+  if (principal.kind === "worker") return {
+    variant: "worker",
+    report: ({ session, count }, ctx) => {
+      if (session !== principal.session) throw forbidden();               // a worker reports only itself
+      return store.report({ session, count, incarnation: ctx.incarnation }, ctx);
+    },
+  };
+  return { variant: "owner", counts: store.counts };
+});
+
+// notes/worker.ts, inside NoteStore
+const counts = self.get(NoteCounts, {});                                   // a reference to the server
+self.routines.on(count, (n, ctx) => inVariant(counts, "worker") && counts.report({ session, count: n }, ctx));
+

The server object keeps the latest value per session (in a document if it must survive a server restart) and +ignores reports from an older worker incarnation (incarnations are uuidv7, so they order by start time). A worker that was replaced may still deliver a late report; the +fence is the caller's incarnation, which Chord stamps on every call from the link handshake (R20): ctx.incarnation, +next to ctx.principal, not part of the principal or its key, never sent as data, so it cannot be forged. Session +status, subagent lists, unread marks and artifact lists in p2 all follow this pattern. The store is local and has no +caller incarnation of its own, so the view passes it on as data.

+
Review: ctx.incarnation across a relay (Q13, open)

A worker's call to the server arrives directly, so ctx.incarnation is the worker's. But a call relayed through the +server (a tab's call to a worker) carries the server's incarnation on the worker's side, because the worker's +handshake is with the server. Review round 3 proposes that relays stamp the origin host's incarnation, as they stamp +the principal. Undecided; the pattern above only needs the direct case.

+
+
Built so far

Routines are layer L8. The design treats every as one of four forms over one runner; review round 3 proposed cutting +to on and loop only (C4), not decided.

+
+
+ +
XI

Work That Outlives the Call

Jobs: owned work started by a method that keeps running after its caller leaves, with cancel, dedupe and a recorded outcome.

+

Importing a 50 MB text file into notes takes a minute. The tab that started it may close; the import must go on, be +cancellable, not run twice when the tab retries after a link blip, and its result must be recorded. A method cannot do +this (it dies with its caller), a routine is standing behaviour rather than a request, and a Durable task is for work +that must resume after a crash. Chord's form for it is the job, built in pi-durable's sandwich: intent (a +transaction), effect (anything), outcome (a transaction in the current code).

+

Defining and running a job

+
// notes/worker.ts
+const ImportJob = defineJob("notes.import", {
+  version: 1,
+  args: obj({ path: str() }),
+  progress: obj({ lines: int() }),
+  result: obj({ added: int() }),
+  concurrency: 1,                                         // one import at a time per NoteStore
+});
+
+const ImportResults = defineState("notes.import-result", {
+  schema: obj({ outcome: union([obj({ added: int() }), obj({ error: str() })]) }),
+  version: 1, initial: () => ({ outcome: { added: 0 } }), keyed: true,
+  checkpointWhen: () => true,
+});
+
+host.serve(NoteStore, ({ session }, self) => {
+  const list = self.state(NoteList);
+  const results = self.state(ImportResults);
+  const imports = self.job(ImportJob, {
+    run: async ({ path }, ctx) => {                                       // 2. effect
+      let lines = 0;
+      for await (const line of readLines(path, ctx.signal)) {
+        await addNote({ id: `${ctx.jobId}:${lines}`, text: line, by: "import" }, ctx);   // keyed by job id
+        ctx.progress({ lines: ++lines });
+      }
+      return { added: lines };
+    },
+    finish: async (outcome, tx) => {                                      // 3. outcome, current code
+      (await tx.draft(results, outcome.id)).outcome =
+        outcome.kind === "done" ? outcome.result : { error: outcome.error };
+    },
+  });
+  return {
+    …,
+    imports: imports.running,                                             // live list of runs
+    startImport: ({ path, requestId }, ctx) =>
+      imports.start({ path }, ctx, { key: requestId }),                   // 1. intent (refined below)
+    abortImport: ({ id }) => imports.abort(id),
+  };
+});
+

defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? }); progress and the outcome +are typed by the schemas.

+
tabNoteStore { S1 }runeffectstartImport({ path, requestId })1. intent: busy check, dedupe by key, commit{ id }start after the commitread lines, add notes (keyed by jobId)progress { lines } (throttled)the tab may close here: the job counts as a referencesettles: { kind: "done", result }3. record outcome, finish(outcome, tx), ack
Figure 8. The job sandwich. The intent commits before anything runs; the effect runs outside any transaction; the outcome commits in the current code.
+

The three steps

+

1. Intent. job.start(args, ctx, { key? }) is shorthand for tx.start(job, args, { key? }) inside a +transaction, which may contain other writes. The busy check (concurrency) and deduplication (the same key returns the +existing run) are atomic and in the same commit; run starts only after the commit. Run ids are uuidv7.

+

2. Effect. run(args, ctx) may do anything: processes, network, other objects. It writes no document itself; it +returns the outcome. External writes carry a dedupe key derived from ctx.jobId (above: note ids), so an interrupted +attempt cannot apply them twice. On a reload run keeps its old code.

+

3. Outcome. When run settles, Chord records the outcome in its own record, then calls the current code's +finish(outcome, tx) and acknowledges the run in the same commit. The outcome is tagged:

+
type Outcome<R> = { id: string } & ({ kind: "done"; result: R } | { kind: "failed"; error: "threw" | "aborted" | "lost" });
+

A throwing finish changes nothing and is retried with a backoff (1 s to 60 s), and after the next reload. finish +is optional and follows Durable's transaction rule.

+

Dedupe ends with the run

+

Deduplication covers unsettled runs only (Q9). Once finish has committed, Chord drops the run's record and key +and releases its waiters' references; it keeps nothing beyond a run's lifetime. Dedupe after completion is the +owner's. Studio records which run a request started, in the same commit as the intent, and finish records the +result by run id (ImportResults above):

+
const Requests = defineState("notes.import-request", {
+  schema: obj({ run: str() }), version: 1, initial: () => ({ run: "" }), keyed: true, persist: true,
+  checkpointWhen: () => true,
+});
+
+startImport: async ({ path, requestId }, ctx) => {
+  const prior = await requests.get(requestId);                  // read outside the transaction (R13)
+  if (prior) return { id: prior.run };                          // started before, maybe finished long ago
+  return self.tx(async (tx) => {
+    const run = await tx.start(imports, { path }, { key: requestId });   // dedupes a concurrent retry
+    (await tx.draft(requests, requestId, { tags: [SessionTag(session)] })).run = run.id;
+    return { id: run.id };
+  }, ctx);
+},
+

Two concurrent retries both miss prior, but the key makes tx.start return the same unsettled run. The client +watches imports for progress and reads the result by run id once it is gone from running.

+

Why Chord does not keep completed runs itself: either 100,000 keyed imports would stay in memory forever, or a +retention limit would let a retried key rerun its effect. The owner knows which results matter.

+

Watching, waiting, aborting

+
    +
  • running is a readable of { id, version, args, progress, phase }. phase is running, finishing (outcome +recorded, finish not yet committed) or blocked (with the reason). concurrency counts only active runs. The +panel shows a progress bar from it.
    • +
  • Progress is in memory, throttled and sent as deltas. For output, use OutputBuffer and Progress from +@earendil-works/chord/output: bounded windows, adaptive throttling, no write amplification.
    • +
  • Waiting: start returns a run with outcome(ctx). The caller's abort ends only its wait, never the run.
    • +
  • Abort is a request: job.abort(id) fires the run's ctx.signal; the run decides what it returns (a result +that already arrived stays a success). A run that ignores the signal keeps its slot until it ends. finish runs in +Chord's own context.
    • +
+

Lifetime, persistence, revocation

+

Jobs keep their owner alive. A running, finishing or blocked job counts as a reference to its object, which ends +only once all its jobs are settled. Starting an import and closing the tab does not abort it. Unsettled runs also +count as activity for the worker's keep-alive.

+

Revocation takes precedence (R11). Revoking a principal ends its objects at once, whatever jobs remain; the jobs +get the abort signal and the normal close timeout. A job runs as its object's principal.

+

Persistence: jobs are in memory by default and gone with the process. persist: true (only for stable +identities) keeps the record: a run live at a crash ends lost at the next activation and finish records it. +Nothing is rerun automatically.

+

Versions: a run started under an older job version keeps its version; finish and running see it through pure +accepts adapters, and stored outcomes are never rewritten. A version bump without an adapter does not refuse the +reload: the old outcome is blocked, kept and reported until a later reload brings the adapter.

+

Native objects: a job in a native object ends aborted when the object restarts on a reload (chapter XV). In a +pure object it keeps running on its old run code.

+

What is not a job

+ + + + + + + + + + + + + + + + + + + + + + + +
WorkUse
request and response that may die with its callera method
input into running work (a pasted code, an approval)a flow (chapter XII)
work that must resume after a crasha Durable task
standing behavioura routine
+
Review: jobs are the largest open simplification (#9, C3)

Decision #9 (deferred until login is ported): a login or a long relocation is a job whose state consumers want to +address and watch (LoginFlow { id }). Options: (a) jobs stay records inside their owner, watched through running; +(b) a job is an object of its own, addressed by its id, living until done. Separately, review round 3 proposed cut C3: +shrink jobs to a runner over documents (no job versions, accepts, blocked, persist) or move them out of the +kernel. Not decided. chord-next-walkthrough.md concept 17 still shows the older outcome shape { ok: true, result }; +the design tags outcomes by kind.

+
+
+ +
XII

Asking a Person

Flows and the Questions service: how running work waits for an answer from someone who may be on any tab, may reconnect, and may answer twice.

+

A provider login asks for a pasted code, then an account choice. A dangerous tool call waits for the owner's +approval. The agent asks everyone present a question. In each case running work waits for a person, who may be on any +tab, may lose the connection halfway, may answer late or twice. Two patterns cover this: a flow kept as its +owner's state, and the Questions service, a shared store for questions.

+

A flow is state, not a call

+

The tempting design is a long-held server call, step(), that blocks until the next step. A link drop fails it, and +the consumer cannot tell whether it missed a step. Chord's answer is that the owner keeps the flow as state, which +re-holds by itself after any drop:

+
// models/contract.ts (sketch from the design)
+start:  method(obj({ provider: str() }), obj({ id: str() }))              // uuidv7
+answer: method(obj({ id: str(), step: int(), value: str() }), Answered)   // ok | "step no longer active"
+cancel: method(obj({ id: str() }))
+
+// LoginFlow { id }: state
+type FlowState =
+  | { kind: "running"; step?: { id: number; prompt: LoginPrompt }; notices: Notice[] }
+  | { kind: "done" | "failed" | "cancelled"; notices: Notice[] };
+

The UI watches the state and renders the current step. A late or second answer to a step that is gone is refused. A +promise-based callback inside the work (pi-ai's prompt during a login) stays inside the owner: its job keeps the +resolver, and nothing crosses an object boundary.

+

Imperative consumers (tests, scripts, a CLI, an agent tool) use a generic client helper over the same state, not a +second protocol:

+
async function* steps(flow: LoginFlowRef, ctx: Ctx) {
+  let last = 0;
+  for (;;) {
+    const s = await until(flow.state, (v) => v.kind !== "running" || (v.step !== undefined && v.step.id > last), ctx);
+    if (s.kind !== "running") return s.kind;
+    last = s.step!.id;
+    yield s.step!;
+  }
+}
+
+for await (const step of steps(await ctx.get(LoginFlow, { id }), ctx))
+  await login.answer({ id, step: step.id, value: answerFor(step) }, ctx);
+

last keeps steps from being lost or repeated across re-holds.

+
Review: where a flow lives (#11, deferred)

Decision #11, deferred until login is ported: (a) a keyed document of its owner, or (b) an object LoginFlow { id } +living until done; tied to #9 (jobs as objects). The LoginFlow { id } address above assumes (b). Also undecided is +how "leaving" ends a login (p2 cancels when the starting owner leaves everywhere); with presence it is a dropped +reference, never an unmount (chapter VIII).

+
+

The Questions service

+

The Questions service (apps/p2/docs/questions.md) is a p2 design, not a Chord feature, built from Chord's +primitives: a shared store with per-principal views. The asker gets ask; viewers get the open questions and +answer.

+
export type Question =
+  | { kind: "text"; message: string; secret?: boolean; help?: string }
+  | { kind: "select"; message: string; options: readonly { id: string; label: string }[] }
+  | { kind: "confirm"; message: string; allow: string; deny: string };
+
+type Policy = { mode: "first" } | { mode: "quorum"; count: number } | { mode: "all"; join?: "live" | "snapshot" };
+
+ask(question, {
+  audience,                // a readable or a list of principals
+  policy,
+  key?,                    // stable per asking flow: a re-ask reattaches to the pending question
+  deadline?,               // then resolve with what is there; `missing` names who did not answer
+}, ctx): Promise<Result>;
+

Its rules:

+
    +
  • The store is the truth. Pending questions and their resolvers live in the store; open is a readable over it, +filtered per viewer. Nothing is a document, so a reload or migration cannot leave a ghost question.
    • +
  • One settlement point. Register the waiter and the abort listener before publishing the question; an answer, an +abort or the deadline atomically takes the entry out before resolving. A late or second answer gets "no longer +active". Answers are validated against the stored question (a select id must be one of the options). Never await an +answer inside a transaction.
    • +
  • Policies over a live audience. first resolves with the first answer; quorum(n) after n answers; all when +every member of the audience has answered. With join: "live" (the default), someone who joins becomes required and +someone who leaves drops out.
    • +
  • Authorization. The asker is stamped by the host (a worker asks only in its own session's scope; secret questions +only from server askers). Each question stores who may answer it; a guest's send access does not imply approval +rights. Secret answers never appear in open, documents or logs.
    • +
  • Link drops and crashes. A rejected ask is not a denial. Askers retry with the same key until their own signal +aborts; a pending ask counts as an active call for keep-alive.
    • +
  • Where it lives. On the host where the asking work runs: session questions on the session's worker, login +questions on the server. UIs reach both through the usual relay.
    • +
+

The store in Chord v2 terms:

+
host.serve(QuestionStore, ({ scope }, self) => {
+  const pending = self.open(async () => new PendingQuestions(), (p) => p.rejectAll("store closed"));
+  const open = self.derive("open", [pending.value.readable], ([all]) => all);   // data only, no resolvers
+  return { open, ask: (args, ctx) => pending.value.ask(args, ctx), answer: (args, ctx) => pending.value.answer(args, ctx) };
+});
+
+host.serve(Questions, ({ scope }, principal, self) => {
+  const store = self.get(QuestionStore, { scope });
+  if (isHost(principal)) return { variant: "asker", ask: (args, ctx) => store.ask({ ...args, asker: principal }, ctx) };
+  if (!mayView(principal, scope)) throw forbidden();
+  return {
+    variant: "viewer",
+    open: self.derive("mine", [store.open], ([all]) => openFor(all, principal)),
+    answer: ({ id, value }, ctx) => store.answer({ id, value, by: principal }, ctx),
+  };
+});
+

The pending map with its resolvers is in-process state with promises, so it is a native of the store (opened and +closed with it). Its close rejects every waiter; askers retry with their key against the new store after a reload.

+
Review: questions.md predates the v2 rules

apps/p2/docs/questions.md writes the store as self.resource(...) with a replace hand-over, which the design +rejected ("resources replaced in place"), and opens Chord call scopes in Durable hooks with host.root, which Q6 +replaced with chord(fn, context) (chapter XIV). Design §20 says the Questions store "must live in an object (it +predates natives only in objects)". The sketch above is this guide's translation, with the derive per viewer; filtering +per viewer in the store, once per variant, may be cheaper for large audiences. The ask signature passes typed +objects as arguments, which is fine (data), but the audience readable form cannot cross a link: across links, +audience must be data (principals) or computed by the store.

+
+

Choosing

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
CaseUse
a multi-step login in one ownera flow: the owner's job keeps the resolver, state shows the steps
a tool needs the owner's approvalQuestions, first, audience the owner
the agent asks everyone presentQuestions, all, audience from presence
deploy needs 2 of 3 approvalsQuestions, quorum(2)
a notice (auth URL, device code, progress)not a question: the asking work's own progress state
+
+ +
XIII

Extension Points

The facet, registries and the HTTP component: how plugins add behaviour to each other without callbacks, and how the owner stays in charge.

+

The notes plugin wants a download route, GET /s/:session/notes.txt. A logs plugin wants to time every request. +Several plugins add model providers, tools for the agent, commands for the TUI. Someone must combine these +contributions, decide conflicts, and replace one plugin's part as a whole when it reloads. Between plugins exactly two +things pass: object types and registries.

+

The facet, completed

+

The facet has appeared since chapter II as the place that serves types. It is also itself a pure object with the +host's lifetime:

+
// notes/server.ts
+export default defineFacet((host, self) => {
+  const prefs = self.state(NotesPrefs);                                  // plugin-wide documents
+  self.contribute(Http, { routes: { "GET /s/:session/notes.txt": exportNotes } });
+  host.serve(NoteCountStore, …, { lifetime: "host" });
+  host.serve(NoteCounts, …);
+});
+
    +
  • It may hold documents, references, routines and contributions; no natives (those need an object with its own +lifetime).
    • +
  • Its identity is the plugin's name, so its persisted documents survive restarts and reloads.
    • +
  • Served objects go live after the facet and may read and write its documents (chapter V).
    • +
  • host.serve and self.contribute are valid only while the setup runs. The facet never self.gets a type it +serves.
    • +
+

Registries

+

A registry is an extension point owned by someone (a Chord component, the app, a plugin). Each contributor +contributes at most one value per registry; the table is an immutable, ordered list of (contributor, value); +the registry's compose turns it into the value consumers use.

+
// owned by the HTTP component
+export const Http = defineRegistry<HttpContribution, Router>("chord.http", {
+  version: 2,
+  accepts: { 1: (old: HttpContributionV1): HttpContribution => upgrade(old) },
+  compose: (table, { report }) => buildRouter(table, report),   // pure; always delivers a value
+});
+
+self.contribute(Http, { routes: { … }, wraps: [ … ] });          // a contributor
+const router = host.registry(Http).value;                        // a consumer: live
+

compose runs at runtime, whenever the table changes: after a reload's commit, when a contributor goes live or +ends, when a contribution changes. It is not a staging check: contributions can be dynamic (known only once a native +is open), so no check before a reload could be complete. compose always delivers a value with its fallback built in +(e.g. the first by plugin name wins) and reports conflicts through report, which become owner notices and telemetry +events. Readers get a plain value and never handle conflicts. Only if compose throws is the registry in an error +state, and readers get an error when they use it. Work that started earlier keeps the value it read.

+

Rules:

+
    +
  • Who contributes (R21): only facets and lifetime: "host" objects; only their self has contribute (a type +error elsewhere). Every object may read registries.
    • +
  • A contribution is in the table exactly while its contributor is live: a facet's while its plugin is loaded; an +object's from live until it ends, fails or restarts.
    • +
  • Order is sorted by plugin name: deterministic on every host and across reloads, and deliberately meaningless. +Meaningful order comes from the owner's composition (HTTP phases, below). Helpers: keyed(table, entriesOf), +ordered(table, itemsOf, orderOf?).
    • +
  • One authoritative declaration (Q8): only the defining plugin's facet declares the registry to the host +(compose, version, accepts). Other plugins' tokens only identify it by name and types. A reload of the defining +plugin replaces the declaration atomically and the table recomposes. If the defining plugin is not loaded, the +registry has no value (readers get an error like a failed reference) and contributions wait in the table.
    • +
  • Versions (Q8): a contribution carries the registry version its plugin was built against; older ones are adapted +by the defining facet's pure accepts. One without a matching version or adapter is left out and reported naming +plugin, registry and versions; its plugin keeps working. Contributions are in-process values (often functions) and +are not schema-checked.
    • +
  • Use sites guard contributions. A contribution is untrusted code (third-party plugins are a goal). Every use site +catches throws and rejections when calling an entry, reports naming the contributor, and continues with the others.
    • +
  • The owner decides conflicts, order and selection. Registries add behaviour; they never replace an owner's checks.
    • +
+

Natives through registries

+

Registries are the only way a native reaches another plugin. An entry lives exactly as long as its contributor; +consumers look up at each use (or follow the composed value) and keep nothing from an entry beyond a use. The models +plugin's host-lifetime object contributes its runtime; the harness looks it up at every request. After a models reload +the next request gets the new runtime, while a running stream finishes on the old one.

+

A handler that stores const r = entry() in a module variable and uses it for days has a bug: after the contributor +restarts it holds a closed native.

+

Contributions that change

+

Some contributions are known only at runtime: MCP tools appear when a server connects. A contribution is either fixed +for its contributor's life (dynamic parts behind handles read at use) or a readable, which Chord subscribes to; +on every change Chord replaces that entry and runs compose again. For in-process values, self.writable(initial) is +a host-local readable of any value (no JSON, no wire, compared by identity) with set:

+
const ext = self.writable(mcpExtension([]));                   // a Durable Extension, no tools yet
+self.contribute(HarnessExtensions, ext);
+self.routines.on(mcp.tools, (tools) => ext.set(mcpExtension(tools)));
+

There is no remove call. An entry leaves when its contributor ends, or after a reload whose new code no longer +contributes. A live contributor with nothing to offer sets its readable to the registry's empty value (an extension +without tools, empty routes); the table keeps exactly one entry per live contributor.

+

Optional dependencies

+

Registries also replace optional references (chapter VI). Radius forwards connections to ports the ports plugin +manages, but ports is optional. So radius owns radius.forwards and ports contributes its forwards; without ports the +table is empty and radius works.

+

The HTTP component

+

HTTP is an optional Chord component (@earendil-works/chord/next/http) built on a registry. A route gets a plain +request summary (method, path params, query, principal, provenance such as local or relayed, selected headers, parsed +and capped JSON bodies, uploads in chunks) and a ctx whose principal is the request's, so ctx.get(Notes, …) gives +exactly the view that person may see. The route returns a description; the component does the HTTP work once (ranges +with 206 and 416, HEAD, download headers with a safe filename, nosniff, CSP and its nonce, caching, streaming, abort +on disconnect):

+
type HttpResponse =
+  | { kind: "json"; value: JsonValue } | { kind: "page"; title: string; body: string }
+  | { kind: "bytes"; name: string; mime: string; size: number; read(offset: number, length: number): Promise<Uint8Array> }
+  | { kind: "redirect"; to: string } | { kind: "error"; status: number };
+
+// notes/server.ts
+self.contribute(Http, {
+  routes: {
+    "GET /s/:session/notes.txt": async (req, ctx) => {
+      const notes = await ctx.get(Notes, { session: req.params.session });   // the requester's view
+      const text = new TextEncoder().encode(render(notes.notes.value));
+      return { kind: "bytes", name: "notes.txt", mime: "text/plain", size: text.byteLength,
+               read: async (offset, length) => text.subarray(offset, offset + length) };
+    },
+  },
+});
+

Routes compose with keyed: two contributors with one pattern is a reported conflict, and the first by plugin name +wins. Wraps are middleware in phases the owner defines (observe, headers, transform, outermost first), +within a phase in contributor order. No numeric priorities: plugins have no single registration order, so Fastify's +lifecycle phases and encapsulation are the model, not Express.

+
// logs/server.ts
+self.contribute(Http, {
+  wraps: [{
+    name: "access-log", phase: "observe", scope: "all",          // default scope "own": only own routes
+    wrap: async (req, next, ctx) => {
+      const t = Date.now();
+      const res = await next(req);
+      log(req, res, Date.now() - t);
+      return res;
+    },
+  }],
+});
+

route narrows a wrap to one route; a route that does not exist is a reported conflict. Wraps may add a security +boundary on top of an owner's checks, never instead of them. A request uses the router as it was when it started. +Request resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary.

+

Channels and big bytes

+

Channels are WebSocket byte channels (send, onData, close) for terminals, separate from object links and +with their own acknowledged windows, so a large state message never holds up keystrokes.

+

Big transfers are chunked by the owner, not by Chord: bytes are a schema type (bytes()), carried natively +(never base64), and a 50 MB upload is a protocol of plain calls:

+
beginUpload({ name, size }, ctx)               → { upload }
+append({ upload, offset, bytes }, ctx)         // about 1 MiB per chunk; the awaited call is the backpressure
+finishUpload({ upload }, ctx)                  → { id }
+read({ id, offset, length }, ctx)              → bytes
+

After a link drop the append in flight fails like any call; the writer asks the owner for the stored offset and +resumes. Unfinished uploads expire through the owner's routine. Chord has no stream type: it would need its own rules +for drops, cancellation, relays and restarts, which this offset protocol already survives.

+
Built so far

Registries are layer L9. The HTTP component, channels and the chunked-transfer helper are deferred until after the +kernel and a first real proof (the harness, one tool with an approval pause, a code mode reload); until then the +playground serves its pages with plain app code. Their design stays the target.

+
+
+ +
XIV

The Agent

pi-durable inside Chord: the harness as a core object, models, environments and tools through registries, and how a tool reaches Chord objects and people.

+

Each Studio worker runs a pi-durable harness for its session. The harness must run for the worker's whole life, +whether anyone watches or not. Its models come from the models plugin, its execution environments from the envs +plugin, its tools from every plugin, and all of those reload independently. This chapter assumes you know pi-durable +and shows only how it plugs into Chord.

+

Shared foundations

+

Durable and Chord v2 share modules: Durable imports Chord's context, delta, json, output and sqlite, and v2 +imports the same, so Context, Draft and JsonValue are one type each. A v2 readable is today's +replicatedState (chord/src/state.ts), so Durable's viewState() and document watches can be exposed as Chord state +members without conversion. Only plain data crosses between Durable and Chord objects.

+

The harness is a core object

+
// studio/src/core/harness.ts (base code)
+import { Harness as DurableHarness } from "@earendil-works/pi-durable";
+
+export const ChordHostKey = createContextKey<Host<P>>("studio.chord-host");
+export const HarnessModels = defineRegistry<Models, Models>("studio.harness-models", { version: 1, compose: firstModels });
+export const HarnessEnvironments = defineRegistry<EnvFactory, EnvFactory>("studio.harness-envs", { version: 1, compose: firstEnv });
+export const HarnessExtensions = defineRegistry<Extension, RegistrySnapshotSource>("studio.harness-extensions", {
+  version: 1,
+  compose: (table, { report }) => extensionSnapshot(table, report),      // one Durable Extension per plugin
+});
+const HarnessRuntime = defineRegistry<() => DurableHarness, () => DurableHarness>("studio.harness-runtime", {
+  version: 1, compose: onlyEntry,                                        // core-private: one contributor
+});
+
+host.serve(Harness, (_input, self) => {
+  const info = self.get(SessionInfo, {});
+  const harness = self.open(async ({ signal }) => {
+    const { id, dataDir } = info.init.value;
+    const base = withContextValue(ChordHostKey, host, withAbortSignal(signal, BACKGROUND_CONTEXT));
+    return DurableHarness.open(await openNodeSqliteStorage(join(dataDir, id, "durable.sqlite")), {
+      models: () => host.registry(HarnessModels).value,                  // read at each use
+      env: (target) => host.registry(HarnessEnvironments).value(target),
+      settings: () => currentSettings(),
+      registry: durableRegistry(host.registry(HarnessExtensions)),      // a RegistryReader over the composed value
+    }, base);
+  }, (h) => h.close(BACKGROUND_CONTEXT));
+  self.contribute(HarnessRuntime, () => harness.value);                 // for core's conversation objects
+  return { … };
+}, { lifetime: "host" });
+
    +
  • Lifetime "host": it runs for the worker's whole life, also while nobody holds it, because Durable tasks +continue.
    • +
  • Base code: it lives in the core facet, so a plugin reload never restarts it; only a base-code change (a host +restart) does.
    • +
  • Registries, looked up at each use: Durable's models and settings accept functions (pi 7ec1f648e), so each +generation gets the current value. After a models reload the next request uses the new runtime; a running stream +finishes on the old one.
    • +
  • Durable's storage is its own. Chord documents never commit together with Durable transcripts.
    • +
+

The design writes the call as Durable.open(storage, { … }); pi-durable's actual entry point is +Harness.open(storage, options, context), used here as DurableHarness to keep it apart from the core object +Harness.

+

Start order

+

At host start Chord loads every facet and waits until every facet and every lifetime: "host" object other than the +harness is live or failed: the start barrier. Then the registries hold the contributions of every plugin that +loaded, and the harness opens and resumes scheduling (R17). An object that must contribute at start, like the model +runtime, is therefore lifetime: "host". Nothing waits for specific contributions; what is missing shows where it is +used:

+ + + + + + + + + + + + + + + + + + + + + + + +
MissingWhat happens
a toolits call gets an error result the model sees
a modelthe generation reports "model unavailable" and waits, not fails, until a model appears
a task definitionthe task stays blocked (missing_task) until the plugin is installed or the task is aborted
a plugin that failedreported with its error; a facet in backoff contributes later and the registries recompose
+

The same holds when a session's database is copied to an instance with other plugins: missing tools, sections and +hooks are skipped, documents of missing plugins stay untouched, stored selections are never rewritten. The UI must +show blocked tasks with their reason and offer to abort them; a turn waiting on one hangs until then.

+
Review: the harness exception to the barrier (C17, R17)

R17 makes the harness start after the barrier, a special case in the barrier's rule. Review round 3 proposed cut C17: +no harness-specific exception; the harness is an ordinary object the app's start code holds after load resolves. +Not decided. chord-next-walkthrough.md concept 28 still says the harness's activation is the start barrier, which +R17 replaced.

+
+

Tools that call Chord

+

Durable tools, hooks and sections receive Durable's ToolExecutionApi (or HookApi) and a Context, never a Chord +ctx (Q6). Core put the host into the harness's base context under ChordHostKey, and every phase context inherits +it. A core helper opens a Chord call scope from any Durable context:

+
// studio/src/core/chord.ts
+export function chord<T>(context: Context, fn: (ctx: Ctx) => Promise<T>): Promise<T> {
+  const host = context.value(ChordHostKey);
+  if (!host) throw new Error("not inside the Studio harness");
+  return host.root(fn, { signal: context.abortSignal, telemetry: context.telemetry });
+}
+

The scope runs as the host principal; attribution is data; references taken with ctx.get are released when fn +settles. There is no wrapping of tools and no app-specific defineTool. The notes plugin gives the agent a tool:

+
// notes/worker.ts
+const addNote = defineTool({
+  name: "notes_add",                                         // tools are named <plugin>_<name>
+  description: "Add a note to this session's notes panel",
+  parameters: Type.Object({ text: Type.String({ maxLength: 2000 }) }),
+  execute: (args, api, context) => chord(async (ctx) => {
+    const info = await ctx.get(SessionInfo, {});
+    const store = await ctx.get(NoteStore, { session: info.init.value.id });
+    const id = await store.add({ text: args.text, by: `agent:${api.conversationId}` }, ctx);
+    return { output: [{ type: "text", text: `added note ${id}` }] };
+  }, context),
+});
+
+export default defineFacet((host, self) => {
+  self.contribute(HarnessExtensions, defineExtension({ name: "@studio/notes", tools: [addNote] }));
+  …
+});
+

A reload of the notes plugin replaces its extension in the table; Durable pins a registry snapshot per task phase, so +a tool call already running finishes on the old code and the next phase uses the new one.

+

Tools that need a person

+

Clients are not addressable, so a tool cannot call the user's tab. A tool that needs the user goes through an owner's +state: the Questions service (chapter XII). The model-facing ask_user tool and an approval hook both call +Questions.ask; tabs render the open questions and answer them.

+
hook(ToolTask, {
+  beforeTool: async (call, api, context) => {
+    if (!needsApproval(call)) return undefined;
+    const decided = await api.memo<boolean>("approval", context);       // a rerun does not ask again
+    if (decided !== undefined) return decided ? undefined : { block: "Denied" };
+    const allowed = await chord(async (ctx) => {
+      const q = await ctx.get(Questions, { scope: { session } });
+      for (;;) {
+        try {
+          const r = await q.ask(
+            { kind: "confirm", message: `Run ${call.name}?`, allow: "Allow", deny: "Deny" },
+            { audience: approvers, policy: { mode: "first" }, key: api.taskId },
+            ctx,
+          );
+          return r.answer;
+        } catch (error) {
+          if (ctx.signal.aborted) throw error;                          // a link blip is not a denial: retry
+          await backoff(ctx);
+        }
+      }
+    }, context);
+    await api.memo("approval", allowed, context);
+    return allowed ? undefined : { block: "Denied by the user" };
+  },
+});
+

Durable runs beforeTool in the tool task's call phase; after the execute checkpoint a crash retries execution +without hooks, so the stored decision covers that retry. The approval must cover the final arguments: put the +approval hook after hooks that rewrite arguments.

+

Conversations for the UI

+

Chat panels need a conversation's live view, submit, steer, stop. The design leaves p3's conversation API to the p3 +design; one construction that follows the rules: a core ConversationStore { conversation } (local) opens the Durable +watch as its own native, getting the Durable harness from the core-private HarnessRuntime registry contribution (a +native may reach another object only through a registry entry), and exposes a bounded window as a derive. Per-principal +Conversation { session, conversation } views in front of it check who may send (chapter XIX).

+

Code mode

+

Code mode is a plugin. It owns its pi-codemode sandbox as a native; its tool entry looks up the current sandbox +when execute() starts, so a call pinned by a phase before a reload (waiting in an approval hook, say) starts on the +new sandbox. On reload and stop the old sandbox is sealed and drained in close: running scripts finish or hit +their deadline, for at most 30 s (Q7). After a worker stop Durable replays as it does today.

+

What lives where

+ + + + + + + + + + + + + + + + + + + +
DataWhere
transcript-bound: entries, documents that fork and rewind with a conversationDurable
app state: session list, notes, drafts, grants, settingsChord documents
data core enforces on conversations (read-only one-shots, cwd followers)a core per-conversation policy document, written when the conversation is created (createConversation({ init }))
+

Telemetry: pass the host's adapter as HarnessOptions.telemetry; Durable stores trace carriers on submissions and +tasks (chapter XVII).

+
+ +
XV

Reload and Rollout

What happens when you press Reload: per host, per object, and across the server, the workers and every tab.

+

Back to the preface. Mario changed the notes plugin's sort order and pressed Reload. Everything the earlier chapters +asked of plugin authors (synchronous factories, natives only in objects, routines instead of timers, data in +documents) exists so that this moment is safe. This chapter shows what Chord does with it.

+

On one host

+
1. load      new bundles; an import or syntax error rejects
+2. staging   facet setups and every live object's new factory run with a recording self
+             (declarations only, nothing activated); any throw rejects
+3. commit    the host switches to the new code atomically
+4. after     per object: pure objects swap, native objects restart
+

Rollback is all-or-nothing per host. Any load or staging error leaves the whole host on its previous code, so a +host is always completely at one revision and its status is one value. Runtime states (a registry conflict, a failed +open, a migration) are not reasons to roll back; after the commit, errors stay with one object. There is no factory +trial, no migrate dry run, and no consumer contract check in staging: an incompatible consumer fails at bind with +contract_mismatch and a notice naming plugin and type.

+
Reloading @studio/notes on worker S1, object by object

Load and stage

The new bundles load. The facet setup and the factories of Notes (three views), NoteStore { S1 } and NoteInbox +run with a recording self: they declare, nothing activates. Nothing throws, so the host commits to the new code.

+

NoteStore: pure, swapped in place

NoteStore has no self.open. Its new methods and derives take over; NoteList, the drafts, its references and the +running import job stay. Routines restart, serialized: the new cleanup waits for the old handler in flight. The gate +opens.

+

Running calls finish on old code

An add that was running when the swap happened finishes on the old code with its old handles; its transaction still +commits. Only new calls get the new handles. Exception (R15): if the reload raised a document's version, old code that +touches the migrated document throws, the document stays intact, and that call fails once.

+

The import job keeps its old run

The job's run keeps the code it started with. When it settles, the new finish records the outcome.

+

The views: pure, swapped

The three Notes views swap in place. Owner, Ann and Bob keep their references; the reader derive is recomputed with +the new sort and the difference is sent. Nobody sees a reconnect.

+

NoteInbox: native, restarts

NoteInbox has a watcher. Its running work ends aborted, the watcher closes with reason reload, the new factory +runs, the watcher reopens. Its documents would stay. Whether a file dropped during the restart is seen is the +watcher's business: Studio's watchDir scans the directory once when it opens.

+
+

Why natives restart instead of being handed over: a native has no line, no transactions and no schema. If old and +new code shared it, they would interleave on it unchecked. In practice restarts are cheap: the Durable harness is base +code, terminal PTYs live in a separate terminal host process, and model runtimes, sandboxes and a search index rebuild +in milliseconds. Only reloading the plugin that owns SSH connections (running remote commands die) or the relay socket +(relayed connections drop) hurts.

+

Removing a type: its objects end (routines stop, natives close, references end with an error); in-memory documents +go with them; persisted ones stay until retired. Consumers elsewhere see a restarting object like any object that +briefly goes away: loading, parked calls, then live.

+

UI facets reload too. A reload unmounts and remounts components. That is why "leaving" (abort a login, close an +overlay) is never derived from an unmount (chapter VIII).

+

Across hosts

+

A revision is (server incarnation, sequence) and names the complete desired code set: every facet with its +digest, not a batch of changes. Every install reconciles everything missing, and a host reports a revision only when it +runs that complete set. A revision also carries the code-set digest (a hash over the facet bundles' content hashes), +so a server restart with unchanged code triggers no reinstall.

+

The order is server, workers, clients. Each host reports installed, installed_failed (with the objects that +failed) or rejected (with the reason; it stays on its previous code). A server that rejects stops the rollout. A +host that starts or reconnects installs the current revision before anything is routed to it. Installs are serialized +per host and bounded (60 s).

+

Calls keep flowing during a rollout. When a call reaches a provider on another revision, the mismatch response +carries the provider's revision and status:

+ + + + + + + + + + + + + + + + + + + +
ProviderThe call
ahead, or behind and installingupdating: values readable, calls park (with the park deadline)
rejectedfails at once; retried on the next revision
same revisioncontract_mismatch: a real mismatch, final
+

A host checks its own install status first: if its own install was rejected, a call fails at once with the reason +instead of waiting on a provider. Calls parked on a binding that is released end with an error and are not resent. +There are no cross-host transactions and no votes.

+
tabrevision 8serverrevision 8worker S1installing 8install revision 8call Notes.add (new contract)relayupdating (rev 7, installing)updatingpark (deadline 60 s)installed 8provider at 8call Notes.add (resent from park)relayresult
Figure 9. A tab calls into a worker that is still installing. The call parks and runs once the worker reports the revision.
+

Start barrier

+

At host start and first load there is one more rule (chapter XIV): load every facet (the core facet with its start +data among them), wait until every facet and lifetime: "host" object other than the harness is live or failed, then +resume dependent work. Plugin reloads need no barrier.

+

Loading and the dev loop

+
    +
  • Roots: plugins come from built-in, global (<data>/plugins), package (npm or git, installed with +--ignore-scripts) and project roots, found by directory listing; no source parsing. The server selects which load. +Dependencies between plugins resolve by contract name at runtime. Names are prefixed: objects <plugin>.<name>, +tools <plugin>_<name>.
    • +
  • Loader D: one base bundle per surface (host packages and the app runtime) plus one script per facet, with host +packages passed in (no module map). Old generations are reclaimed by the garbage collector; a stable filename per +(facet, digest), and V8's compilation cache off on hosts that reload in place. The loader is an adapter: static +imports (tests, Durable Objects) work the same, with the app passing the plugin name to load.
    • +
  • Dev loop: a source watcher marks plugins dirty and never applies anything. A reload control (the playground's web +shell, a CLI, an agent tool such as p2's reload_plugins) reloads the changed plugins, or restarts all hosts for +base code. Reloads are always explicit.
    • +
+
Review: project plugins (#14, deferred)

Per-directory plugins with trust prompts and snapshots (apps/p2/docs/plugin-loading.md, still the reference for +roots, trust and scopes) need their own pass over this design. Until then, treat the project root as designed but not +settled.

+
+
Built so far

Reload and rollout are layer L10, the last before telemetry. The frozen kernel has a reload path with host values and +a development freeze, both removed from the design.

+
+
+ +
XVI

Changing Contracts

How a contract gets a new version while old tabs, old workers and old plugins are still talking to it.

+

Chapter XV ended with a tab that calls into a worker on another revision. That case is short-lived: the rollout +finishes and both sides run the same code. Other cases last longer. A phone kept a tab open overnight with yesterday's +web bundle. A plugin from another author was built against Notes version 1 and is not updated yet. In both, a +consumer holds a contract that the provider no longer has. This chapter is about what the provider does then. It is +also the one part of Chord v2 that is built: everything in the sections on evolve, checkAccepts and fingerprints +runs today in packages/chord/src/next/contracts/.

+

What has a version

+

Everything that is stored or crosses a boundary has a version. Each surface handles an old version its own way:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SurfaceVersionWhen an old one shows upChapter
object type (contract)version plus fingerprintevolve and checkAccepts on the provider; otherwise contract_mismatchthis one
documentversion plus schema fingerprintpure migrate on access; a schema change without a bump fails the object brokenV
jobversionaccepts on outcomes and running; blocked without an adapterXI
registryversionaccepts adapts contributions; one without a match is left out and reportedXIII
nativenonethe object restarts on reloadIX
Durable tasks and documentsDurable's versionDurable's migrateXIV
link protocolprotocol versionrefused; hosts update together, a web page reloadsXVII
UI components, Chord and app APIsnone yetexposed as they are; an SDK will version them later
+

The rule behind every row is the same as for documents: every change bumps the version, additive ones included. +Chord never guesses compatibility from the shape of two schemas.

+

A history, not a file of versions

+

A contract history starts with a base: a full remote() or local() declaration with an explicit version. The +Notes contract from chapters II and III is a base at version 1. Studio now wants pinned notes: notes get a pinned +flag and editors get a pin method. Instead of editing the declaration, the plugin continues the history:

+
// notes/contract.ts
+import { bool, evolve, nul } from "@earendil-works/chord/next";
+
+export const NoteV2 = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int(), pinned: bool() });
+export const PublicNoteV2 = obj({ id: str(), text: str(), at: int(), pinned: bool() });
+
+export const NotesV2 = evolve(Notes, (e) =>
+  e.state("editor", "notes", obj({ items: arr(NoteV2) }))
+    .state("reader", "notes", obj({ items: arr(PublicNoteV2) }))
+    .addMethod("editor", "pin", obj({ id: str(), pinned: bool() }), nul()),
+);
+

evolve(prev, (e) => …) appends one version to prev's history: NotesV2.version is 2, and NotesV2 still knows +version 1 exactly as it was declared. Every version is materialized and fingerprinted at declaration. That is what +makes old code safe: an old plugin bundle that contains its own copy of Notes version 1 computes the same +fingerprint as the history inside NotesV2. There is no build step, no generated file and no fingerprint copied by +hand.

+

The edits:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EditChangesKind
e.state(variant, member, schema)one state member of one variantremote
e.addState(variant, member, schema) / e.removeState(variant, member)adds or removes a state memberremote
e.state(member, schema), e.addState(member, schema), e.removeState(member)the same, flatlocal
e.args(name, schema) / e.result(name, schema)one side of a methodboth
e.method(name, args, result)both sides of a methodboth
e.addMethod(variant, name, args, result)adds a method (or an existing one to another variant, same signature)remote
e.addMethod(name, args, result)adds a methodlocal
e.removeMethod(name)removes a method from every variantboth
+

Edits chain. The method(...) wrapper appears only in base declarations; edits take the schemas directly, so a +method without a result is added with nul(). The rules of chapter III still hold across versions: a method has one +signature in every variant, and a member's kind never changes, not even after it was removed and added again.

+

What evolve cannot do: change the input or the set of variants. Both change what an object is (its identity, +who gets which members), and no adapter can turn one into the other. That needs a new history base, below.

+

The provider says what it accepts

+

NotesV2 says what changed. It does not say how version 1 consumers are served; the provider decides that, with +mappings that live in its own code, never in the contract. The worker facet declares them with checkAccepts:

+
// notes/worker.ts
+import { checkAccepts } from "@earendil-works/chord/next";
+
+const toV1 = ({ pinned, ...note }: Static<typeof NoteV2>) => note;
+const toPublicV1 = ({ pinned, ...note }: Static<typeof PublicNoteV2>) => note;
+
+const notesAccepts = checkAccepts(NotesV2, {
+  1: {
+    state: {
+      editor: { notes: (current) => ({ items: current.items.map(toV1) }) },
+      reader: { notes: (current) => ({ items: current.items.map(toPublicV1) }) },
+    },
+    methods: {},                                   // add and remove did not change; pin is new
+  },
+});
+// notesAccepts.versions: [{ version: 1, fingerprint: Notes.fingerprint }]
+
+host.serve(NotesV2, notesFactory, { accepts: notesAccepts });    // working name for the objects-layer option
+

checkAccepts is pure: it checks the mappings without calling them and returns only the version metadata of the +old versions it lists. A provider that lists nothing accepts only its current version. The objects layer later +compares a consumer's version and fingerprint against this list.

+

The mappings point in the direction the data flows:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Member in the old versionMappingDirection
state, changed(currentMember) => oldMemberprovider's current value to what the old consumer expects
state, removed(currentVariantState) => oldMember, or "refuse"receives the whole current state of that variant
method arguments, changedargs: (old) => currentold caller's arguments to current ones
method result, changedresult: (current) => oldcurrent result back to the old shape
method, changed or removed{ call: async (oldArgs, ctx) => oldResult }, or "refuse"replaces the method for old callers
member added sincenothingan old consumer never asks for it
+

A changed state member cannot be refused: the old consumer still has that member and must get a value. A removed one +can be computed from what the variant has now, or refused. e.method(name, args, result) replaces both sides and +therefore needs both mappings, even if one side stayed the same (then an identity mapping).

+

Required, at type level and again at runtime. For every member that exists in the old version and changed or was +removed since, the mapping is required by the type of checkAccepts. Because plugins are bundled without type +checks, the same check runs at declaration: a missing mapping, a key for a member that needs none, an unknown version +or a wrong shape throws, naming the member's path (for example accepts[1].state.reader.notes). An absent state or +methods section counts as empty and may be left out only when nothing there needs a mapping.

+

Conservative with no-op edits. An edit that sets a schema to what it already was, or changes a member and then +changes it back, still requires an identity mapping. TypeScript erases constraints such as { max: 2000 } from the +static type, so two schemas that look the same to the compiler may differ; Chord does not try to tell.

+

Old versions map straight to the current one

+

Studio later replaces remove with archive, which keeps the note in a trash list:

+
export const NotesV3 = evolve(NotesV2, (e) =>
+  e.removeMethod("remove").addMethod("editor", "archive", obj({ id: str() }), nul()),
+);
+
+const notesAccepts = checkAccepts(NotesV3, {
+  1: {
+    state: { editor: { notes: (c) => ({ items: c.items.map(toV1) }) }, reader: { notes: (c) => ({ items: c.items.map(toPublicV1) }) } },
+    methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } },
+  },
+  2: { methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } } },
+});
+

Each entry maps one old version directly to the current one, not through the versions in between. Version 1 +needs the state mappings (its notes have no pinned) and the remove replacement; version 2 needs only the +remove replacement, and leaves state out because its state is unchanged. A member that an old version never had +needs nothing for that version, even if it was added, changed and removed later. Dropping support for version 1 is +deleting its entry: version 1 consumers then get contract_mismatch.

+

Binding an old consumer

+

A hold carries only the consumer's contract version and fingerprint, never a description of the contract (R6). +The provider decides with its own schemas:

+
THE HOLD ARRIVESOUTCOMEholdversion 1, fingerprint F1equal to current?version and fingerprintaccepted old version?F1 from the provider's own historybindcurrent membersbind adaptedthrough the mappingscontract_mismatchor updating, during a rolloutyesnoyesnoA mismatch fails only that consumer's binding. The provider object and its other bindings are not affected.
Figure 10. What a provider does with a hold. The decision uses only the provider's own history; a peer's schema is never an input.
+

Once a version 1 tab is bound, it is served as if version 1 were current:

+
A version 1 tab on a version 3 worker

Bind

The tab holds Notes { S1 } with version 1 and fingerprint F1. The worker's history computes F1 for version 1, and +notesAccepts lists version 1, so the hold binds adapted. The tab gets the editor variant as version 1 defined it: +notes, add, remove. It never sees pin or archive.

+

State through the mapping

Chord serves the adapted state like a derive of the provider's state: it applies the mapping to the current value +and keeps the result per binding. The snapshot the tab receives has no pinned fields.

+

A commit

The owner pins a note from a current tab. The worker sends the current tab the draft's ops. For the version 1 tab, +Chord applies the mapping to the new value and sends the difference to its last adapted value (diffRevisions). The +mapping produced nothing new for a pin, so the version 1 tab receives no change at all. A large list is never resent +whole on every commit.

+

A call

The tab calls remove({ id }). Its arguments are checked against the provider's own version 1 schema, then the +call replacement runs archiveNote on version 3 code. Its null result is what version 1 expects. Had the method +only changed, args would have mapped the arguments forward and result the result back.

+

The worker moves on

A later revision drops the version 1 entry. When the tab re-holds (or the worker restarts and the tab binds again), +the hold gets contract_mismatch, and Studio's panel shows "reload to update" instead of failing silently.

+
+

A new history base

+

Some changes cannot be adapted. Studio decides that notes belong to a project, not a session: the input changes from +{ session } to { project }. That is a new base, written in full, with an explicit version that continues the +published numbering:

+
export const NotesV4 = remote("notes", {
+  version: 4,                                      // explicit: the published history used 1 to 3
+  input: obj({ project: str() }),
+  placement: () => Server,
+  variants: {
+    editor: { state: { notes: obj({ items: arr(NoteV2) }) }, methods: { add: method(…), pin: method(…), archive: method(…) } },
+    reader: { state: { notes: obj({ items: arr(PublicNoteV2) }) } },
+  },
+});
+// checkAccepts(NotesV4).versions is empty
+

A new base has no history before its version and cannot accept older versions, not even for an additive change. +Every consumer built against versions 1 to 3 gets contract_mismatch and must update. Later evolve calls continue +from version 4. The same applies to a change of the variant set. Do not spread an older declaration into a new one or +copy its fingerprint; the contract objects are branded and evolve and checkAccepts reject copies.

+

Local contracts

+

local contracts (chapter IV) evolve the same way, with a flat members map and no variants:

+
export const SessionInfoV2 = evolve(SessionInfo, (e) => e.state("init", InitV2));
+checkAccepts(SessionInfoV2, { 1: { state: { init: (current) => toInitV1(current) }, methods: {} } });
+

Same-host consumers of a local type are usually reloaded together with their provider, but not always: another +plugin on the same host may be built against an older version. The rules are identical; only the variant argument is +missing.

+

Implementing a contract

+

Implementation<typeof NotesV2> is the type of everything a provider must supply for the current version: one state +getter per member and variant, and one asynchronous method per method, each receiving (args, ctx):

+
const impl: Implementation<typeof NotesV2> = {
+  state: { editor: { notes: () => list.value }, reader: { notes: () => publicList.value } },
+  methods: { add: async ({ text }, ctx) => addNote(text, ctx), remove: async ({ id }) => removeNote(id), pin: async (a) => pin(a) },
+};
+

Keep declaration types inferred (no annotation on NotesV2): the inferred type carries the complete history, and +checkAccepts needs it to know what changed.

+
Review: L1's implementation shape and the factory's return value

Implementation<C> in src/next/contracts/types.ts is { state: { variant: { member: getter } }, methods }. The +design's factories (design.md §6, used throughout this guide) return { variant, member: value, method: fn }. +The objects layer (L4) has to connect the two, and also define how serve receives the checked accepts (this guide +uses the working name { accepts } as a third argument) and what "refuse" does at runtime. The natural reading, +used in the stepper above, is that a refused method call and a hold that needs a refused state member end with +contract_mismatch; no source says so.

+
+

The fingerprint

+

The fingerprint is a 64-bit hash (hash64, portable across runtimes) of a normal form of the contract's +schemas. The normal form decides what counts as a change:

+ + + + + + + + + + + + + + + + + + + +
IgnoredSignificant
order of object keys, required and enum (sorted)order of union and intersection branches (anyOf, oneOf, allOf) and of tuples
annotations on schema nodes: title, description, examples, $comment, deprecateddefaults, formats, patterns, bounds, $ref names
placementinput, variants, member kinds and every member schema
+

Union order is significant because it is meaning: TypeBox picks defaults and decodes by the first matching branch. +Annotations are dropped only on schema nodes; an object property that happens to be named title is data and counts.

+

Only data schemas are allowed. One walker copies, validates and normalizes each schema against a whitelist of +TypeBox keywords: primitives, arrays and tuples, objects and records, unions and intersections, $defs and $ref +(use $ref for recursive types), constraints and annotations. Conditional and dependent schemas, not, +definitions, contentSchema, unevaluated*, contains, propertyNames, unknown keywords, executable or custom +TypeBox features, arbitrary Unsafe and refinements, accessors, symbols and cyclic objects are rejected at +declaration. Bytes are the one special leaf: bytes({ min, max }) checks a Uint8Array and has the data marker +{ type: "bytes", minLength?, maxLength? }, so it hashes like any schema; TypeBox's checking still works on it.

+

Schemas are deep-copied into a private, frozen history; your own schema objects are copied, not frozen. +NORMAL_FORM_VERSION is part of every digest, so a change of the normal form is a link protocol change.

+

Not a security boundary. A fingerprint says "we were compiled against the same contract"; it proves nothing. A +hostile client can send any fingerprint it likes. Providers therefore always validate arguments with their own +schemas (chapter IV), whatever the fingerprint says.

+

Rules of thumb

+
    +
  • Treat a published contract like a published API: every change is evolve, every evolve needs a decision about +old consumers.
    • +
  • Keep mappings small and pure. A state mapping runs for every commit and every adapted binding; make it a map over +the value, not a query.
    • +
  • Accept the versions your users still run (one or two back during a normal rollout), and delete entries when the +metrics show no old bindings.
    • +
  • Change the input or the variants only with a new base, and only when updating every consumer is acceptable.
    • +
  • Contract, document and job versions are separate and each is decided on its own. Pinned notes needed NotesV2 +for the wire and NoteList version 2 with a migrate that adds pinned: false (chapter V), but no new +ImportJob version, because its outcome did not change.
    • +
+
Built so far

This chapter's contract layer is built: packages/chord/src/next/contracts/ (commit 7d2bb8ea8, about 1,300 lines +with remote, local, method, evolve, checkAccepts, Implementation, the schema helpers, fingerprintOf, +schemaNormalForm and NORMAL_FORM_VERSION), with nine test files covering evolution, fingerprints, the hash, +local contracts, long chains, schemas, types and review findings. It stands alone: it hosts nothing, calls no +mapping and does not import the frozen runtime. Binding by version and fingerprint, adapted state and calls belong to +the objects and protocol layers, not built yet.

+
+
+ +
XVII

On the Wire

What travels over a link, what a relay does with it, how big things are kept in check, what each hop costs, and how to see it in traces.

+

Most plugin code never thinks about the wire: it declares state, commits transactions and calls methods, and Chord +moves the bytes. You need this chapter when something is slow, when a value is large, when a link misbehaves, or when +you want to read a trace. It also explains why the earlier chapters insisted on small operations, shared structure and +no callbacks: each of those rules removes a cost on some hop.

+

What a state member sends

+

Every exposed document, derive or readable is a state member. Over a hold it produces one of these:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
UpdateWhen
snapon hold and on re-hold: the whole value, once
opsper commit: the draft's recorded ops, no diff
ops from a diffper change of a derive (and of an adapted binding, chapter XVI)
throttled opsjob progress
reseta migration commit replaced the value (adapted for older consumers)
errorthe member failed
+

Commits are atomic for consumers. On the same host, subscribers are notified once, after every document of the +commit is adopted. Remotely, one state message per hold carries a batch of member updates that is applied before +listeners run, and a commit that touches several holds on one link travels as one state message on that link. +A consumer never sees half a commit.

+

Messages

+

A message is a header plus opaque parts (R1). There is one part per hold the message concerns; the header says +which hold each part belongs to and how large it is.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DirectionKindParts
consumer to providerhold, release, call, cancelcall has one (the arguments)
provider to consumerboundone: the snapshot
stateone per hold the commit touched on this link
resultone
gonenone
+

There are no request ids in application code. Hold ids and call ids exist only inside the protocol, per link.

+

One binary format on every link (Q4): u32 header length | header JSON | part bytes, as one Uint8Array. A +WebSocket sends it as a binary frame, IPC passes it with serialization: "advanced", a memory link delivers it +asynchronously. Encoding, framing, header checks, id resolution, flow control and size limits are one shared protocol +layer; link adapters only move bytes. Tests on memory links therefore run the production path.

+

A part is encoded once, at its origin, with that hold's own Delta encoder (paths are interned per hold, end to +end between provider and consumer). A JSON part above a threshold (about 16 KiB) is compressed once at its origin, +whatever the link; byte segments stay raw; relays forward compressed parts as they are; the header names the codec.

+

The header is a TypeBox schema (R4), a union tagged by kind; the protocol types derive from it. Headers from +untrusted links (client WebSockets) are checked with the schema compiled once; trusted links (IPC between server and +workers) are not checked. Unknown ids are ignored on both sides (Q1). They are normal races: a result crossing a +cancel, a call crossing a gone, a state part crossing a release. The side that ended a hold or call settles +its own pending work: a consumer that receives gone(h) rejects its pending calls on h, and the order per +direction guarantees nothing else for h follows. Only a header that fails the schema is a protocol error, and it +closes that link.

+

The link protocol has a version. Hosts with different protocol versions refuse each other; a web page reloads to get +the server's protocol.

+

Through a relay

+

A tab never talks to a worker directly (chapter I). The server relays, and a relay reads and rewrites only +headers: it maps hold and call ids between its two links, stamps the principal on hold and call (a client's +principal field is always overwritten; only a trusted parent link may carry one), and groups parts per destination +link, so each link gets only its own holds' parts. It never parses a payload.

+
owner's tabhold h1Ann's tabhold h2serverrelayworker S1NoteStore, Notescall c7 Notes.add (1 part)map ids, stamp ownercall c3 (part forwarded as is)method runs, self.tx commitsstate: parts for h1', h2'group parts per linkstate (h1 part)state (h2 part)result c3result c7
Figure 11. The owner adds a note while Ann watches. The commit leaves the worker as one state message with two parts; the server splits it by destination without decoding either part.
+

The state message reaches the owner's tab before the result, so when await notes.add(…) resolves, the new note +is already in notes.value. Ann's view is a different variant (reader) and gets its own part from its own derive; the +relay neither knows nor cares.

+

Flow control

+

Each link has one byte window, counted from each message's byteLength (parts are already encoded bytes; nothing +is measured or stringified to count it). The window covers every message, first snapshots included.

+
    +
  • Snapshots are always admitted (R19), even when larger than the window; they use it up. Whether a hold +overflows is decided on the changes queued after the snapshot.
    • +
  • Under backpressure a commit's ops are collected per hold and encoded into one part when credits return. The +consumer still gets the commit atomically.
    • +
  • After an overflow the hold is dropped and established again once the previous snapshot's debt has drained +(credits back at the full window, Q3), and not more often than a minimum interval. So there is at most one snapshot +in flight per hold: no re-hold storm, and no stall for a snapshot larger than the window.
    • +
  • Calls are throttled by call credits the provider grants (a generous window). Beyond them calls wait locally. +Nothing is refused for being "too many", and holds are not limited by count; authorization and input validation +protect them.
    • +
  • Byte-heavy traffic such as terminals uses separate channels (chapter XIII), so a large state message never +holds up keystrokes.
    • +
+

Recovery: after a link drops, the consumer re-holds every hold over the new link and receives snapshots; calls in +flight fail as on any disconnect, calls made meanwhile park. A relay whose window overflows releases the hold and +re-holds; the provider sends a fresh snapshot. After a routed link drops, Chord re-resolves the address (with the +per-address backoff) while holds on it exist, so a passive viewer recovers without anyone pressing anything.

+

Size limits

+

Limits apply per part (R2, Q2), both to the compressed and the decoded size, and are checked from the sizes in the +header before anything is decoded; zlib's maxOutputLength enforces the decoded limit in the one decode function.

+ + + + + + + + + + + + + + + + + + + +
SituationWhat fails
a part over the limitonly its own hold or call, with too_large; the other parts of the message apply
a snapshot over the limitits hold ends failed, with no automatic re-hold
a WebSocket frame over maxPayload (e.g. 64 MiB)the connection closes (code 1009); this is the transport's hard cap
+

Defaults for client WebSockets are 16 MiB compressed and 64 MiB decoded per part, configurable per link kind. The +decoded limit travels with relayed work: the server stamps the admitting link's limit on the header at admission +(never taken from the client), and the host that decodes enforces it. IPC therefore needs no limit of its own.

+

Large data is never a large state member or a large argument. Files go through chunked transfers or the HTTP +component (chapter XIII); long lists are keyed documents or bounded windows (chapter VIII).

+

What each hop costs

+

Chord v2's goal is as little compute and memory as possible while staying correct: O(size of the change) per +commit and per viewer, independent of the size of the value. A 500 KB value is touched as a whole only when a new +viewer takes its first snapshot. Delta, the value library under documents, was built for this, and its own primitives +are almost free: diffRevisions on shared subtrees takes 0.018 ms, applyImmutable of a small change 0.004 ms.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
HopWork
commitDelta's prepare (copies and checks only the values placed in this commit), adopt by pointer swap; storage writes the delta
reads in a transactionthe draft overlay, no copy; an unchanged draft writes nothing
subscribers on the same hostthe same value and ops, by reference
derivescompute plus diffRevisions; unchanged parts stay the same objects (mapShared, chapter VIII)
sendone encoding per hold part at its origin, size taken from the encoded bytes
relayheader only: ids mapped, principal stamped, parts grouped; parts never parsed
receivecompiled header schema on untrusted links; arguments stripped and validated once at the provider (compiled schema); inputs normalized
applyapplyImmutable, structure shared
new holdthe value encoded once (unavoidable)
memoryone authoritative value per document per host; replicas and derived values share structure; one cached value per adapted binding
+

These are targets. The frozen kernel measured far worse, which is why the design is so strict here:

+
35 msone small change to a 500 KB value in the frozen kernel, no viewer
+13 msper viewer on top (10 viewers: 130 ms)
17 / 33 / 101 msa tiny SQLite commit after 0 / 10 / 50 earlier deltas
+

Nearly all of that came from layers around Delta: whole-value validation on every access, whole-value freezing, +JSON-ness walks on every received frame, double stringification to measure a frame, snapshots cloned per viewer, and +a SQLite storage that replayed history, and schemas rebuilt for every check (a compiled TypeBox check takes +0.012 ms, an interpreted one 5.6 ms). The redesign removes each of them (chapters V, VIII and this one); the +before/after per hop is in apps/p2/docs/p3-reviews/v2-cost-model.md.

+
Review: v2-cost-model.md is partly outdated

The measurements and the per-hop "after" targets in apps/p2/docs/p3-reviews/v2-cost-model.md are current. Its +closing section "Decisions this needs" is not: items 6 and 7 (validation by the caller's trust; contract descriptions +once per link and fingerprint) were replaced by R7 (validate for every caller) and R6 (no descriptions on the wire), +and §10 still describes the description fetch.

+
+

Cost guards

+

Chord reports full-value walks, deep copies, encodes, schema validations and materializations as ordinary +telemetry metrics, always on; with the NOOP adapter they cost an empty call where such work happens. Tests use the +in-memory adapter and assert them per hop. The reference case: one appended note to a 500 KB document with 10 viewers +does no walks, no copies, one encode per hold part at its origin and none at relays, and no validations. A +vitest bench suite measures the hops.

+

Ways to make it slow anyway

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MistakeWhy it costsInstead
a derive that builds new objects for unchanged dataevery commit diffs and sends O(value)keep unchanged items identical: mapShared, previous (chapter VIII)
one long array that changes often (R24)sharing is per path; each change copies one slot per element and visits eachkeyed documents or a bounded window
bytes as base64 stringsinflated, parsed, compressed as textbytes() leaves; chunked transfers for files
awaiting I/O inside self.txthe object's line is blocked for every callerdo the I/O first, then a short transaction
storing registry entries or references in documentsstale after a reload, not datalook up the current value at use
opening natives at module level or in a facetno owner: never closed, never restartedself.open in an object (chapter IX)
callbacks as argumentscannot cross a link, keep old code aliveregistries, jobs, flows
one big object that everybody viewsone large snapshot per viewer and per reconnectsplit by viewer need; views of the same variant share one value
+

Telemetry

+

Chord reports traces, events and metrics through @earendil-works/pi-telemetry. Each host gets one adapter, +createHost({ telemetry }), NOOP by default; the adapter decides sampling, buffering, export and filtering. Only hosts +hold the adapter, to start root spans. Code sees spans:

+
    +
  • ctx.telemetry is the current span (Ctx extends Context, so it is the same field Durable and pi-ai read). Every +span carries its adapter's metrics and propagation, so code with a ctx can record a metric or inject a carrier.
    • +
  • ctx.trace(name, attributes, fn) runs fn in a child span with a child ctx.
    • +
  • App values such as the principal stay out of Durable's context.
    • +
+

Across links, call and hold carry the caller's trace carrier as a bounded header field (R5: limited key count +and string lengths); the provider runs the method under it, and each relay adds a chord.relay span. The call from the +sequence above, as a trace:

+
chord.call Notes.add            [owner's tab]
+└─ chord.relay                  [server]
+   └─ chord.method Notes.add    [session:S1] principal=owner
+      · event chord.commit docs=2 ops=5
+

Adapters trust typed input: no validation of malformed values and no try/catch around recording. Only carriers from +another process are checked (keys present, strings), and a bad one falls back to a new root.

+

Shared work (one activation serving several gets, a worker start) is its own root span, linked to the trigger +that started it; each waiting span links to it. Routine runs, job runs, eviction, backoff retries and background +closes are root spans too, linked to their trigger where there is one.

+ + + + + + + + + + + + + + + + + + + +
TierWhat
spanscalls, methods, relays, holds, activations, opens, closes, loads, revocations, resolves, routine runs, job runs
eventscommits inside methods, retries, parking
metricsstate messages, WebSocket bytes, commit durations, in-flight calls, active holds, open links, park durations, backpressure waits, the cost guards
+

Ids are attributes of spans, never of metrics. Nothing is measured or walked for telemetry; only sizes the code +already has are reported. Errors carry a fixed category and a message cut to 500 characters; cancelling is not an +error. Chord ships a typed telemetry schema, as pi-ai (AI_TELEMETRY_SCHEMA) and Durable +(DURABLE_TELEMETRY_SCHEMA) do.

+

Storing and reading: @earendil-works/chord/telemetry-sqlite over chord/sqlite (Node and Cloudflare) stores one +row per settled span (events and links inline) and metrics aggregated per series per 10 s, flushes in batches every +second, keeps 7 days, and drops (and counts) beyond a bounded queue. The read interface TelemetryStore is shared: +findTraces (by name, attributes, errors, time window, with limit and cursor) returns complete traces, getTrace +one, findMetrics metrics. One renderer, pi-telemetry/render, prints traces as above; it prints links and never +follows them. The playground's trace viewer reads the same store.

+

Durable already emits submit and phase spans, stores carriers on submissions and tasks (TaskOptions.trace), +and pi-ai emits pi.ai.request and pi.ai.poll. With the harness given the host's adapter (chapter XIV), an agent +tool that calls Notes.add through chord(fn, context) shows up as one tree from the model request down to the +commit.

+
Built so far

Nothing of the protocol is built in v2: messages, relays, flow control and limits are layers L6 and L7, telemetry in +Chord is L11. Built in pi: pi-telemetry (propagation, links, metrics, the store interface, the renderer), +chord/context with Context.telemetry, pi-ai and Durable telemetry, and chord/telemetry-sqlite. Delta is built; +mapShared and bytes as atomic leaves in diffRevisions are still to do. The frozen kernel sends descriptions on the +wire and closes a link on an unknown id; both are replaced.

+
+
+ +
XVIII

Building p2: the Core

From a list of what the app must do to a set of owners: the base code every p3 plugin builds on.

+

The notes plugin used every Chord concept. A real app is the same concepts many times over. This chapter and the +next two assemble p3, the Chord v2 version of p2, from the parts the guide introduced. p3 is built next to p2 in +apps/p3, fully isolated with its own data directory, and without migrating p2's data (no import of +catalog.sqlite, shares.sqlite or the subagents table). Its plugins are derived from what the app must do, not +ported one to one from p2's plugin layout.

+
Review: p3 is not designed yet

The design fixes the rules (design.md §25) and lists the p3 decisions still open (#17, #18, #20 to #23); the +per-domain design of p3 has not started. What these three chapters show is one construction that follows the rules, +derived from the functionality map in apps/p2/docs/p3-functions/. Object names are working names. Where a domain +has an open decision, the text says which.

+
+

Start from functions, not from plugins

+

The functionality map in apps/p2/docs/p3-functions/ lists what p2 does, in plain words, with four facts per +function: the data and its size, how durable it is, who may see or do it, and where it must be available. It groups +the functions into 22 domains:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#Domain#Domain
1sessions12access
2session workers13models, credentials, settings
3conversations and agent14prompt resources
4shell commands15status and presence
5chat mode16search
6agents (subagents, one-shots, peer messages)17remote access and forwards
7code mode18logs and notices
8session files19dev and reload
9environments and execution20web shell
10workspace21TUI shell
11terminals22transport
+

The four facts answer the four questions every Chord object needs, in order:

+
THE MAP SAYSCHORD NEEDSdatawhat, how bigdurabilityephemeral · until restart· persisted ·transcript-boundwhoowner · guest read · guestwrite · workerwhereserver · worker · web ·TUI · environmentthe ownersmallest object it belongstowhere it is keptDurable, persisted orin-memory document, tabvariantsand revocationhost and codeplacement; core or pluginDurability decides storage: transcript-bound data is Durable's; persisted app data is a persisted document with astable identity; until-restart data is an in-memory document or a native; tab-scoped UI state stays in the client.
Figure 12. From a function in the map to code. Each fact in the map answers one question Chord needs answered.
+

One owner per thing, the smallest object it belongs to. A guest's grant belongs to the access owner, not to the +session; a draft belongs to one (conversation, author) pair, not to the chat plugin as a whole. Ownership nests, and +rules set at creation are written in the creating commit (Durable's createConversation({ init })).

+

Core or plugin. Core owners are what the app always needs and everything builds on; they are always present, so +other code may depend on them directly. Plugin owners may be missing; anything that depends on an optional feature +does so through a registry it owns itself (chapter XIII). The risk to avoid is a core god object: each core owner +owns only what belongs to it.

+

The ownership map

+

The result of that procedure for p3, as a starting point:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DomainOwner object(s)HostCode
accessAccess (grants, devices, share links), authenticateservercore
sessionsCatalog (session records), lifecycle jobsservercore
session workersthe supervisor (route(Session) + idle stop)servercore (app code)
session locationSessionLocation { session }workercore
conversations and agentHarness, ConversationStore, Conversation viewsworkercore
status and presenceSessionStatus (server copy), reporters on workersbothcore
shell commandsShellJob in the chat pluginworkerplugin
chat modechat pluginworkerplugin
agentssubagents plugin (Durable child conversations)workerplugin
code modecode mode plugin (sandbox native)workerplugin
session filesSessionFiles { session } (proposed, #17)servercore or plugin, open
environmentsenvs plugin (config, SSH pools #20)server and workerplugin
workspaceworkspace pluginworkerplugin
terminalsterminal host process (#23), terminals pluginseparate processplugin
models, credentialsmodels plugin, login job + flowserverplugin
settingsSettings (single writer, #21)servercore
prompt resourcesprompt plugin (watchers in the session's environment)workerplugin
searchsearch plugin (index on the server, feed on workers)bothplugin
remote access, forwardsradius plugin, ports pluginserverplugin
logs and noticesNotices (one channel), log captureservercore
dev and reloadthe reload control, reload_plugins toolserverplugin
web and TUI shellsUI facets over the objects aboveclientsshells
+

The rest of this chapter builds the core rows. Chapters XIX and XX build the plugins.

+

The base code

+
apps/p3/
+  src/
+    hosts.ts           Server, Session host kinds
+    principal.ts       P, Revocation, principalKey, revokedBy, can()
+    server/main.ts     createHost, load, route(Session), WebSocket and Radius entry points, supervisor
+    worker/main.ts     createHost with the start data, connect to the parent
+    core/              the core facets: Access, Catalog, SessionStatus, Settings, Notices (server);
+                       SessionInfo, SessionLocation, Harness, ConversationStore (worker)
+  plugins/             chat, subagents, codemode, envs, workspace, terminals, models, login,
+                       prompt, search, ports, radius, files, dev
+

hosts.ts, principal.ts, the SessionInfo start data and the Harness object are the ones the guide already +built for Studio (chapters I, III, IX and XIV). Base code restarts the hosts when it changes; everything in +plugins/ reloads.

+

Who is calling: authentication

+

Chord stamps principals; the app produces them. p3's server has four ways in, each turned into a principal by +authenticate(req) when a WebSocket connects (chapter III):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
Way inPrincipal
loopback without a cookie (TUI, CLI, first page load){ kind: "owner", access: "local" }
loopback owner browser cookie (a per-port secret file){ kind: "owner", access: "local" }
a device cookie (HMAC-signed, per port, 30 days), obtained with a one-time sign-in token{ kind: "owner", access: "device", device }
a guest cookie from a share link{ kind: "guest", session, mode, grant, revision }
the Radius relay channel{ kind: "owner", access: "relay" }
+

The checks that are not Chord's stay in this function: origin checks and Fetch Metadata (a browser request from +another site gets no principal, except a top-level navigation), proxy trust, the sign-in token rules (10 minutes, +single use, at most 32 outstanding, memory only), cookie expiry and revocation lists. A failed check closes the socket +before Chord sees it.

+

One permission matrix. Every function in the map names who may use it, and p2 checks those rules in several +places. p3 writes them once, in principal.ts, and every factory and authorize uses it:

+
// src/principal.ts
+type Action = "read" | "send" | "upload" | "stop" | "workspace" | "terminal" | "manage";
+
+export function can(p: P, action: Action, session?: string): boolean {
+  if (p.kind === "owner" || p.kind === "server") return true;
+  if (p.kind === "worker") return action !== "manage" && p.session === session;
+  if (p.kind !== "guest" || p.session !== session) return false;
+  if (action === "terminal" || action === "manage") return false;     // terminals: owner only
+  if (p.mode === "read") return action === "read";                     // no workspace for guest read
+  return true;                                                         // guest write: send, upload, stop, workspace
+}
+

Factories pick variants from it (can(p, "send", session) ? "writer" : "reader"), so the matrix is visible in one +file and the members a caller lacks are simply not there.

+ +

Access is the core owner of everything that lets someone in: share links per session and mode, guest records, the +device list and the signing key. It lives on the server as the chapter IV pattern: a local store that persists the +documents (only stable identities may), and a remote view for the owner's settings screen:

+
// src/core/access.ts (server, in the core server facet)
+const Links = defineState("access.links", { schema: map(ShareLink), version: 1, persist: true,
+  initial: () => ({}), checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64 });
+const Guests = defineState("access.guests", { … });         // same shape: map(Guest), persisted
+const Devices = defineState("access.devices", { … });       // map(Device), persisted
+
+host.serve(AccessStore, (_input, self) => {                 // local: may persist, acts for the host
+  const links = self.state(Links), guests = self.state(Guests), devices = self.state(Devices);
+  return {
+    links, guests, devices,
+    removeGuest: async ({ grant }, ctx) => {
+      await self.tx(async (tx) => { (await tx.draft(guests))[grant].blocked = true; }, ctx);
+      host.revoke({ grant });                                 // live access ends now
+    },
+    resetLink: async ({ session, mode }, ctx) => {
+      const blocked = await self.tx((tx) => rotateLink(tx, links, guests, session, mode), ctx);
+      for (const grant of blocked) host.revoke({ grant });
+    },
+    signOut: async ({ device }, ctx) => {
+      await self.tx(async (tx) => { (await tx.draft(devices))[device].revoked = true; }, ctx);
+      host.revoke({ device });
+    },
+    …                                                         // shareLink, join, signOutAll
+  };
+});
+
+host.serve(Access, (_input, principal, self) => {           // remote view: owner only
+  if (!can(principal, "manage")) throw forbidden();
+  const store = self.get(AccessStore, {});
+  return {
+    variant: "owner",
+    guests: self.derive("guests", [store.guests], ([g]) => publicGuests(g)),   // names, modes; no tokens
+    removeGuest: (args, ctx) => store.removeGuest(args, ctx),
+    resetLink: (args, ctx) => store.resetLink(args, ctx),
+    signOut: (args, ctx) => store.signOut(args, ctx),
+  };
+});
+

authenticate is base code on the same host and checks AccessStore's current documents on every new connection +(it holds a reference to the store, taken with server.root at start). Live access and lasting denial stay separate, as chapter VII explained:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EventLasting effect (data)Live effect
remove a guestguest record blockedrevoke({ grant })
reset a share linknew token; its guests' grants blockedrevoke({ grant }) for each
archive a sessiongrants of that session void while archivedrevoke({ session })
restore a sessiongrants valid again; revision bumpednone: guests reconnect
sign out a devicedevice revoked until its expiryrevoke({ device })
sign out all devicessigning key rotated, tokens clearedrevoke({ device }) for each
+

A guest who rejoins with the same browser keeps its identity; a removed guest cannot mint a new one, because the +block is in Guests and authenticate refuses it. Names are unverified, cleaned, at most 40 characters, unique per +session ("Ann (2)").

+

The catalog

+

Catalog owns the session records: id, title (and whether it was edited), location, model default, pinned, +archived, timestamps. It lives on the server, persisted, and it never starts a worker: listing sessions, showing +their status or their unread marks reads server-side copies only.

+
export const Catalog = remote("p3.catalog", {
+  version: 1,
+  input: obj({}),
+  placement: () => Server(),
+  variants: {
+    owner: {
+      state: { sessions: obj({ items: arr(SessionRow) }) },
+      methods: {
+        create: method(CreateArgs, str()), rename: method(RenameArgs), pin: method(PinArgs),
+        archive: method(obj({ id: str() })), restore: method(obj({ id: str() })),
+      },
+    },
+    guest: { state: { sessions: obj({ items: arr(SessionRow) }) } },     // only the guest's own session, if not archived
+  },
+});
+

Behind it sits a persisted local CatalogStore; the guest variant is a derive over it, as in chapter IV. The lifecycle operations (archive, clone, fork, +relocate, delete) are chapter XIX; here only the rule that keeps the catalog small: the catalog does not know which +plugins exist. Archiving a session writes archived: true and revokes the session; every plugin that keeps +per-session things reconciles from catalog.sessions with a routine (chapter X), as the notes plugin did. The ports +plugin closes forwards of archived sessions that way, also after a crash.

+

On the worker: start data, location, harness

+

Three core objects exist on every worker from its start:

+
    +
  • SessionInfo (chapter IX): the start message as constant(init), with the session id, data directory and the +location the server chose at start.
    • +
  • SessionLocation { session } (working name), lifetime host: the current cwd and environment, with a location +revision. Relocation changes it while the worker runs (chapter XIX); natives that depend on the location follow it +with a routine and a generation (chapter IX), and conversations that follow the session's cwd are listed in the core +per-conversation policy document.
    • +
  • Harness (chapter XIV): the Durable harness for the session's store, one SQLite database per session, opened +under an exclusive lock keyed by the real data directory and session id. It refuses to start when the saved root +model is not configured, and resumes interrupted Durable work on start.
    • +
+

ConversationStore and the Conversation views in front of it are chapter XIX.

+

Workers: start, keep alive, stop

+

Starting, supervising and stopping workers is app code (chapter I); Chord only routes and counts. p3's supervisor is +server.route(Session, { authorize, resolve }) plus a small server object that keeps per-session worker state:

+
A worker's life, as the supervisor sees it

Authorize, then start

A tab gets Notes { S1 }. authorize checks can(p, "read", "S1") and that the session exists and is not archived +(an archived session refuses, except an index rebuild that starts the worker without resuming work). resolve forks +worker/main.ts with the start data from the catalog, bounded by a 60 s readiness timeout.

+

A start failure sticks

If the worker fails to start (an unconfigured environment, a missing model), the error is kept per session and +every get fails at once with it, until the owner presses Retry. That clears the error and the related notices. +Guests see that the session is unavailable, not the error text.

+

Counting

The core facet on the worker reports host.activity ({ references, calls, jobs, lingering }) and the harness's +running Durable task count to the supervisor object, from a routine whenever either changes. The call carries +ctx.incarnation, so a report from an older worker incarnation is ignored (chapter X).

+

Idle stop

When everything has been zero for five minutes (checked every second, as in p2), the supervisor stops the worker: +the worker closes its host (routines stop, natives close, storage flushes) and exits. Durable work stays persisted +and resumes on the next start. Detached work that must keep the worker alive (a shell command whose caller left) is +a job, so it counts.

+

Crash

If the worker exits on its own, the link drops. The supervisor records the exit code and a redacted stderr tail +(at most 4,000 characters) for that incarnation and posts one notice. References that still exist make Chord call +resolve again with backoff; a passive viewer recovers without a click.

+

Shutdown

On server shutdown the supervisor stops every worker, with a forced exit after three seconds. Durable work remains +persisted.

+
+

Status without a worker

+

The session list shows per session whether it is running, busy (which conversations), its task count, usage and cost, +and an error. None of that may start a worker. The pattern is chapter X's pub/sub: a reporter on each worker (a +routine over the harness's state) calls SessionStatus.report(summary) on the server; SessionStatus keeps the latest +value per session and ignores calls whose ctx.incarnation is older than the newest it has seen; clients read +SessionStatus, which lives on the server. Children lists, unread marks, search index rows and artifact lists use the +same pattern (chapter XIX).

+

Settings and notices

+

Settings (default model and thinking, run settings, compaction, queue modes, code mode switch, owner name) have +one writer: a core Settings object on the server owns the file, validates writes and serializes them; every +other process follows its state through a reference instead of watching and rewriting the file itself. p2 watches +settings, models and auth in every process and writes them from two, which is decision #21.

+

Notices are one channel: one core Notices object on the server, written by everything that fails or finishes +(worker crashes, reload results, model errors, login outcomes, grants), keeping one latest notice per source, at most +50. The owner's shell reads one list; dismissing is a method. p2 has five notice channels for the same thing.

+
Review: settings and file formats (#21, #22)

Single writer for settings, models and auth (#21) and versioning on-disk formats such as envs.json like documents +(#22) are open p3 decisions in design.md §26. The construction above assumes "one owner on the server, others +follow its state" for #21.

+
+
Built so far

Nothing of p3 exists. The functionality map is complete (apps/p2/docs/p3-functions/, five area files plus the +condensed README.md), and the design's app rules (design.md §25) are decided. p3 starts after the layered rebuild +and the playground's first real proof.

+
+
+ +
XIX

Sessions and Conversations

The session lifecycle, status without a worker, chat, shell commands, subagents and search, each mapped to an owner and the Chord parts it uses.

+

This chapter builds the domains a person touches most: the session list and its operations, the conversation view +and composer, shell commands, chat mode, subagents, messages between sessions, search and unread marks. Each section +names the owner, the host, and the Chord parts it uses; code appears only where it shows something new.

+

The session lifecycle

+

The catalog (chapter XVIII) owns the session records. Its operations, by the Chord part each needs:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OperationOwner and formNotes
createCatalog.create, a methodabsolute directory, ~ expanded, must be a directory; on an environment, resolved there through the envs plugin; refuses an unknown model; model and thinking default to the last used
rename, pinmethods on Catalogtitle 1 to 120 characters, whitespace collapsed; an explicit title is never replaced automatically
automatic titlea routine on the search feedthe first user message, line breaks to spaces, 100 characters
agent renamean agent tool on the worker calling Catalogroot conversation only
archive, restoremethods; revocation; reconciling routinesbelow
clone, fork, relocatejobs on the catalog storeoutlive the call, cancellable, one outcome
deletea job; decision #18below
+

Archive and restore

+

Archiving must stop the worker whatever it is doing, end guest access, and close the session's port forwards, while +keeping its storage. The catalog does none of the plugin-specific parts:

+
Archiving S1

One write

Catalog.archive({ id: "S1" }) sets archived: true in the session record, in one transaction.

+

Revoke the session

The catalog calls host.revoke({ session: "S1" }). Every guest principal of S1 has { session: "S1" } in its +revokedBy, so their objects end and their links close on the server and on the worker (chapter VII). New +connections are refused by authenticate, because grants of an archived session are void.

+

Stop the worker

The supervisor reconciles from catalog.sessions with a routine: an archived session's worker is stopped, and +authorize refuses to start it again.

+

Plugins reconcile

The ports plugin's routine on catalog.sessions closes forwards of archived sessions. The search plugin marks the +session's rows archived. If the server crashed between the write and these steps, the routines run again with the +current value at the next start: nothing is a notification that can be lost.

+

Restore

Catalog.restore clears the flag. Grants are valid again, so guests reconnect and authenticate as before; forwards +stay closed.

+
+

Clone, fork, relocate

+

Each is work that outlives its call, can be cancelled and has one outcome, so each is a job (chapter XI) on the +catalog's local store, keyed by the owner's request id. They differ in what they pause:

+
    +
  • Clone copies the session's store, uploads, downloads and artifacts under a new id, with the source worker +paused (refused while work runs) and title "(copy)"; directories 0700, files 0600, no links, no overwrite; a +partial copy is removed on failure.
    • +
  • Fork before a user message copies the active context as of the entry before it into a new session (title +"(fork)"), with the message waiting in the new session's editor. It needs two workers: the source reads history, +the new one is seeded empty. A failure archives the copy.
    • +
  • Relocate (cwd, environment or both): refused while the session is busy (naming the running subagents) or when +a subagent has its own cwd and the move crosses machines. The job fences the worker (new admissions wait), updates +the record, and moves the running worker in place by updating SessionLocation: natives that follow the location +switch with a generation (chapter IX), and conversations that follow the session's cwd are rewritten in one Durable +commit before work resumes. After 60 s it restarts the worker instead; on failure it releases the fence.
    • +
+
Review: jobs as objects (#9)

The design defers whether a long relocation or a login is a record inside its owner (watched through running) or an +object of its own (Relocation { id }). The construction above uses records. Decide when login is ported +(design.md §15, §26).

+
+

Delete

+

p2 has no session delete. p3 needs one that removes the session everywhere: every document tagged with the session +(store.retireByTag(SessionTag, id), chapter V), the worker's databases (the Durable store), the session's files, the +search rows, read marks, children copies, forwards, guest records, and finally the catalog entry. The design commits to +this outcome (design.md §25); how plugins take part is open.

+
Review: session delete (#18)

Options in design.md §26: per-session objects plus SessionTag plus one session directory (delete is retire by tag +and remove a directory, plugins need do nothing), or owner hooks contributed through a catalog registry. Also open: +whether the search index is per session (deleted with it) or global with a hook. Who holds the store for +retireByTag is not specified either.

+
+

Status, children, unread: reports to the server

+

The session list must show status, subagents and unread marks for sessions whose workers are not running, and must +not start them. All three follow chapter X's pub/sub pattern: a routine on the worker calls a server object with the +current summary, the server object keeps the latest value per session (persisted where it must survive a restart), +ignores calls from an older ctx.incarnation, and clients read the server object.

+ + + + + + + + + + + + + + + + + + + + + + + +
SummarySource on the workerKept on the server
status: running, error, task count, busy conversations, usage and costthe harness's task graph and usageSessionStatus, in memory: a stopped worker is "not running"
children: name, model, thinking, tools, cwd, last entry, archivedthe subagents plugin's child listChildren, persisted
latest root answerthe search feedUnread: compared with the owner's read mark, persisted, shared by all owner devices
+

Guests see their own session's status without the error text. The owner's read mark is written when the root +conversation is visible, and a new session starts as read.

+

Conversations

+

The harness (chapter XIV) owns the Durable store. The UI needs, per conversation: a live window of entries, older +pages, usage numbers, the working indicator, the inbox (queued inputs), and the actions. Two core objects provide them:

+
    +
  • ConversationStore { conversation } (local, on the worker): opens the Durable watch as its native, reaching +the harness through the core-private HarnessRuntime registry entry, and exposes a window as a derive: the +newest entries (at most 100 or 512 KiB, at least 20) plus the conversation's documents. The window's start stays +put while it is within 1.5 times the limits, so new entries arrive as small appends instead of a shifted list +(chapter VIII's window pattern).
    • +
  • Conversation { session, conversation } (remote, placement Session): the per-principal view. Variants by +can(): owner (everything, including model, thinking and handoff), writer (guest write: send, steer, stop, +compact, shell), reader (the window only).
    • +
+
host.serve(Conversation, ({ session, conversation }, principal, self) => {
+  const store = self.get(ConversationStore, { conversation });
+  const policy = self.get(ConversationPolicy, { conversation });         // core: readOnly, followsSessionCwd
+  const author = authorOf(principal);                                    // structured: kind, id, display name
+  if (!can(principal, "send", session)) return { variant: "reader", view: store.window, page: store.page };
+  return {
+    variant: can(principal, "manage") ? "owner" : "writer",
+    view: store.window,
+    page: store.page,                                                    // older history: returns data
+    send: ({ text, attachments, mode, requestId }, ctx) => {
+      if (policy.value.readOnly) throw forbidden("read-only conversation");
+      return store.submit({ type: "input", content: text, attachments, author, whenBusy: mode, requestId }, ctx);
+    },
+    stop: (_args, ctx) => store.stop(author, ctx),                       // allowed even when read-only
+    …
+  };
+});
+

What each function becomes:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FunctionHow
send, retried sendDurable submit with the author's requestId: a retry with the same id is admitted once (Durable dedupes, chapter XIV)
steer or queue while busywhenBusy: "steer" or a follow-up; the inbox is docs["pi.inbox"] in the window
attributionthe view derives the author from the principal; the entry stores it structured, the model sees a speaker block; nothing parses text
wait for prompt resourcesthe first input of a fresh worker waits until the prompt plugin's resources are loaded (a registry readable the store follows)
stop, dequeuewithdraw queued inputs, abort the run; the caller's texts return to its editor, other authors' to their restored drafts
restored draftsa keyed document per (conversation, author), tagged SessionTag, persisted in the store; claimed once, first tab wins
read-only conversationsthe core per-conversation policy document, written at createConversation({ init }) by whoever creates the conversation; send, dequeue, configure, compact and shell refuse; stop is allowed
history pagesa method returning a page (100 entries, 1 MB, 2,000 scanned); compaction and reset markers always included
model and thinkingowner variant only; validated against the models plugin's catalog; the root conversation's model is also reported as the session default
compaction, handoffDurable operations through the store
+

Drafts in the composer are UI state, not Chord state: the web shell keeps one draft in sessionStorage, the TUI +keeps one per conversation across UI replacements, and pending sends keep their request id until acknowledged +("Sending when connected"). Only restored drafts, which the worker produces, are documents.

+

Shell commands

+

!cmd runs a command in the conversation's cwd on its environment and records the result as an entry; !!cmd does +the same but keeps the output out of the model's context. One command at a time per conversation; the live tail is +at most 50 KB or 2,000 lines, sampled every 200 ms; the full output spills to a file; the result entry is placed at the +run's next boundary while the agent is busy and never starts a run. This is the design's job example almost as is:

+
const ShellJob = defineJob("chat.shell", {
+  version: 1,
+  args: obj({ command: str({ max: 100_000 }), include: bool(), author: Author }),
+  progress: obj({ output: str() }),
+  result: ShellResult,
+  concurrency: 1,                                                   // one per conversation
+});
+
+const results = self.state(ShellResults);                           // keyed by run id: dedupe after completion
+const shell = self.job(ShellJob, {
+  run: async ({ command, include, author }, ctx) => {
+    const out = new OutputBuffer({ maxBytes: 50_000, maxLines: 2000, retain: "tail" });
+    const env = await environmentFor(ctx);                          // the conversation's environment, chapter XX
+    const r = await env.exec(command, { cwd: env.cwd, signal: ctx.signal, spill: spillPath(ctx.jobId) },
+      (chunk) => { out.push(chunk); ctx.progress({ output: out.snapshot().text }); });
+    const data = { command, output: out.snapshot().text, exit: r.code, cancelled: ctx.signal.aborted };
+    await conversation.submit(
+      { type: "write", entry: shellEntry(data, author, include), requestId: `shell:${ctx.jobId}` }, ctx);
+    return data;
+  },
+  finish: async (done, tx) => {
+    (await tx.draft(results, done.id)).outcome = done.kind === "done" ? done.result : { error: done.error };
+  },
+});
+

The request id shell:${ctx.jobId} makes the entry idempotent: an attempt interrupted after the write cannot write +it twice. Cancel is shell.abort(id), allowed for the command's author and the owner. A retried runShell with the +same request id returns the running job (dedupe of unsettled runs), or the stored result (the owner's own dedupe after +completion, chapter XI). p2's "cancel when nobody watched for 30 s" is dropped: a job keeps its owner alive and does +not depend on viewers.

+

Chat mode

+

Chat mode is a per-conversation switch that everyone in the session sees and only the owner changes; forks and new +subagents start with it off. Because it must fork and rewind with the conversation, it is transcript-bound: a +Durable conversation document, not a Chord document. Messages without @pi become display-only Durable write +entries (author, text, attachments, reply quote); a message with @pi becomes an input with a digest of the chat +since the last mention (at most 100 messages, 20 files, 4 images), recording which entries it passed on. The reply +quote is derived on the worker from the entry id the client names: the client never supplies author or quote text.

+

Subagents and one-shot agents

+

Subagents are Durable child conversations owned by tasks, as in Durable's own background-subagent example: one +subagent tool spawns, steers, waits for, stops and lists them; each child is owned by a background anchor task, and +each message is delivered by a reporter task that posts the answer back to the parent once, with request ids so a +restart sends nothing twice. Chord adds only three things:

+
    +
  • the child list for the sidebar is reported to the server (Children, above), so it shows without a worker;
    • +
  • a child that follows the session's cwd, or a read-only one-shot agent created by a code-mode script, is marked in +the core per-conversation policy document when it is created, so chat and relocation enforce it without knowing +the subagents plugin (design.md §25 porting guideline);
    • +
  • archive and restore of a child from the UI are methods on the subagents plugin's view, refused for one-shots.
    • +
+

Subagents cannot spawn subagents; thinking levels are clamped to what the model supports, in one helper shared with +the session's model choice.

+

Messages between sessions

+

The root agent can list other sessions and send one an attributed message. The tool runs on the sender's worker and +calls a server object through chord(fn, context) (chapter XIV):

+
const peerSend = defineTool({
+  name: "session_send",
+  execute: (args, api, context) => chord(async (ctx) => {
+    const peers = await ctx.get(Peers, {});                       // server, worker variant
+    await peers.send({ to: args.session, text: args.text, mode: args.mode, requestId: `peer:${api.taskId}` }, ctx);
+    return { output: [{ type: "text", text: "sent" }] };
+  }, context),
+});
+

Peers checks that the sender is a root conversation of a non-archived session, refuses sending to itself and +archived recipients, then gets the recipient's Conversation (which starts its worker) and submits with the +sender as a structured author and the same request id (the tool task's id, stable across restarts), so a retried +tool call delivers once. The sender's +incarnation is ctx.incarnation; a call from a replaced worker is refused. Only owner-initiated turns may use the +tool; guests never.

+ +

The search plugin indexes user and assistant messages of root and subagent conversations (no shell output, no +Durable internal inputs) for title, cwd and message search across sessions, active or archived.

+
WORKER S1SERVERDurable commitin any conversationwakea routine on the harness's commitsignalread from cursor200 entries per pageSearchIndexnative: SQLite FTScursor per conversationmoves after the whole rangequeryterms, scope, at most 41 hitsreport(page)Reports carry ctx.incarnation. Indexing is idempotent: a page sent twice changes nothing.
Figure 13. The search feed. The worker pushes pages from a cursor; the server owns the index and never starts a worker to answer a query.
+

The index is a native owned by a host-lifetime object on the server; cursors live with the index, so after a crash +the worker asks for the cursor and continues. Rebuilding one session's index drops its rows and rescans, starting +an archived session's worker without resuming its work. Queries are methods returning data: at most 41 hits, +snippets of 300 characters, queries up to 500 characters. The same feed sets automatic titles, moves "updated at" +forward and supplies the latest root answer for unread marks.

+

Presence

+

Viewers per session are host.references(Conversation) on the worker, counted per client (chapter VIII), and +reported with the status. Connected guests (name and mode) are the references whose principal is a guest. Presence +is not keep-alive: the supervisor uses host.activity, which also counts calls and jobs.

+
Built so far

None of this exists in p3. The parts it relies on are Durable (built: submit with request ids, steer and queue, +child conversations, background tasks) and the Chord layers L3 to L9 (not built).

+
+
+ +
XX

Machines, Files, Models

Environments, workspace, terminals, session files, ports and Radius, models and logins, prompt resources, logs, reload and the shells: the rest of p3.

+

The last domains are the ones that touch the outside world: other machines over SSH, the file system and git, +terminals, the relay, model providers and their credentials. Most of them are native objects, and most of them are +places where p2 has the same thing several times (four SSH connections to one environment, five notice channels, +three copies of the environment config parser). The rule that fixes that is the one from chapter XVIII: one owner per +thing.

+

Environments

+

The envs plugin owns the environment configuration: name, host, user, port, identity file, shell and its +variables, plus a managed ed25519 key per environment and its own known_hosts (kept apart from the user's, aliased +per environment name).

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FunctionOwner and form
list, add, edit, delete (refused while a session, archived too, uses it)Envs on the server: a persisted document behind an owner-only view
validation (no leading -, no whitespace or control characters in host and user)the view's methods, before the write
a hand-edited envs.json is picked up; an invalid file keeps the last valid set and shows an errora native file watcher (poll fallback) in the envs store; one writer (#21)
environment revision: a hash of the config, part of its identitya derive
connection test (connect, deploy the daemon, platform, latency, warnings)a job; the result per revision in a document
unknown or changed host key: fingerprints, trust after confirmation, forgeta flow (chapter XII) on the server
browse directories (~, relative paths, at most 10k scanned, 1,000 shown)a method with a call-scoped SSH session (chapter IX) or the pooled connection
a changed environment restarts its sessions' workers once idlea routine on the envs store and the status reports
+

Execution in the agent. The worker's harness asks HarnessEnvironments for an environment per target (chapter +XIV). The envs plugin contributes it from its worker facet: local execution at the session's cwd, or a remote one over +SSH through the pi-env daemon, deployed per platform and connecting at the first operation. A remote session never +falls back to local tools. The environment follows SessionLocation: after a move, the old connection closes once +the move has committed and watchers reopen at the new location.

+
// envs/worker.ts
+export default defineFacet((host, self) => {
+  const location = self.get(SessionLocation, { session });
+  const pool = self.get(EnvPool, {});                              // one owner of SSH connections (#20)
+  self.contribute(HarnessEnvironments, (target) =>
+    target.env === "local" ? localEnv(target.cwd) : pool.environment(target.env, target.cwd));
+});
+
Review: connection pools (#20)

SSH connections to one environment exist four times in p2 (worker environment, browse connection, test connection, +terminal host). design.md §26 asks for one owner with a pool or deliberate per-use connections. EnvPool above is +the first option: a native object that closes idle connections after 10 minutes and invalidates them on a config or +key change. Reloading the plugin that owns the pool kills running remote commands (chapter XV), so it should change +rarely.

+
+

Workspace and file index

+

The workspace plugin serves the viewed conversation's working tree (a subagent's own cwd included) on the worker: +list a folder (at most 2,000 entries, .git hidden), read a file (1 MiB, binary detected), git status, diff, log and +single commits, literal text search, and all paths for name filtering. Every query is a method returning data, not +state: results are bounded and requested by the UI. Only change notifications are state: a counter in a document, +bumped by a native watcher of the tree, HEAD, the index and config (debounced 100 ms); viewers reload what they show +when it changes.

+

Two rules live in one place each instead of three:

+
    +
  • Confinement: path rules, a canonical root, no .git, the app's private data directory denied (also excluded +from search). One helper used by listing, reading, uploads and the file index.
    • +
  • Git hardening: no optional locks, no fsmonitor, filters disabled, no prompts or pager, no system config, 10 s +timeout, output caps.
    • +
+

Downloads (25 MiB, byte ranges for media) go through the HTTP component (chapter XIII) with a route on the server that +reads chunks from the worker; uploads into the project use the chunked protocol of chapter XIII (1 MiB chunks, temp +file then rename, never overwrite, writes serialized). Guest read gets no workspace: the view has no variant for it.

+

The file index for @ completion belongs to the same owner, so completion consumes the index instead of scanning +on its own: built from the environment's listing (else git or a walk, at most 50k paths), rebuilt at most once per +second while files change, discarded after a move. Review comments drafted on files and diffs are browser state until +sent; sending them is an ordinary Conversation.send.

+

Terminals

+

Terminals must survive server and worker restarts, so their PTYs live in a detached terminal host process, one per +data directory, with its own process lock; it exits after 10 minutes with no terminals and no clients. The terminals +plugin on the server is the owner the UI talks to: list (title, cwd, environment, size, exit status), open at the +session's cwd and environment (refused for archived sessions), attach, close (asking first if a foreground job runs). +Terminals are owner only.

+

Bytes do not go over object links. Attaching opens a channel (chapter XIII) with a single-use token (30 s): JSON +control on the object, bytes on the channel, with its own acknowledged window (256 KiB, acknowledgements per 64 KiB; a +viewer that falls behind gets a fresh screen snapshot; an 8 MiB backlog disconnects). Attaching gets the current +screen from a headless emulator with 10k lines of scrollback.

+
Review: the terminal host process (#23)

The terminal host runs its own code. Is it part of the revision, reloaded or restarted with it (and running terminals +end), or deliberately outside it with a "host runs older code" notice as in p2? Open in design.md §26.

+
+

Session files

+

Attachments (25 MiB, at most 20 per message), sent-file snapshots made by the agent's send_file tool, versioned +artifacts (HTML, SVG, Markdown, text; 2 MiB; each update a new immutable version) and shell spill files are all +per-session bytes with caps, immutable ids and range reads; p2 has two upload routes, two caps and two directory +layouts for them. The proposal is one owner:

+
export const SessionFiles = remote("p3.session-files", {
+  version: 1,
+  input: obj({ session: str() }),
+  placement: () => Server(),                                  // readable without a worker
+  variants: { writer: { … }, reader: { … }, worker: { … } },
+});
+

SessionFiles { session } on the server holds the index (per kind, readable without a worker), the chunked upload +and read protocol, the caps per kind, copying on clone and fork, and deletion with the session. Attachments are copied +into a remote environment before an input is admitted, with no fallback on failure. Artifact previews run in a +sandboxed frame with its own CSP and no app identity; ordinary file previews allow only safe types.

+
Review: session files (#17)

Proposed in design.md §26, not decided. Open sub-points: one preview limit (p2 has 256 KiB for sent files and +512 KiB for artifacts), what a fork copies of the artifact list, and how attachments are copied into remote +environments. Related: #22, whether on-disk layouts such as downloads and uploads get versions like documents.

+
+

Ports and Radius

+

The radius plugin keeps the outbound relay connection, a native in a host-lifetime object on the server: sign-in +by device code from the CLI (credential in a private file), token refresh, a heartbeat in the link adapter (20 s ping, +60 s timeout; TCP does not notice a sleeping laptop), backoff from 1 to 30 s, a disable switch. The relay delivers two +kinds of channels:

+
// radius/server.ts (the entry point: one line, as in chapter I)
+radius.onChannel((ws) => host.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } }));
+radius.onForward((channel) => proxyToForward(channel, host.registry(RadiusForwards).value));
+

Object links become owner principals with access: "relay": owner, but treated as remote (no localhost-only +powers). p2 had to fake a loopback request with a secret header for this; with accept that is gone.

+

Forward channels go to localhost:<port> (Host, Origin and Location rewritten; 502 when nothing listens; anonymous +visitors only to public forwards). Which forwards exist is not radius's business: it owns a registry, +radius.forwards, and the ports plugin contributes its forwards (at most 50, random ids, a public toggle that +needs confirmation, an agent tool ports for top-level owner-initiated conversations). Without the ports plugin the +table is empty and radius still works: an optional dependency is a registry owned by the dependent (chapter XIII). The +ports plugin closes forwards of archived sessions with a routine on catalog.sessions (chapter XIX).

+
Review: may a plugin call accept?

design.md §3 shows the Radius line in the server's start code and calls entry points "app or adapter code". If +radius is a plugin, its facet needs the host's accept, which no other part of this guide gives a plugin. The +alternative is that the start code keeps the entry point and gets channels from the radius object. Not specified.

+
+

Models and credentials

+

The models plugin owns the model runtime (pi-ai's models with custom providers and the model cache) in a +host-lifetime object, and the credentials. It contributes the runtime to HarnessModels (chapter XIV), so after a +models reload the next request uses the new runtime while a running stream finishes on the old one. Credentials +never go to clients: the views expose providers, login methods and which type is signed in, never tokens. Every +secret is registered for redaction in one place, the log capture.

+

Views: a model catalog for pickers (names, ids, context windows, thinking levels; at most 300 matches in the web), +logout (refused during a login), refresh a provider's model list. The default model and the cycling scope are +settings (chapter XVIII).

+

Login

+

A login is the guide's flow example (chapter XII): a job on the server that runs pi-ai's login, and a flow that +shows its steps to the starting owner.

+
const LoginJob = defineJob("models.login", {
+  version: 1, args: obj({ provider: str(), type: AuthType }), progress: LoginState, result: obj({ ok: bool() }),
+  concurrency: 1,                                                    // one login at a time on the server
+});
+
+const login = self.job(LoginJob, {
+  run: async ({ provider, type }, ctx) => {
+    await runtime.value.login(provider, type, {
+      signal: ctx.signal,                                            // 10 minute deadline, cancel, leaving
+      prompt: (prompt) => steps.ask(prompt, ctx),                    // text, secret, select, manual_code
+      notify: (event) => steps.notice(event, ctx),                   // info, auth_url, device_code, progress
+    });
+    return { ok: true };
+  },
+  finish: async (done, tx) => { /* record the outcome; refresh the model list */ },
+});
+

prompt is a promise-based callback inside the work; the resolver stays in the job (steps.ask records the step in +the flow state and waits for answer), and nothing crosses an object boundary. On a relayed connection the device +code method is selected automatically. The flow is visible only to the owner principal that started it; others are +refused. Leaving ends it: when the starting owner's references are all gone (presence, not an unmount), a routine +aborts the job. Codes, URLs and typed secrets are redacted from logs.

+

Prompt resources

+

AGENTS.md, skills and templates are read in the session's environment, watched (poll fallback, last valid state), +and expanded on the worker before sending. The prompt plugin owns them per session on the worker, contributes the +system prompt section and skills to HarnessExtensions as readable contributions (a change recomposes), and exposes a +readiness readable: the first input of a fresh worker waits until it is ready (chapter XIX). After a move the watchers +reopen at the new location.

+

Code mode

+

Code mode is a plugin (chapter XIV): its pi-codemode sandbox is a native; scripts call tools as nested Durable calls +and replay after a restart; the tool entry looks up the current sandbox when it starts. A reload seals and drains the +old sandbox for at most 30 s. The on/off switch is a setting.

+

Logs and notices

+

Log capture tees stdout and stderr of the server, workers and detached hosts without changing their behaviour, +redacts known secrets and credential patterns, and keeps bounded rings (at most 128 sources, 1,000 lines or 256 KiB +each). Detached hosts publish redacted snapshots to a private file. Logs are read without waking a stopped worker. +Notices are the one channel from chapter XVIII; dismissal stores only notice ids.

+

Dev and reload

+

The dev plugin is chapter XV's reload control for p3: a source watcher marks plugins dirty and never applies anything; +the reload_plugins agent tool and a control in the owner's shell reload the changed plugins, or restart all hosts for +base code (through the supervisor, reopening the sessions that were running). UI bundles are published only after a +successful backend change; a web page detects a new manifest and reloads with its drafts and navigation state saved; +the TUI swaps its sandboxed UI code with a snapshot of its state. A local Unix socket (0700) accepts development +commands without a network admin port.

+

The shells

+

The web and TUI shells are UI facets over the objects above. Two rules matter for Chord:

+
    +
  • UI state survives navigation, reload and reconnect on purpose. Composer drafts, pending sends with their request +ids, reading positions and review drafts are kept in tab storage or the TUI's replacement snapshot. A reconnect +re-holds and gets snapshots; nothing is lost that was not in flight.
    • +
  • Leaving is never derived from an unmount. A plugin reload remounts components (chapter XV). Cancelling a login, +closing an overlay or giving up a lease is an explicit action or a dropped reference.
    • +
+

The TUI launcher keeps downloaded UI code in a permission-dropped worker with no access to files, network or +credentials; local side effects (copy, upload, download, open a URL, an external editor) need a real user gesture or a +confirmation. That is app code around Chord, not part of it.

+

Transport

+

The last domain, transport, is chapter XVII: one message format, per-part limits (16 MiB compressed and 64 MiB decoded +on client WebSockets, instead of p2's 1 MiB input cap and 32 pending calls), one byte window per link, re-holds with +snapshots after a reconnect, and a protocol version that makes an outdated web page reload.

+
Built so far

None of p3 exists. pi-ai's login API, pi-codemode and the pi-env daemon exist and are used by p2 today; the HTTP +component and channels are deferred until after the kernel's first real proof (chapter XIII).

+
+
+ +
XXI

Other Ways to Do It

Cordis and DeepSeek Harness attack the same problem, plugins that come and go while an agent runs, inside one process. What that buys, and what it costs.

+

Chord v2 is not the only answer to "replace plugin code while the app keeps running". The closest relative is +Cordis, a TypeScript plugin framework by Shigma that has been the plugin kernel of the Koishi chatbot framework +since 2019, and its most visible user, DeepSeek Harness (dsh), DeepSeek's open-source agent harness, whose +motto is "everything is a plugin". This chapter compares them with Chord v2 on the questions the guide cared about: +who owns side effects, how plugins depend on each other, what a reload keeps, what a crash keeps, and how far the model +reaches beyond one process.

+
Sources and status

DeepSeek Harness at commit 687f282 (release 0.2.1-rc.2), which vendors Cordis as @deepseek-ai/cordis 4.0.5 (Cordis +RC7 with local changes, vendor/README.md): its Cordis tutorial (docs/cordis-tutorial/), docs/architecture.md, +docs/subsystems/persistence.md, docs/subsystems/core.md and packages/boot/hmr/README.md; the Cordis paper "A +Programming Paradigm for Spatiotemporal Composability" (arXiv 2608.25512). Cordis and dsh are built and in use; Chord v2 +is the design this guide describes. The comparison is between a working system and a design.

+
+

Cordis in one example

+

A Cordis plugin is a function (or an object with apply, or a Service subclass) that receives a context. It may +declare the services it needs with inject, and everything it does to the world goes through the context:

+
export const name = "greeter";
+export const inject = ["tools"];                       // activate only while a `tools` service exists
+
+export function apply(ctx: Context) {
+  ctx.tools.register(defineTool({ name: "greet", /* … */ }));   // a registration is an effect: undone on unload
+  ctx.on("session/start", (s) => log(s.id));                     // an effect-bound listener
+  ctx.effect(() => {                                              // any other side effect, with its disposer
+    const timer = setInterval(work, 200);
+    return () => clearInterval(timer);
+  });
+}
+

Each plugin instance is a fiber with states PENDING → LOADING → ACTIVE → UNLOADING → DISPOSED (or FAILED). +Every effect, registration, listener, child plugin and provided service is recorded on the fiber, and unloading runs +the disposers. The paper calls this temporal composability: a component can be removed at any time and reverses its +own effects. Its counterpart, spatial composability, is the dependency side: inject is live, so a plugin starts +when its services appear, and unloads (and later reloads) when a service it depends on goes away or is replaced. The +paper's claim is that independently written components can therefore be combined and replaced without disturbing each +other. The next sections check how far that holds.

+

Services are named capabilities on the context (ctx.provide("x", value) or class X extends Service). Names are +runtime strings in one flat namespace per application; TypeScript declaration merging makes ctx.x typecheck, but +nothing checks the service's shape at runtime, and optional dependencies are probed with ctx.get("x"). Scopes come +from the context tree: ctx.extend() derives a child context, ctx.isolate(name) gives one subtree its own binding of +a service, so two parts of an app can use two different shell providers.

+

Hot reload follows from the two halves: the reload plugin unloads the old fiber (unwinding all its effects) and +loads the new module with the same config. Nothing held in the old instance's closures or fields survives; a plugin +that needs state across reloads must keep it in a service or on disk. Config edits are diffed by entry id and +mount, unmount or reconfigure only what changed; config fields declared volatile update in place without a remount.

+

DeepSeek Harness on Cordis

+

dsh applies "everything is a plugin" to an agent: the model adapters (ctx.llm), the tool registry (ctx.tools), the +session log (ctx.sessions), the agents (ctx.agents) and the agent loop itself (ctx.agentLoop) are plugins and +services. A running dsh is a plugin tree composed at boot from a profile (web, headless, sdk, acp) that +stacks bundles of Cordis config rows, with user patches on top. There is no privileged core: replacing the agent +loop is a config patch.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConcernHow dsh does it
sessionsa durable, append-only event log per session (JSONL backend, single writer, flush as the durability barrier); model history is derived from the events
crash mid-turnthe log keeps everything appended; on resume the agent loop closes the open turn by appending missing tool errors and a synthetic turn/end { reason: "interrupted" }
toolsctx.tools.register(defineTool(…)), an effect of the registering plugin; calls go through pre/execute/post policy waterfalls
approvalclosed by default; allowed-once is the only grant; plugin installation needs full access or per-call approval
reloada source watcher evicts changed modules; Loader entries running affected plugins stop and start again with the new code; a failed replacement is an activation failure, its previous code no longer runs
processesone plugin tree in the Host process; the browser client and Electron's renderer talk to it over RPC streams
distributiona Plugin Manager installs and enables bundles per profile; plugin markets are optional indexes
subagentslocal subagents with their own durable sessions, and external ones
+

What the guarantees are in practice

+

Reading the Cordis source vendored in dsh and dsh's own documentation, the guarantees are narrower than the paper's +framing. An earlier review for p2 came to the same result (apps/p2/docs/p3-reviews/dsh-compare.md).

+
    +
  • Reversal is cooperative and unordered. _unload runs all of a fiber's top-level disposers concurrently +(Promise.all in vendor/cordis/src/fiber.ts); order holds only inside one effect, and the primer tells authors to +"keep the related work in one effect" when teardown order matters. A disposer that throws is logged and skipped. A +disposer that never settles stalls the teardown; dsh's jobs service works around this by force-failing a job whose +cancel throws and warning that "the work may be orphaned" (packages/jobs/jobs-local/README.md). Reversal is as +good as every disposer, and nothing checks that a plugin registered all its effects.
    • +
  • Behaviour depends on registration order, and a reload changes it. emit, waterfall, serial and bail +listeners run in registration order (docs/cordis-primer.md). Registering appends the listener (push in +vendor/cordis/src/events.ts, unless prepend), so when a plugin reloads, its listeners move to the end of the +list. A tool-policy listener that ran first runs last after its own plugin reloads; a waterfall that short-circuits +now sees a different decision first.
    • +
  • Dependencies are strings, and every provider reload restarts its dependents. A fiber's activation key is the +concatenation of its injected providers' fiber ids (_refresh in fiber.ts), so replacing a provider unloads and +reloads every plugin that injects it, with their in-memory state. Service APIs are "pre-stable; update every +consumer" (AGENTS.md): there is no versioned contract between a provider and its consumers.
    • +
  • Reloading a core service ends live work. Unloading the agent provider "stops and drains every live handle it +made" (packages/core/agent/README.md), so running agents stop; reloading the session persistence backend under +live sessions invalidates their write handles, and "writes fail loudly until the sessions restart" +(.agents/notes/implemented/architecture/2026-08-27-handle-based-session-persistence.md).
    • +
  • Registrations leave a gap. A reloading plugin's tools and other registry entries are removed with its old fiber +and added again by the new one; anything that looks them up in between does not find them.
    • +
+

None of this is a bug in the narrow sense: each point is documented, and dsh is built around it. But "components +interleave without disturbing each other" holds only for plugins that own nothing other plugins depend on, register +every effect, order nothing by registration, and hold no state that must outlive a reload.

+

Side by side

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Cordis and dshChord v2
unita plugin instance (fiber) in one context treean object per (type, input, principal), placed on a host (chapters II, III)
side effectsrecorded on the fiber with ctx.effect and friends; on unload the disposers run concurrently, each as good as its author made itdeclared through self (documents, natives, routines, jobs, contributions); Chord activates and undoes them (chapters V, IX to XI)
dependenciesservice names as strings in one flat namespace, via inject; a provider reload restarts every dependenttyped contracts via get(Type, input); a reference parks and recovers through restarts (chapter VI); optional dependencies are registries (chapter XIII)
orderinglisteners run in registration order, which changes when a plugin reloadsregistries compose with a declared compose function; nothing depends on load order (chapter XIII)
where code runsone process; remote UIs over RPC to the Hosta tree of hosts (server, workers, tabs); objects reachable across links, relayed by the server (chapters I, XVII)
who is callingnot a framework conceptprincipals stamped by links, variants per caller, revocation (chapters III, VII)
statethe plugin's own business; dsh's session log is a servicedocuments written in transactions, persisted, versioned and migrated (chapter V)
reloadunload the old instance, load the new one: in-memory state and running work of that plugin endpure objects swap their code in place with their documents and running work kept; only objects with natives restart (chapter XV)
a running agent turn during reloadentries of changed plugins stop; unloading the agent provider stops its live agentsthe harness is base code; Durable pins tools per task phase, so a running call finishes on old code and the next phase uses the new (chapter XIV)
crash recovery of a turnthe turn is closed as interruptedpi-durable continues the turn's tasks from their checkpoints; replay-safe tools rerun (chapter XIV)
contracts and versionsTypeScript types only, pre-stable APIs, every consumer updated together; config validated with a schemacontract versions with fingerprints and provider-side mappings for old consumers (chapter XVI)
statusin production in Koishi since 2019; dsh releaseda design with the contract layer built
+

What the comparison says

+

The core idea is the same. "Factories declare, Chord activates" (chapter II) and Cordis' reversible effects are +two forms of one rule: a plugin may only change the world through its runtime, so the runtime knows what to undo. +Both reject the module-level timer and the stored callback for the same reason.

+

The difference is what survives a reload. Cordis reloads at plugin granularity: the old instance unwinds and the +new one starts from nothing, together with every plugin that depends on it. That is simple, but a plugin's in-memory +state and its dependents' state and work in flight end with it, and correctness rests on every disposer and on +registration order. Chord v2 keeps state out of closures on purpose (documents, references, jobs as +records) so that it can swap the code of a pure object while its data and running work stay, and restarts only what +holds OS handles. That is more machinery: transactions, a store, versions on every document and contract.

+

The other difference is reach. Cordis composes plugins inside one process, and dsh adds RPC to its own clients. +Chord v2 makes the process boundary part of the model: objects live on the host their placement names, callers are +principals, and a reload rolls out across the server, the workers and every tab. An app like p2, with a process per +session and guests in other browsers, needs that; a single-process agent does not.

+
Review: what a reload does to a running tool call in dsh

The sources say that Loader entries running affected plugins stop and that unloading the agent provider stops its live +agents (packages/boot/hmr/README.md, docs/subsystems/core.md). They do not say what happens to a tool call in +flight when only the tool's plugin reloads; the comparison above does not claim either way.

+
+
+ +
∴

What's Next

Where Chord v2 stands, what is still undecided, and where the written sources disagree with each other.

+

The guide built Studio from one object to a p2-sized app. Every piece used the same few rules: one owner per thing, +factories that only declare, state that changes only in transactions, long-lived behaviour in declared forms, and +mappings and migrations owned by the provider. Those rules are what let Mario press Reload on his phone in the +preface while the agent keeps running. This last chapter says how much of that exists and what is still open.

+

Status

+
~10klines in the frozen first kernel (steps 1 to 6, 443 tests), kept as a reference only
1 of 11layers of the rebuild started: L1 contracts, built and reviewed
6–7ktarget size of Chord v2 without tests
+
    +
  • The design is decided except the points marked below. packages/chord/docs/next/design.md is the target; +packages/chord/docs/next/decisions.md is the log and wins where they differ.
    • +
  • The rebuild goes layer by layer, each with a one-page spec the owner approves, a minimal build with counting and +garbage-collection tests, the cost-guard metrics, and a cleanup loop: L1 data and contracts; L2 readable and +derive; L3 documents and store; L4 objects; L5 references and ctx; L6 protocol and links; L7 relay, routes, +revocation; L8 routines and jobs; L9 registries; L10 reload and rollout; L11 telemetry in Chord.
    • +
  • Built: the contract layer (src/next/contracts/, commit 7d2bb8ea8, chapter XVI). In pi: Delta, +pi-telemetry, chord/context, chord/telemetry-sqlite, pi-ai and Durable telemetry, Durable's required +checkpointWhen.
    • +
  • Next, in order (apps/p2/docs/p3-handoff.md): decide Q12 and Q15, which the L1 spec needs; write the spec for +the rest of L1 (values, inputs and identity, argument checks, principals, errors; draft in +packages/chord/docs/next/layer-1.md); then Q11, Q13, Q14 and the cuts C3 to C17 before the layers they affect.
    • +
  • In pi: mapShared and bytes as atomic leaves in Delta's diffRevisions.
    • +
  • Then the playground web shell (React, esbuild) with the reload control and a preview at p3.mariozechner.at, +and the first real proof: the harness, one tool with an approval pause, a code mode reload. Then the HTTP and +channel components, then p3, domain by domain.
    • +
+

Deferred decisions

+

Kept with stable numbers in design.md §26, each waiting for a concrete case:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#TopicWhenChapter
9jobs as objects: a record inside the owner, or an object addressed by its idwhen login is portedXI, XIX
11where a flow lives: a keyed document of its owner, or an object LoginFlow { id }when login is portedXII
14project plugins: per-directory plugins with trust prompts and snapshotsown pass over plugin-loading.mdXV
16a Durable Object profilewhen a DO target becomes concrete
17session files: one SessionFiles { session } owner on the server (proposed)p3 designXX
18session delete: retire by tag plus a session directory, or owner hooksp3 designXIX
20connection pools: one owner of SSH connections per environment, or per-use connectionsp3 designXX
21config files with a single writer: one owner on the server, others followp3 designXVIII
22file formats (envs.json, downloads, uploads) as versioned interfacesp3 designXX
23the terminal host process: inside the revision or deliberately outsidep3 designXX
+

Open from review round 3

+

Decisions (apps/p2/docs/p3-reviews/v2-review3-summary.md; Q1 to Q10 are decided):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#ProblemProposal
Q11a singleton that does not exist yet: undefined or initial()?singletons read initial(); undefined only for absent keyed documents
Q12stripping arguments (R7) against "nothing is copied"copy on change: a new object only when the argument had extra fields; never mutate the caller's value (proposed in the L1 draft)
Q13ctx.incarnation on relayed callsrelays stamp the origin host's incarnation, as they stamp the principal
Q14a writable holding functions exposed as a state memberonly members with a JSON schema in the contract are exposable, checked at serve
Q15arguments of local methods: validated or notapply R7 to local too, or keep local as the one exception
+

Cuts the owner may choose (each removes a primitive this guide teaches):

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#CutWhat it costs
C3shrink jobs to a runner over documents (no versions, accepts, blocked, persist), or move jobs out of the kerneljob versioning, or self.job for core-only users
C4routines: on and loop onlyevery's due time across reloads
C5merge lifetime: "host" objects into the faceta plugin's always-on natives share one failure domain
C6remove broken() for opens and retryOnnothing; reload and Retry still work
C7remove provider-side lingerten lines of app code
C8remove state: false (method-only references)a method-only consumer pays one snapshot
C9remove previous from derivesderives that used it move to mapShared
C10one cell(initial) for constant and writablenothing
C11remove ctx.trace (use traceIn(ctx, …))nothing
C12remove call credits (the byte window covers calls)nothing
C13two timeouts (closeTimeout, readyTimeout) instead of eightper-thing overrides
C14remove the handle.change and handle.retire shorthandsnothing
C15remove inVariant and expectVariant (keep RefIn)nothing
C16remove facet-owned mutable documents and their transaction exceptionatomic writes across an object and its facet move to one owner
C17no harness-specific start-barrier exceptionfactories before the barrier cannot depend on the harness
+

Where the sources disagree

+

This guide follows the decision log where it is newer than the design, and the design where it is newer than +everything else. The places that need a fix in the sources:

+
Review: drift in the written sources
    +
  • design.md §26 lists only the deferred decisions; it omits the open review-3 items Q11 to Q15 and the cuts C3 to +C17.
    • +
  • design.md §3 has a garbled sentence, "Values are method arguments are stripped".
    • +
  • apps/p2/docs/p3-handoff.md §3 is stale: it says all R items are decided and, two lines later, that R2 to R5 and +R7 to R26 are open, and "resume R6 next".
    • +
  • kernel.md and step-*.md describe the frozen code: host values, a development freeze, consumer-side linger, +JSONL storage, descriptions on the wire, and a link closed on an unknown id. None of these is the target.
    • +
  • chord-next-walkthrough.md lags the log: validation by trust (now R7, every caller), dev mode and freeze (removed +by R9), descriptions once per link (gone with R6), an IPC frame-count window (now one byte window, Q4), job outcome +{ ok } (now { kind }), harness activation as the start barrier (R17), registries "newer refused at load" (Q8), +and hooks using host.root (now chord(fn, context), Q6).
    • +
  • apps/p2/docs/questions.md uses self.resource and replace (both rejected) and host.root in hooks.
    • +
  • apps/p2/docs/p3-reviews/v2-cost-model.md: the measurements are current, its "Decisions this needs" section is +not (chapter XVII).
    • +
  • L1's Implementation<C> shape differs from the factory return value used in the design (chapter XVI).
    • +
+
+
+ +
A

API Reference

Every name the guide uses, grouped by where you meet it. Working names; only the contract layer is built.

+

Hosts

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CallMeaningChapter
createHost<P, R>({ address?, principal?, storage?, principalKey?, revokedBy?, telemetry?, onError?, timeouts })a host; with an address it is addressable and its own code acts as principal; without, a clientI
host.load([{ name, facet }])load facets by plugin name; base code passes its core facet the same wayI
host.connect(link, { principal })the link to the parent (at most one)I
host.accept(link, { principal })a link from a client; every request on it acts as principalI
host.route(Kind, { authorize, resolve })how to reach hosts of a kind; resolve returns { link, principal }I
host.revoke(r)end matching objects and links here and belowVII
host.root(fn, { principal?, signal?, telemetry? })a call scope with a ctx for code outside objectsVI
host.get(Type, input, { signal, state?, parkTimeout? }), host.hold(…)a reference from outside objects; release() ends itVI
host.serve(Type, factory, { lifetime?, linger? })serve an object type (in a facet)II
host.registry(Registry)the composed value of a registry, a readableXIII
host.references(Type)who holds references to objects of a type on this host (presence)VIII
host.activity{ references, calls, jobs, lingering }, for keep-aliveI
host.close()end the host: objects, natives, linksI
+

Inside a factory: self

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MemberMeaningChapter
self.state(Definition)a document handleV
self.tx(async (tx) => …, ctx)a transaction on the object's lineV
self.derive(name, [sources], fn)a derived readableVIII
self.get(Type, input)a reference held for the object's lifeVI
self.open(open, close, { closeTimeout? })a native; .value once openIX
self.routines.on(readable, fn), .every(interval, fn), .loop(fn), .custom(…)standing behaviourX
self.job(Job, { run, finish? })owned work that outlives its callXI
self.contribute(Registry, value)a contribution (facets and lifetime: "host" objects only)XIII
self.writable(initial)a host-local readable of any in-process value, with setXIII
self.retryOn(readable)a trigger that retries a broken activationVI
+

Document handles: value, get(key) (keyed), keys, change(fn, ctx) / change(key, fn, ctx) and retire(ctx) / retire(key, ctx) (shorthands for +one-document transactions). Transactions: tx.draft(handle, key?, { tags? }), tx.retire(handle, key?), +tx.start(job, args, { key? }). Jobs: job.start(args, ctx, { key? }), job.abort(id), job.running, +run.outcome(ctx).

+

References and ctx

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MemberMeaning
ref.<member>a state member (readable: .value, subscribe) or a method
ref.state, ref.onState(fn)loading, live, failed (cause, retryAt), released
ref.retry()retry a failed reference now
ref.release()end a reference taken outside objects
ref.membersthe variant's members
ctx.signal, ctx.principal, ctx.incarnation, ctx.getcancellation, caller, calling host's incarnation, call-scoped references
ctx.telemetry, ctx.trace(name, attributes, fn)the current span; a child span with a child ctx
ctx.jobId, ctx.progress(value)in a job's run
+

Definitions

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NameWhat it declaresChapter
hostKind(name, input?)a kind of host and its addressI
remote(name, { version, input, placement, variants })an object type reachable across linksII
local(name, { version, input, members })an object type on its own hostIV
method(args?, result?)a method in a base declarationII
evolve(prev, (e) => …), checkAccepts(contract, accepts)the next contract version; the provider's mappings (built)XVI
Implementation<typeof C>the type of a provider for the current version (built)XVI
str num int bool nul lit oneOf union obj arr map opt bytes unknown, Staticschema helpers (built)II
defineState(name, { schema, version, initial, checkpointWhen, persist?, keyed?, ordered?, migrate? })a documentV
defineTag<T>(name)a tag; store.retireByTag(tag, value)V
defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })a job typeXI
defineRegistry(name, { version, compose, accepts? })an extension pointXIII
bindFacet<P>()base code's defineFacet, with factories' principals typed as the app's P (working name)III
defineFacet((host, self) => …)a plugin's setup for one kind of host; imported from base code (@studio/core)II, III
constant(value)a readable that never changes (start data)IX
broken(error)a failure that is not retried by the backoffVI
inVariant(ref, …), expectVariant(ref, …), RefIn<…>narrowing a reference by variantIII
mapShared(fn)an array map that keeps unchanged items identical (planned in Delta)VIII
OutputBuffer, Progressbounded output and throttled progress (@earendil-works/chord/output)XI
memoryLinkPair(), ipcLink(child), webSocketLink(ws)link adaptersI
DocumentStorage, registerDocumentStorageConformancethe storage port and its shared test suiteV
+
+ +
B

Timeouts and Defaults

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Timeout or defaultBoundsDefaultOverride
closeTimeouta native's close, then reported as a possible leak30 sper native
call drainan ending object's own call handlers= closeTimeoutper type
readyTimeoutactivation; an awaited get60 sper get
park deadlineparked calls (link drop, loading, rollout)60 sper get (parkTimeout); per type
routine handovera new routine waiting for the old handler on reload10 sper routine
finish retrybackoff of a throwing job finish1 s to 60 sper job
rollout installone host installing a revision60 snone
lingeran object kept alive after its last referenceper typeper type
backoffretries of transient failures, per address1 s doubling to 60 snone
evictionkeyed persisted documents idle in memory10 minutes
client WebSocket limitsper part16 MiB compressed, 64 MiB decodedper link kind
compressionJSON parts compressed at their originabove about 16 KiB
worker keep-alive (p2, app code)idle time before the supervisor stops a worker5 minutesapp
+
+ +
C

Versions

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SurfaceVersionOn changeChapter
object typeversion plus fingerprintevolve and checkAccepts; otherwise contract_mismatchXVI
documentversion plus schema fingerprintpure migrate on access; no bump fails brokenV
jobversionaccepts on outcomes and running; blocked withoutXI
registryversionaccepts on contributions; unmatched ones left out and reportedXIII
nativenonerestarts on reloadIX
Durable tasks and documentsDurable's versionDurable's migrateXIV
link protocolprotocol versionrefused; hosts update togetherXVII
UI components, Chord and app APIsnone yetexposed as they are
+
+ +
D

Errors and States

+

ChordError has a stable code and a finer reason; on the wire it is { code, reason?, message, details? } +(specified in layer-1.md, not built).

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CodeMeaning
invalidinput, arguments or placement rejected (invalid/input, invalid/args, invalid/placement)
contract_mismatchthe consumer's version and fingerprint are neither current nor accepted
forbiddenthe caller may not do this
gonethe object or hold ended
not_readynot activated in time
releasedthe reference was released
too_largea part exceeded its size limit
unavailablecould not be reached before the deadline (unavailable/locked and others)
abortedcancelled by a signal
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeValues
FailureCause{ kind: "transient"; error } (retried with backoff) or { kind: "broken"; error } (retried only by a trigger)
reference stateloading, live, failed (cause, retryAt), released
job outcome{ kind: "done"; result } or { kind: "failed"; error: "threw" | "aborted" | "lost" }
job phaserunning, finishing, blocked
rollout status of a providerupdating (calls park), rejected (fail at once), same revision (contract_mismatch)
host install reportinstalled, installed_failed, rejected
+
+ +
E

Rejected Designs

+

From design.md §31, grouped. Each was considered and is not coming back without a new reason.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AreaRejected
reloaddrain-based handover; object restart for state changes; a factory trial; a migrate dry run; a consumer contract check during staging; reload votes; per-plugin rollback within a host; keeping natives across reloads by name; restart_required; host values and a development freeze
stateversion fences without transactions; volatile documents; per-group lines; optimistic transactions; tx.get; a JSONL document storage; validating state and documents on commits or reads; copying values on same-host calls; op-rewriting projections and a view algebra; Durable's Session as the object store
objects and referencesresources replaced in place (replace, deps, stale handles); revoking proxies on .value; consumer-side linger; detached references counting toward presence; an optional flag on references; opaque members; callbacks as method arguments; generic call or request ids; a stream type in Chord
registriesregistries created at runtime; entries resolved through mutable indirection; generic registry keys with a conflict flag; numeric middleware priorities; registry composition as a staging check
principals and placementprincipal field patterns and re-stamping for revocation; storeKey; placement as data naming an input field; host-name strings in app code; "magic" input-field matching for delete
wirean outbox for transient delivery errors; cross-relay reset coordination; count limits that refuse calls or holds; contracts taken from the running app
process and toolinga topology layer above the host API; loader variants A and B; Chord ending the process on a storage failure; defaultProjectTrust: "always"; ceremony (native/ directories, defineNative, lints); weakening type checking
+
+ +
F

Glossary

+

Adapted binding: a consumer bound at an older accepted contract version, served through the provider's mappings. +Base code: the app's own code, imported statically; a change restarts the hosts. Client: a tab or TUI host, +not addressable. Contract: an object type's declaration (name, input, placement, members, version). +Contribution: a value a plugin adds to a registry. ctx: what every call receives. Derive: a value +computed from readables. Document: a named JSON value an object owns. Facet: a plugin's code for one kind of +host. Factory: the synchronous function that declares an object. Fingerprint: a hash of a contract's or +schema's normal form. Flow: input into running work, kept as state by its owner. Host: one participant of the +app, with an identity and an incarnation. Incarnation: the id a host gets on every start. Input: an object's +key among objects of its type; with the type and the caller's principal key it identifies the object. Job: owned work that outlives its call. Link: an ordered connection between +two hosts. Local type: an object type reachable only on its own host, without principals. Native: an OS-level +value opened with self.open. Park: a call waiting for a reference to become live again. Placement: the +function from input to host address. Plugin: a directory with a package.json, reloadable. Principal: who +calls, stamped by the link. Principal key: the string that identifies a principal in object identities. Pure +object: an object without natives; swapped in place on reload. Readable: a value plus its ordered changes. +Reference: what get returns. Registry: an owner's extension point. Relay: a host forwarding messages +between two links, rewriting only headers. Reload: swapping a plugin's code in running hosts. Restart: ending a +process. Revision: the complete code set the hosts should run. Revocation: ending live access for principals +matched by revokedBy. Routine: standing behaviour of an object. Server: the addressable root host. Shared +store: a local object that holds data once behind per-principal views. Tag: a typed label on documents, used to +retire them together. Transaction: an atomic set of document writes on the object's line. Variant: the set of +members a principal gets. View: a per-principal remote object in front of a shared store. Worker: a host the +server starts for one session.

+
+ +
G

Sources

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SourceUsed forWeight
packages/chord/docs/next/decisions.mdevery decision, review rounds 2 and 3wins where newer
packages/chord/docs/next/design.mdthe target design, all chaptersauthoritative otherwise
packages/chord/src/next/contracts/ and its README.mdchapter XVIbuilt code
packages/chord/docs/next/layer-1.mderrors, inputs, argument checksdraft spec
apps/p2/docs/p3-reviews/v2-review3-summary.mdopen Q11 to Q15, cuts C3 to C17current
apps/p2/docs/p3-reviews/v2-cost-model.mdmeasurements, per-hop costscurrent except its last section
apps/p2/docs/p3-functions/chapters XVIII to XXcurrent
apps/p2/docs/p3-handoff.mdstatus and order of work§3 partly stale
packages/durable/README.mdthe harness, submit, tasks, subagentscurrent
chord-next-walkthrough.md, kernel.md, step-*.md, apps/p2/docs/questions.mdbackground onlylag the decisions (see What's Next)
DeepSeek Harness at 687f282 (its Cordis tutorial, architecture, persistence, core and HMR docs), the Cordis paper (arXiv 2608.25512)"Other Ways to Do It"external, built systems
+
+ +

Chord v2 · a guide · first edition · 2026-10-20 · based on design.md (target design v2), redesign-decisions.md (review rounds 2 and 3), chord-next-walkthrough.md, layer-1.md, contracts/README.md, the p3 functionality map, questions.md and the pi-durable README

+
+ + diff --git a/packages/chord/docs/next/explainer/src/00-front.md b/packages/chord/docs/next/explainer/src/00-front.md new file mode 100644 index 000000000..36d62e8d0 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/00-front.md @@ -0,0 +1,111 @@ +--- +title: Chord v2: a guide to building apps that reload +cover: Chord v2 / A Guide +kicker: A guide · Chord v2 target design · 2026-10-20 +subtitle: How to build an app like p2 from nothing on Chord v2 and pi-durable, one object at a time, so that new plugin code takes over while the agent keeps running +stats: + - 11 | layers in the rebuild, L1 to L11 + - ~1,300 | lines built so far (L1 contracts) + - 6–7k | target size of Chord v2, without tests + - 0.004 ms | Delta's apply per small change +note: | + This guide teaches the **designed** API of Chord v2 (`packages/chord/src/next`, export `@earendil-works/chord/next`) + from `packages/chord/docs/next/design.md` and the newer decision log `packages/chord/docs/next/decisions.md`, which wins + where the two differ. Most of it is **not built yet**: a first kernel of about 10k lines is frozen as a reference and + Chord v2 is being rebuilt layer by layer; only the L1 contract layer exists (`src/next/contracts/`, commit + `7d2bb8ea8`). Code in this guide is therefore the target, not something you can run today. Every chapter says what is + built. API names are working names. Notes marked *review* are points that are undecided, or where the sources + disagree; they name the files. The reader is assumed to know pi-durable; Chord is explained from the start. +colophon: Chord v2 · a guide · first edition · 2026-10-20 · based on design.md (target design v2), redesign-decisions.md (review rounds 2 and 3), chord-next-walkthrough.md, layer-1.md, contracts/README.md, the p3 functionality map, questions.md and the pi-durable README +--- + +# Before we begin {.preface} + +It is late in the evening. Mario is on his phone, connected to his laptop at home through the Radius relay. In session +S1 the agent is twenty minutes into a refactoring; right now it is running a test suite through `bash`. A colleague, +Ann, follows the session read-only from her own browser. Mario notices that the notes panel sorts notes the wrong way +round. He fixes one line in the notes plugin on the laptop and presses **Reload**. + +What should happen is easy to say. The notes panel on both screens shows the new order. The test run keeps running. The +agent's turn continues as if nothing happened. Ann's view does not flicker into an error. Nothing that was written is +lost, and no process restarts. + +The obvious ways to get there fail. **Restarting the processes** loses the turn in flight, the open terminal and every +connection; pi-durable would resume the turn, but the test run and the connections are gone. **Hot-swapping modules** +in place, as front-end tooling does, leaves the old code alive in closures, timers, event listeners and sockets that +nobody knows about: the old sort function keeps running inside a callback someone stored, a file watcher opened by the +old code is never closed, and two versions of the plugin now write the same data without any order. + +Chord v2 is the runtime that makes the first outcome normal. It does so by being strict about one thing: **every piece +of state, every open OS handle and every piece of running work has exactly one owner, an object, and objects only +declare what they need.** Chord itself activates, connects, reloads and ends them. Because Chord knows every timer, +watcher, transaction and reference an object has, it can swap the object's code while its data stays, restart only +what touches the operating system, and route calls between browser tabs, the server and per-session worker processes +without the plugin author writing any of that. + +This guide teaches Chord v2 by building an app called **Studio**: a small version of p2 (Pi Mobile), with a server, +one worker process per session, browser tabs as clients, and an agent from pi-durable on each worker. The first +chapters grow one plugin, `@studio/notes`, until it uses every concept; the last three chapters assemble the rest of a +p2-like app from the same parts. + +:::review | Review: what exists today +Nothing in this guide runs yet except the contract layer. `packages/chord/src/next/contracts/` (L1 part 1: `remote`, +`local`, `method`, `evolve`, `checkAccepts`, the schema helpers) is built and reviewed. The rest of L1 (inputs, +argument checks, principals, errors) is a draft spec (`packages/chord/docs/next/layer-1.md`). The frozen kernel in +`src/next/*.ts` implements an older version of the design (host values, a development freeze, JSONL storage, +descriptions on the wire); do not read it as the target. Build on the design, not on the frozen code. +::: + +## Ten things to know + +:::findings +1. **An app is a tree of hosts.** One server, any number of workers it starts on demand, any number of clients (tabs, + TUIs) that connect to it. Clients reach workers only through the server, which relays. +2. **Objects are the unit of everything.** An object type is a contract (name, input, placement, members, version); an + object is one instance per input and caller. The first `get` creates it, later gets share it, the last release ends + it. +3. **Factories declare, Chord activates.** A factory is synchronous: it declares documents, references, derived + values, OS handles, routines and jobs through `self`, and returns the members. Chord opens, binds and starts them, + and undoes them in reverse. +4. **Links stamp who is calling.** A caller's data never says who it is; the link it arrived on does. Each caller gets + its own object and the variant of members it may use. +5. **Shared data lives once, behind views.** A `local` shared store holds the data; per-caller `remote` views expose + what each caller may see. Only stable identities persist documents. +6. **State changes only in transactions.** Documents are JSON values written in `self.tx`; commits are atomic for + every viewer and travel as small operations, never as whole values. +7. **Failures look like disconnects.** A reference survives link drops, restarts and failures: calls park or fail, + values stay readable, and Chord retries by itself. There are no request ids. +8. **Everything long-lived has a declared form.** Standing behaviour is a routine, work that outlives a call is a job, + input from a person is a flow, behaviour from other plugins comes through a registry. Nothing else may hold a timer + or a callback. +9. **Reload swaps pure objects in place and restarts native ones.** Objects without OS handles get the new code with + their data and running work intact; objects with OS handles close and reopen them. A broken change leaves the whole + host on its old code. +10. **pi-durable runs inside a core object.** The agent harness is a host-lifetime object on each worker; models, + environments and tools reach it through registries, and tools reach Chord through `chord(fn, context)`. +::: + +## Words used throughout + +A **host** is one participant of the app: a process, a browser tab or a Durable Object, identified by a logical +identity and an **incarnation** that changes on every start. A **link** connects two hosts. The **server** is the one +addressable root host; a **worker** is a host the server starts for one session; a **client** is a tab or TUI, not +addressable. A **plugin** is a directory with a `package.json`; its code for one kind of host is a **facet**. **Base +code** is the app's own code (Chord calls its facet the core facet); it restarts the hosts when it changes, while +plugins reload. An **object type** (or **contract**) describes objects; an **object** is one instance; its **input** is +its key among objects of that type, and its **placement** computes from the input on which host it lives. A **factory** builds an +object; **`self`** is what a factory declares through. A **principal** is who calls, stamped by the link; a +**variant** is the set of members a principal gets. A **reference** is what `get` returns. A **readable** is a value +plus its ordered changes. A **document** is a named JSON value an object owns, written in a **transaction**. A +**derive** is a value computed from readables. A **native** is an OS-level value (a socket, a watcher, a process) +opened with `self.open`; an object with natives is a **native object**, one without is a **pure object**. A +**routine** is standing behaviour, a **job** is owned work that outlives its call, a **flow** is input into running +work. A **registry** is an owner's extension point, filled with **contributions**. **`ctx`** is what every call +receives: cancellation, caller, call-scoped references. A **reload** swaps a plugin's code inside running hosts; a +**restart** ends a process; a **revision** names the complete code set the hosts run. + +:::note | Reading the code in this guide +Imports are shown once per file and then omitted. `obj`, `str`, `arr` and friends are Chord's schema helpers (they +return ordinary TypeBox schemas). Helper functions that belong to the app (`authenticate`, `watchDir`, `uuidv7`) are +named and not shown. Where the design gives no name, the guide says it uses a working name. +::: diff --git a/packages/chord/docs/next/explainer/src/01-shape.md b/packages/chord/docs/next/explainer/src/01-shape.md new file mode 100644 index 000000000..db18b5cfe --- /dev/null +++ b/packages/chord/docs/next/explainer/src/01-shape.md @@ -0,0 +1,246 @@ + +# The Shape of an App {#shape lede="Which processes a Chord app has, how they find each other, and the start code that connects them."} + +Before any plugin exists, Studio needs processes and connections between them. In Chord each participant is a host, +and the app's own start code creates every host, connects them and decides how workers start. Chord deliberately does +none of that itself: a laptop app forks child processes, a test runs everything in one process, a Cloudflare +deployment routes through the edge. This chapter writes Studio's start code: `server.ts`, `worker.ts`, the browser +entry, and a test setup. + +## Three roles + +A Node app on Chord has one shape: + +:::flow wide | The three roles. Clients connect to the one server; the server starts workers and relays between them. +row "Clients: not addressable" muted +box tab1 "browser tab" "owner, local" +box tab2 "browser tab" "guest Ann" +box tui "TUI" "owner" +row "Server: exactly one" accent +gap +box server "server" "`Server()`" accent +gap +row "Workers: one per session, on demand" blue +box w1 "worker" "`Session({ id: \"S1\" })`" blue +box w2 "worker" "`Session({ id: \"S2\" })`" blue +box w3 "worker" "`Session({ id: \"S3\" })`" dashed +arrow tab1 -> server "WebSocket" +arrow tab2 -> server "WebSocket" +arrow tui -> server "WebSocket" +arrow server -> w1 "IPC" +arrow server -> w2 "IPC" +note "A worker for S3 does not exist until something asks for an object that lives there." +::: + +| Role | Addressable | Who it is comes from | Lifecycle | +| --- | --- | --- | --- | +| client | no | the authentication of its link | connects by itself | +| server | yes, exactly one | its own declaration | runs | +| worker | yes, many, by a typed address | stamped by the server, its parent | started on demand by the app | + +The **server** holds what all sessions share (the session list, access, settings) and starts workers. A **worker** +runs one session: its agent, its files, its terminal connections. **Clients** are browser tabs and TUIs. Nobody can +send a request *to* a client, so a client can only serve objects that nobody outside it needs; chapter IV names them +`local` types. + +Things that look like hosts but are not: a code mode sandbox (a worker thread inside a worker, owned by an object), a +remote environment's daemon (it speaks its own protocol, owned by an object), the TUI's sandbox behind its launcher. +A detached process such as a terminal host could become another kind of child of the server. + +## Host kinds and addresses + +Every addressable host has a typed address. The app declares its **host kinds** once, in base code: + +```ts +// studio/src/hosts.ts +import { hostKind, obj, str } from "@earendil-works/chord/next"; + +export const Server = hostKind("server"); +export const Session = hostKind("session", obj({ id: str() })); + +Server(); // the address of the server +Session({ id: "S1" }); // the address of the worker of session S1 +``` + +There are no host-name strings anywhere in app code. Chord serializes addresses internally; your code only calls the +kinds. An object type names its host with a placement function from its input to an address (chapter II), so the +type checker knows that `Session({ id: session })` needs a `session` field. + +## Links + +A **link** is an ordered, two-way channel of messages between two hosts; it only delivers. Chord ships three: a memory +pair for tests, IPC for a Node child process forked with `serialization: "advanced"`, and WebSocket. All three carry +the same binary message format and the same flow control (chapter XVII). Recovery policy belongs to the link adapter: +IPC needs no heartbeat, a WebSocket through a relay needs one, because TCP does not notice that a laptop went to sleep. + +Each link has a **direction**. A host has at most one link to its parent (`connect`) and any number of links from +below (`accept` for clients, `route` for workers). Requests arriving on a link act as the principal the link was given; +chapter III explains why that matters. + +## The server + +```ts +// studio/src/server.ts +import { createHost, ipcLink, webSocketLink } from "@earendil-works/chord/next"; +import { fork } from "node:child_process"; +import { WebSocketServer } from "ws"; +import { Server, Session } from "./hosts.ts"; +import { coreServerFacet } from "./core/server.ts"; +import type { P } from "./principal.ts"; + +const server = createHost

({ + address: Server(), + principal: { kind: "server" }, // who this host's own code acts as + storage: await openServerStorage(dataDir), // persisted documents (chapter V); memory if omitted + onError: (e) => { if (e.kind === "storage_failed") process.exit(75); else log(e); }, +}); + +await server.load([ + { name: "@studio/core", facet: coreServerFacet(config) }, // base code, under a fixed name + ...(await loadPlugins("server")), // [{ name, facet }] from the plugin loader +]); + +const wss = new WebSocketServer({ server: http, path: "/chord", maxPayload: 64 * 1024 * 1024 }); +wss.on("connection", (ws, req) => { + const principal = authenticate(req); // app code: cookie, loopback, device token + if (!principal) return ws.close(4401); + server.accept(webSocketLink(ws), { principal }); +}); + +server.route(Session, { + authorize: (p, { id }) => + p.kind === "owner" || p.kind === "server" || (p.kind === "guest" && p.session === id), + resolve: async ({ id }, { signal }) => { + const principal: P = { kind: "worker", session: id }; + const child = fork("worker.ts", [], { serialization: "advanced", signal }); + child.send({ id, principal, init: await initFor(id, signal) }); // the app's own start message + return { link: ipcLink(child), principal }; // Chord stamps `principal` on it + }, +}); +``` + +Each call has one job: + +| Call | Meaning | +| --- | --- | +| `createHost

({ address?, principal?, ...options })` | with an address: an addressable host whose own code acts as `principal`; without: a client. `P` is the app's principal type | +| `load([{ name, facet }])` | the facets this host runs, each with its plugin's `package.json` name; duplicates and invalid names are refused. Base code passes its core facet the same way | +| `accept(link, { principal })` | a link from a client; every request on it acts as `principal` | +| `route(Kind, { authorize, resolve })` | how to reach hosts of a kind. `authorize(principal, address)` runs before anything starts; `resolve(address, { signal })` returns `{ link, principal }` | +| `connect(link, { principal })` | the link to the parent, at most one, replaced after a reconnect; `principal` says who the parent is | +| `root(fn, { principal?, signal? })` | a call scope for code outside objects (a CLI command): gives `fn` a `ctx` (chapter VI) | + +Chord has no `listen`. Entry points differ too much: a loopback WebSocket server, a tunnel, the Radius relay +delivering channels, a Durable Object edge, a memory link in a test. Each entry point authenticates and calls +`accept`. Radius, for example, is one more line: + +```ts +radius.onChannel((ws) => + server.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } })); +``` + +## The worker + +```ts +// studio/src/worker.ts +const { id, principal, init } = await startMessage(process); // app code: the first IPC message +const worker = createHost

({ + address: Session({ id }), + principal, // { kind: "worker", session: id } + storage: await openSessionStorage(init.dataDir, id), // one storage per session +}); +await worker.load([ + { name: "@studio/core", facet: coreWorkerFacet(init) }, // serves the start data (chapter IX) + ...(await loadPlugins("worker")), +]); +worker.connect(ipcLink(process), { principal: { kind: "server" } }); +``` + +## A browser tab + +```ts +// studio/web/main.tsx +const client = createHost

(); // no address: a client +await client.load(await loadPlugins("web")); +client.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }); +const notes = await client.get(Notes, { session: "S1" }); // tab → server → worker S1 +``` + +## The first `get`, step by step + +The tab asks for the notes of S1. No worker for S1 runs yet. + +:::sequence animate | The first get for an object on a worker that is not running. The server authorizes, the app starts the worker, Chord stamps the link. +actor tab "browser tab" "owner" +actor server "server" accent +actor app "route(Session)" "app code" muted +actor worker "worker S1" blue +tab -> tab: placement: Session({ id: "S1" }) +tab -> server: hold Notes { session: "S1" } +server -> server: not mine: route(Session) +server -> app: authorize(owner, { id: "S1" }) +app --> server: true +server => app: resolve({ id: "S1" }) +app -> worker: fork + start message +worker -> worker: createHost, load, connect +app --> server: { link, principal: worker S1 } +server => worker: hold (stamped: owner) +worker -> worker: placement check, run factory +worker --> server: bound: snapshot +server --> tab: bound: snapshot +::: + +`resolve` runs once per address while its link lives. When the link drops (the worker crashed or the supervisor +stopped it) and references to that worker still exist, Chord calls `resolve` again by itself, with a per-address +backoff, so a tab that only watches recovers without a click. A worker start that already produced a link finishes +even if the request that triggered it went away. + +**Incarnations** make restarts visible. Chord mints a fresh id (a uuidv7) on every host start and sends it in the link +handshake; the other side sees a restart as a new incarnation of the same identity, and references to it re-hold +(chapter VI). + +## Tests: everything in one process + +The same code runs in a test with memory links. `resolve` builds the worker host in memory and returns one end of the +pair: + +```ts +server.route(Session, { + authorize: () => true, + resolve: async ({ id }) => { + const [up, down] = memoryLinkPair(); + const principal: P = { kind: "worker", session: id }; + const worker = createHost

({ address: Session({ id }), principal }); + await worker.load([{ name: "@studio/core", facet: coreWorkerFacet({ id, dataDir }) }, notesWorker]); + worker.connect(up, { principal: { kind: "server" } }); + return { link: down, principal }; + }, +}); +``` + +Memory links deliver the same encoded messages asynchronously, so a test runs the production protocol path. + +## What stays app code + +Starting processes, the start message, authentication, supervising and stopping idle workers are app code. Chord +provides one readable for the last one: `host.activity`, with `{ references, calls, jobs, lingering }` (viewers, calls +in flight, unsettled jobs, objects kept warm after their last reference). Studio's core reports it to the server +together with the agent's running tasks, and the server's supervisor stops a worker when everything has been zero for +five minutes, as p2 does. Chord counts; it has no idle policy. + +## Host options and timeouts + +`createHost` takes the options introduced later in this guide: `storage` (V), `principalKey` and `revokedBy` (III, +VII), `telemetry` (XVII) and `onError`. `onError` receives background errors and `{ kind: "storage_failed" }`; every +app must handle the latter, because Chord never ends the process. Studio exits with its own code so its supervisor +restarts it. + +Timeouts are host defaults with per-thing overrides; the full table is in the appendix. The two you meet first: +`readyTimeout` (60 s: how long an activation and an awaited `get` wait) and the park deadline (60 s: how long a call +waits through a link drop or a restart before failing `unavailable`). + +:::note | Built so far +Nothing in this chapter is built in Chord v2 yet: hosts, links and routes are layers L6 and L7. The frozen kernel has +working versions with the same shape (`createHost`, `connect`, `accept`, `route`, `load`), which the rebuild +simplifies. +::: diff --git a/packages/chord/docs/next/explainer/src/02-first.md b/packages/chord/docs/next/explainer/src/02-first.md new file mode 100644 index 000000000..5c01e5213 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/02-first.md @@ -0,0 +1,187 @@ + +# A First Object {#first-object lede="A plugin, a contract, a factory, and a browser tab that calls it on another process."} + +Studio's first feature is session notes: a list of short notes per session, shown in a panel, that anyone allowed can +add to. Several places need "the notes of S1": the owner's tab, a guest's tab, later a download route and an agent +tool. If each built its own copy, the copies would drift apart and nobody would know when to clean them up. In Chord +there is **one object** for "the notes of S1", everyone asks for it by its type and input, and Chord routes them all to it. + +## The plugin's files + +A plugin is a directory with a `package.json`. Its `name` (npm style, globally unique and stable) is the plugin's +identity: Chord finds its stored documents, reports its errors and orders its contributions by it, never by a +directory name or load order. + +| File | Runs on | Holds | +| --- | --- | --- | +| `package.json` | read by the loader | `"name": "@studio/notes"` | +| `contract.ts` | every host that imports it | object types: names, shapes, method signatures; no running code | +| `worker.ts` | every session worker | the worker facet | +| `server.ts` | the server | the server facet (later: the download route) | +| `web.tsx` | browser tabs | the web facet: the panel | + +## The contract + +An object type is a contract: a name, an **input** (the object's key among objects of this type), a **placement** +(on which host it lives, computed from the input), the **members** (state you can read and methods you can call) and a **version**. + +```ts +// notes/contract.ts +import { arr, int, method, obj, remote, str, type Static } from "@earendil-works/chord/next"; +import { Session } from "@studio/core/hosts"; + +export const Note = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int() }); +export type Note = Static; + +export const Notes = remote("notes", { + version: 1, + input: obj({ session: str() }), // the key: { session: "S1" } + placement: ({ session }) => Session({ id: session }), // input → the host's address + variants: { + editor: { + state: { notes: obj({ items: arr(Note) }) }, + methods: { + add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), // returns the new id + remove: method(obj({ id: str() })), // no result + }, + }, + }, +}); +``` + +`remote` means objects of this type can be reached from any host, across links (chapter IV introduces `local`). The +schema helpers (`obj`, `str`, `int`, `arr`, `map`, `opt`, `lit`, `oneOf`, `union`, `bool`, `num`, `nul`, `bytes`, +`unknown`) return ordinary TypeBox schemas; short options such as `{ min, max }` bound lengths or values. Schemas must +describe data: no callbacks, no class instances, no custom codecs. + +`method()` has four forms: `method()` (no arguments, no result), `method(args)`, `method(args, result)` and +`method(undefined, result)`. + +A contract contains no code that runs, so every host can import it. When you declare it, Chord materializes its +description and computes a **fingerprint**, a hash of its input, variants and members. Two hosts compare contracts by +version and fingerprint (chapter XVI). Placement is not part of the fingerprint. + +## Placement + +The input says *which* object; `placement` says *where* it lives. The caller runs the placement function on the input and sends the resulting address with its request; the server +routes by it. The provider runs the function too and refuses an object that is not its own with `invalid/placement`. +So a worker for S2 never builds "Notes for S1", even if a caller has an outdated rule. Because placement is a typed +function, a `Session` placement on a type whose input has no `session` field does not compile. + +Inputs are **normalized** before anything else: defaults are applied, unknown fields removed, and a canonical key +computed. `{ session: "S1" }` and `{ session: "S1", extra: 1 }` are the same object. + +## The facet and the factory + +A **facet** is a plugin's setup for one kind of host. It runs once when the plugin loads on that host and tells the +host which object types it serves: + +```ts +// notes/worker.ts +import { defineState } from "@earendil-works/chord/next"; +import { defineFacet } from "@studio/core"; // Studio's defineFacet: principals typed as Studio's P (chapter III) +import { Note, Notes } from "./contract.ts"; + +const NoteList = defineState("notes.list", { // a document (chapter V); in memory for now + schema: obj({ items: arr(Note) }), + version: 1, + initial: () => ({ items: [] }), + checkpointWhen: () => true, +}); + +export default defineFacet((host, self) => { + host.serve(Notes, ({ session }, principal, self) => { + const list = self.state(NoteList); // THE note list of this object + return { + variant: "editor", + notes: list, // a document handle is a readable: expose it as state + add: ({ text }, ctx) => + self.tx(async (tx) => { + const id = uuidv7(); + (await tx.draft(list)).items.push({ id, text, by: "someone", at: Date.now() }); + return id; + }, ctx), + remove: ({ id }, ctx) => + list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx), + }; + }); +}); +``` + +Read `host.serve(Notes, factory)` as "this host builds `Notes` objects like this". The factory receives the +normalized input, the caller's principal (chapter III) and `self`, through which it declares what the object needs. +It returns the members of one variant, tagged with `variant`. + +**Factories declare; Chord activates.** The factory is synchronous and has no side effects. It does not await, open +files or start timers. `self.state(NoteList)` does not load anything: it declares that the object has this document +and returns a **handle** that stays valid for the object's whole life. Chord then performs the activation: it loads the +document, binds references, opens OS handles, starts routines, and only then lets calls in. The rule exists for +reload: a reload runs the factory again, and anything the factory did itself would happen twice. + +## Calling it + +```tsx +// notes/web.tsx +export function NotesPanel({ session }: { session: string }) { + const notes = useRef(Notes, { session }); // app-side hook over host.get / release + const items = useReadable(notes?.notes)?.items ?? []; + return ( +

+ {items.map((n) =>

{n.text}

)} + notes?.add({ text })} /> +
+ ); +} +``` + +Underneath the hooks is the reference API: + +```ts +const notes = await client.get(Notes, { session: "S1" }); // live and hydrated once awaited +notes.notes.value.items; // the current list +notes.notes.subscribe((v) => render(v.items)); // every committed change +const id = await notes.add({ text: "check the flaky test" }); +notes.release(); // the object may end now +``` + +An awaited reference is **live and hydrated**: every state member has a value. State members are **readables**: +`.value` is the current value, `subscribe` is called after every commit. The first `get` for `{ session: "S1" }` runs +the factory; every later `get` for the same input and caller shares that object. The object ends when the last +reference is released (a type may keep it a little longer with `linger`). + +:::sequence animate | A call to add. Arguments are checked once at the provider; the commit reaches the viewer as one small change. +actor tab "browser tab" +actor server "server" muted +actor worker "worker S1" blue +actor obj "Notes { S1 }" accent +tab -> server: call add { text } +server -> worker: relay (header only) +worker -> obj: strip + validate args +obj => obj: self.tx: push note, commit +obj --> worker: result: id +worker --> server: state: ops of the commit +worker --> server: result +server --> tab: state, then result +::: + +## Two rules for members + +**Methods return data, never objects.** A method that wants to point you to another object returns its address +(`{ session: "S2" }`), and you `get` it yourself. That keeps every reference owned by someone who releases it. + +**No callbacks as arguments.** A callback carries code, and everything it closes over, from one object into another. +Suppose an inbox object passed `onFile` to another object, which stored it. After a reload of the inbox's plugin, the +other object still calls the *old* `onFile`; after the inbox ends, it calls into an object whose file watcher is +already closed. Chord forbids this by construction: members are data (JSON plus bytes). Behaviour from elsewhere goes +through registries (chapter XIII), multi-step work through jobs (XI), input from a person through flows (XII). + +## What is wrong with this version + +This first version works for one person. It has two problems, which the next two chapters fix. Everyone gets the +`editor` variant, including a read-only guest. And, less obvious, the owner and a guest do not see the same notes: as +chapter III shows, a remote object exists per caller, so each caller's `Notes` object has its own `NoteList`. + +:::note | Built so far +`remote`, `method`, the schema helpers, `Static` and fingerprints are built (`src/next/contracts/`). Serving, `get`, +references and documents are layers L3 to L5. +::: diff --git a/packages/chord/docs/next/explainer/src/03-principals.md b/packages/chord/docs/next/explainer/src/03-principals.md new file mode 100644 index 000000000..52a4c1064 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/03-principals.md @@ -0,0 +1,190 @@ + +# Who Is Calling {#principals lede="Principals: how Chord knows who calls, why a caller cannot lie about it, and how the answer picks the object and its members."} + +The owner may add and remove notes. Ann, a read-only guest, may only read them and should not even see who wrote +what. The object must therefore know who is calling, and the caller must not be able to claim to be someone else. In +Chord the answer is a **principal**, a small JSON value the app defines and the links stamp. + +## The app's principal type + +Chord defines no principal kinds; the app does. Every variant has a `kind`. Studio uses p2's set: + +```ts +// studio/src/principal.ts +export type P = + | { kind: "owner"; access: "local" | "relay" | "device"; device?: string } + | { kind: "guest"; session: string; mode: "read" | "write"; grant: string; revision: number } + | { kind: "server" } + | { kind: "worker"; session: string }; +``` + +| `kind` | Who | +| --- | --- | +| `owner` | the person running the app, on loopback, through the relay, or on a paired device | +| `guest` | someone invited into one session, with a mode and the grant (and its revision) that let them in | +| `server` | the server's own code calling a worker | +| `worker` | a session worker calling the server | + +## Links stamp principals + +A principal never comes from a request's data. The server authenticates a client once, when its WebSocket connects, +and stamps the result on the whole link (`server.accept(link, { principal })`). The server tells Chord who each worker +is when it resolves it (`return { link, principal }`). When the server relays a client's request to a worker, Chord +writes the client's principal into the forwarded request; a principal field coming from a client is always +overwritten, and only a trusted parent link may carry one. + +Where principals come from is app code. Studio's sharing feature creates a share link per session and mode; a guest +who opens it gets a guest record (grant `g1`) and a cookie; `authenticate(req)` turns the cookie into +`{ kind: "guest", session: "S1", mode: "read", grant: "g1", revision: 1 }` on every new connection. Chord never sees +cookies or grants, only the finished principal. + +## The caller is part of the identity + +A `remote` object exists once per **(type, input, principal key)**. The **principal key** is a string computed from +the principal; by default its canonical JSON (sorted keys). The app may pass its own `principalKey` to `createHost`, +and must pass the same function on every host, or identities split silently. + +```ts +const principalKey = (p: P) => + p.kind === "owner" ? `owner:${p.access}:${p.device ?? ""}` + : p.kind === "guest" ? `guest:${p.grant}:r${p.revision}` + : p.kind === "worker" ? `worker:${p.session}` + : p.kind; +``` + +The design wants the key to include how the client is connected and the grant's revision, so that two access paths +never share objects and a changed grant never reuses the old grant's object: + +| Caller | Principal key | `Notes { S1 }` object | +| --- | --- | --- | +| owner, tab A, local | `owner:local:` | 1 | +| owner, tab B, local | `owner:local:` | 1 (shared with tab A) | +| owner through the relay | `owner:relay:` | 2 | +| guest g1, revision 1 | `guest:g1:r1` | 3 | +| guest g1 after a change to write mode, revision 2 | `guest:g1:r2` | 4 (new) | + +## Variants + +The contract lists named sets of members; the factory receives the principal and returns the variant that principal +may use: + +```ts +// notes/contract.ts (version 1, final form for now) +variants: { + editor: { + state: { notes: obj({ items: arr(Note) }) }, + methods: { add: method(obj({ text: str({ min: 1, max: 2000 }) }), str()), remove: method(obj({ id: str() })) }, + }, + reader: { + state: { notes: obj({ items: arr(PublicNote) }) }, // no `by`: authors are hidden + }, +}, +``` + +```ts +// notes/worker.ts +host.serve(Notes, ({ session }, principal, self) => { + if (canWrite(principal, session)) return { variant: "editor", notes: …, add: …, remove: … }; + return { variant: "reader", notes: … }; +}); +``` + +A reader has no `add` at all. Chord refuses a call to a member that is not in the caller's variant, so `add` needs no +check of its own. Three rules keep variants compatible: a member name has the same kind (state or method) in every +variant that has it; a method has the same arguments and result in every variant; only state schemas may differ per +variant (the reader's notes have no authors). `remote(...)` enforces them at the type level, and at runtime for +contracts that arrive as data. + +## How a facet knows `P` + +`P` is Studio's type, not Chord's, and every host is created with it (`createHost

`). A plugin's facets are written +and compiled separately, then handed to `host.load` as values, so the type checker cannot infer `P` from the host into +the factory's `principal`. Studio's base code therefore exports its own `defineFacet`, already bound to `P`, and every +facet of every plugin imports that one instead of Chord's: + +```ts +// studio/src/facet.ts (base code, exported as @studio/core) +import { bindFacet } from "@earendil-works/chord/next"; +import type { P } from "./principal.ts"; + +export const defineFacet = bindFacet

(); // a facet's principals are Studio's P +export type { P, Revocation } from "./principal.ts"; +``` + +```ts +// notes/worker.ts (server.ts, web.tsx and tui.ts import the same defineFacet) +import { defineFacet } from "@studio/core"; + +export default defineFacet((host, self) => { + host.serve(Notes, ({ session }, principal, self) => { // principal: P + if (principal.kind === "guest" && principal.mode === "read") return { variant: "reader", notes: … }; + … + }); +}); +``` + +The import is cheap: `@studio/core` is a host package plugins already import at runtime (the contract imports `Session` +from it), `P` is a type-only import, and `bindFacet` is a one-line function. The host checks the pairing: `load` +accepts only facets typed for its own `P`, so a facet compiled against another app's principal type, or one built with +Chord's untyped `defineFacet`, does not compile into Studio's start code. Client facets (`web.tsx`, `tui.ts`) use the +same `defineFacet`; they rarely see a principal, because clients serve only `local` types. + +None of this is checked at runtime, and none of it needs to be: the principal a factory receives is the one the link +stamped. + +:::review | Review: binding facets to the app's principal type (open) +The design shows `defineFacet((host, self) => …)` without saying how `P` reaches the factory. The frozen kernel takes +it as an explicit type parameter, `defineFacet

(…)`, defaulting to `{ kind: string }` +(`packages/chord/src/next/facet.ts`), and the playground writes `defineFacet(…)`. The pre-bound +`defineFacet` above (`bindFacet` is a working name) is the owner's preferred direction, recorded as an open point in +`packages/chord/docs/next/layer-1.md` §4. +::: + +## Narrowing on the caller's side + +Owner and guest run the same `web.tsx`, and which variant they get is known only at runtime. The types force every +caller to narrow before using variant members: + +```ts +const notes = await client.get(Notes, { session: "S1" }); +notes.add({ text: "hi" }); // type error: `add` exists only in "editor" +if (inVariant(notes, "editor")) await notes.add({ text: "hi" }); +expectVariant(notes, "editor"); // or assert (throws) +function Editor({ notes }: { notes: RefIn }) { … } +``` + +The protection is not the type checker; it is Chord refusing the call at the provider. A client that ignores the +types gets the same error. + +## Admission before a worker starts + +Principals also decide whether a caller may reach a host at all. `route(Session, { authorize })` runs before Chord +resolves an address, so a guest of S1 asking for anything in S2 is refused at the server, and no worker starts for +them. Finer authority lives in the per-principal view's methods; there is no asynchronous "may this caller activate +this object" guard. An object whose address carries authority (say, a login flow by id) may exist; what a caller can +see and do is decided by the variant its factory picks. + +## Inside methods + +`ctx.principal` is the caller of the current call. In Studio's notes the editor's `add` uses it to record the +author: + +```ts +add: ({ text }, ctx) => store.add({ text, by: authorOf(ctx.principal) }, ctx), +``` + +Where principals act, in summary: + +| Purpose | Where | +| --- | --- | +| identity: which object a caller gets | principal key | +| variant: which members | the factory | +| admission: whether a host is reachable | `route.authorize` | +| attribution and fine checks | `ctx.principal` in methods | +| revocation | `revokedBy` (chapter VII) | + +## The bug from chapter II + +Now the bug is visible. The owner's `Notes { S1 }` and Ann's `Notes { S1 }` are two objects (rows 1 and 3 above), and +each declared its own `NoteList`. The owner's notes are not in Ann's list. Per-caller objects must not hold the data; +they must be views over something that exists once. That is the next chapter. diff --git a/packages/chord/docs/next/explainer/src/04-store.md b/packages/chord/docs/next/explainer/src/04-store.md new file mode 100644 index 000000000..44d3739ff --- /dev/null +++ b/packages/chord/docs/next/explainer/src/04-store.md @@ -0,0 +1,132 @@ + +# One Store, Many Views {#shared-store lede="Local types, and the pattern every Chord plugin uses: shared data once in a store, per-caller views in front of it."} + +Every caller has its own `Notes` object, which is what makes per-caller variants possible. But data must exist once. +The fix is the most common pattern in Chord: a **shared store**, a `local` object that holds the data and does the +heavy work once, behind thin per-principal **views**, the `remote` objects callers get. + +## `local` types + +```ts +// notes/contract.ts +export const NoteStore = local("notes.store", { + version: 1, + input: obj({ session: str() }), + members: { + notes: obj({ items: arr(Note) }), // state: a schema + public: obj({ items: arr(PublicNote) }), + add: method(obj({ text: str(), by: str() }), str()), // methods: method(...) + remove: method(obj({ id: str() })), + }, +}); +``` + +A `local` contract has flat `members` and no variants. The differences: + +| | `remote` | `local` | +| --- | --- | --- | +| reachable from | any host, across links | its own host only | +| identity | (type, input, principal key) | (type, input) | +| principals, variants | yes | no | +| factory | `(input, principal, self)` | `(input, self)` | +| arguments | stripped to the provider's schema and validated, for every caller | see the review note below | +| values on the same host | passed by reference, never copied | passed by reference, never copied | +| `ctx.principal` | the caller | none | +| may persist documents | no (chapter V) | yes | + +A local object has no principal: it is trusted code on its own host, acting for the host. So attribution ("who wrote +this note") is passed as data (`by`), by the view that knows who called. Clients serve only local types, because +nobody can address a client. + +## The pattern + +```ts +// notes/worker.ts +export default defineFacet((host, self) => { + host.serve(NoteStore, ({ session }, self) => { + const list = self.state(NoteList); + const pub = self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) })); + return { + notes: list, + public: pub, // computed once, for every reader + add: ({ text, by }, ctx) => + self.tx(async (tx) => { + const id = uuidv7(); + (await tx.draft(list)).items.push({ id, text, by, at: Date.now() }); + return id; + }, ctx), + remove: ({ id }, ctx) => list.change((d) => { d.items = d.items.filter((n) => n.id !== id); }, ctx), + }; + }); + + host.serve(Notes, ({ session }, principal, self) => { + const store = self.get(NoteStore, { session }); // a reference, declared: held for the view's life + if (canWrite(principal, session)) return { + variant: "editor", + notes: store.notes, // the store's readable, exposed as-is + add: ({ text }, ctx) => store.add({ text, by: authorOf(principal) }, ctx), + remove: (args, ctx) => store.remove(args, ctx), + }; + return { variant: "reader", notes: store.public }; + }); +}); +``` + +:::flow wide | One store, three views. The redaction runs once in the store; both readers read the same value. +row "Per-principal views (remote)" blue +box ve "Notes { S1 }" "owner · editor" blue +box vr1 "Notes { S1 }" "guest Ann · reader" blue +box vr2 "Notes { S1 }" "guest Bob · reader" blue +row "Shared store (local)" accent +gap +box store "NoteStore { S1 }" "notes, public (a derive)" accent +gap +arrow ve -> store "self.get" +arrow vr1 -> store "self.get" +arrow vr2 -> store "self.get" +note "Authority is the view's: the reader variant has no add. The store does not check; it is trusted and acts for the host." +::: + +`self.get(NoteStore, { session })` in a factory declares a **reference** that the view holds for its whole life; Chord +binds it during activation and waits until the store is live (chapter VI). The view exposes the store's readables +directly. Owner, Ann and Bob now give three `Notes` objects, one `NoteStore`, one note list and one redacted list. + +Any heavy or shared part belongs in a store like this: a pi-durable watch, a file watcher, a sandbox, a search index. +Base code uses the same pattern with the same primitives; nothing about it is special to core. + +## Derived values, briefly + +`self.derive(name, sources, fn)` computes a value from readables and recomputes it when a source changes; viewers get +only the difference. The reader's list above is one. Chapter VIII covers derives in depth, including the one footgun +that matters: `stripAuthor` must return the *same* object for an unchanged note, or every commit costs the size of the +whole list. The tool for that is `mapShared`. + +## Arguments are checked at the provider + +Trusted code is not type-checked code: plugins are bundled with esbuild without type checks, and agents write +JavaScript plugins. So the provider checks arguments itself (decision R7): every method's arguments are stripped to +the **provider's own** schema (unknown fields removed) and validated with it, whoever the caller is, and never with a +schema a peer sent. Schemas are compiled lazily on first use and cached per schema object. Bytes are checked as leaves +(`instanceof Uint8Array` plus an optional length bound), never walked. Stripping copies only when a field was removed; +the caller's value is never mutated (proposal Q12, in the L1 draft). Results and state are not validated: they come +from the provider. + +On the same host nothing is copied: arguments and results are passed by reference. Values are **immutable by +convention** (Delta's rule, as in Durable): nobody mutates a value it received, and Chord neither copies nor freezes +values to enforce it. There is no development mode that would freeze them; one behaviour everywhere (R9). + +:::review | Review: are local arguments validated? (Q15, open) +Design §5's table says `local` arguments are "neither validated nor copied"; §3 and R7 say arguments are validated +"for every caller". Review round 3 raised this as Q15 (apply R7 to `local` too, or keep `local` as the one exception), +and `layer-1.md` lists it as open. Until it is decided, write local methods so that they do not depend on either +answer: trust the shape, but never trust content (lengths, ids) that came from a remote caller without the view +checking it. `chord-next-walkthrough.md` concept 6 still describes the older rule ("validated only when the caller is +not a host principal"), which R7 replaced. +::: + +## Choosing between the two + +Use a `remote` type for what callers address: views, things with per-caller authority. Use a `local` type for what +exists once per input on one host: stores, caches, owners of OS handles, and the core objects a host provides +(chapter IX). If a type is reached across a link, it is remote; if a type persists documents, it is local (or a +host-lifetime object, chapter V). diff --git a/packages/chord/docs/next/explainer/src/05-documents.md b/packages/chord/docs/next/explainer/src/05-documents.md new file mode 100644 index 000000000..2a25a1caa --- /dev/null +++ b/packages/chord/docs/next/explainer/src/05-documents.md @@ -0,0 +1,258 @@ + +# Writing It Down {#documents lede="Documents and transactions: how an object stores data, changes it atomically, survives restarts and migrates it."} + +Notes must be stored, changed safely while others read them, sent to viewers as small changes, and survive a worker +restart. An object's data lives in **documents**: named JSON values the object owns, changed only in +**transactions**. The design is a port of pi-durable's store, so most rules will look familiar; this chapter states +them for Chord. + +## Defining a document + +```ts +// notes/worker.ts +const NoteList = defineState("notes.list", { + schema: obj({ items: arr(Note) }), + version: 1, + initial: () => ({ items: [] }), + persist: true, // stored; default: memory only + checkpointWhen: (_value, _ops, info) => info.deltasSinceBase > 64, +}); + +const Draft = defineState("notes.draft", { // one unfinished note per author + schema: obj({ text: str() }), + version: 1, + initial: () => ({ text: "" }), + keyed: true, // many per object, by key + persist: true, + checkpointWhen: () => true, // small: a full base per write +}); +``` + +`checkpointWhen` is required on every definition, with no default (R14, as in Durable since pi `5c1736f82`): it says +when a commit stores a complete base instead of a delta, which bounds how much is replayed on load. Small documents +checkpoint on every write; growing ones after a number of deltas. + +A document's root is always a JSON object (lists live in a field). Documents hold JSON only; bytes are a value and wire +type, not a document type. + +## Singleton and keyed + +```ts +host.serve(NoteStore, ({ session }, self) => { + const list = self.state(NoteList); // THE note list of this object + const drafts = self.state(Draft); // all drafts of this object + … +}); +``` + +| | singleton (no `keyed`) | keyed | +| --- | --- | --- | +| how many | one per object | many per object, by a short key | +| loaded | at activation, kept in memory | on demand | +| read | `list.value` (synchronous) | `await drafts.get(key)`: the value or `undefined` | +| list | (it is one value) | `for await (const key of drafts.keys())` pages keys | +| expose as state | yes: the handle is a readable | no: expose a method or a singleton index | + +There is no live key list. A collection shown live is a singleton (an object holding a list or a map); a large +collection is a method returning pages. + +## Transactions + +All writes go through `self.tx(fn, ctx)`. Inside, `await tx.draft(handle, key?, { tags? })` returns a draft of the +current document, or a draft over `initial()` if it is absent. Drafts record operations (assignments, `push`, +`Object.assign`); asking for the same document twice in one transaction returns the same draft. `tx.retire(handle, +key?)` removes a document. There is no `tx.get`: reading is drafting, and a draft is an overlay over the current +value, so nothing is copied. A draft you do not change commits nothing, so reading never stores an absent document. + +```ts +saveDraft: ({ author, text }, ctx) => + self.tx(async (tx) => { (await tx.draft(drafts, author)).text = text; }, ctx), +``` + +Shorthands for one document: `handle.change(fn, ctx)` (keyed: `handle.change(key, fn, ctx)`) (loads the document, then runs the synchronous `fn` on +its draft) and `handle.retire(ctx)` (keyed: `handle.retire(key, ctx)`). + +### Four patterns + +Two decisions (R12 and R13, both as in Durable) shape how transactions are written. **Drafts are revoked when the +transaction settles**, so a transaction returns at most plain values, never something taken from a draft. And +**there is no existence check inside a transaction.** + +```ts +// 1. Return data: write in the transaction, read after the commit. +await self.tx(async (tx) => { (await tx.draft(list)).items.push(note); }, ctx); +return list.value.items.length; // the committed revision, or a newer one + +// 2. Remove if present: retire of an absent document is a no-op. +await drafts.retire(author, ctx); + +// 3. Create with tags if missing: the draft creates it, only if it changes. +await self.tx(async (tx) => { + (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; +}, ctx); + +// 4. Branch on existence: read first, outside the transaction. +if ((await drafts.get(author)) === undefined) { … } +``` + +### Durable's rule + +Inside the callback, only `await tx.*`. No other objects, no network, no models, no processes, no second `self.tx`, no +reads past `tx`. Breaking it is not an error but a hang: the object's line is held while the callback waits. Fetch +first; decide and write inside. + +```ts +// Wrong: a fetch inside the transaction holds the object's line. +await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title: await fetchTitle(url) }); }, ctx); +// Right: fetch, then write. +const title = await fetchTitle(url, ctx.signal); +await self.tx(async (tx) => { (await tx.draft(list)).items.push({ …, title }); }, ctx); +``` + +## What a commit does + +:::flow wide animate | The commit sequence, Durable's. Nothing is copied, frozen or revalidated; viewers get one batch per commit. +row "Inside self.tx" accent +box line "wait for the line" "per object" +box run "run the callback" "drafts record ops" accent +box prep "prepare" "Delta checks placed values" +box write "storage batch" "deltas or a base" accent +row "After the write" blue +box adopt "adopt" "pointer swap" +box local "notify locally" "once, with the ops" blue +box wire "send" "one batch per link" +box revoke "revoke drafts" "touching one throws" +arrow line -> run +arrow run -> prep +arrow prep -> write +arrow write -> adopt "stored" +arrow adopt -> local +arrow local -> wire +arrow wire -> revoke +note "An error or an abort before the storage write discards everything. Storage commits are serialized per host." +::: + +Lines are per object, never across objects; reads outside a transaction do not take the line. Subscribers on the same +host get the new value and the commit's operations by reference. Remote viewers get the commit's operations, never the +document again; chapter XVII shows how one commit touching several holds still arrives as one message. + +## Tags and deleting by tag + +Tags are typed tokens set when a document is created: + +```ts +export const SessionTag = defineTag("studio.session"); // base code, shared by every plugin +await self.tx(async (tx) => { (await tx.draft(drafts, author, { tags: [SessionTag(session)] })).text = text; }, ctx); +// when the session is deleted +await store.retireByTag(SessionTag, "S1"); // every document tagged S1 +``` + +The design names the operation `store.retireByTag(tag, value)` without saying who holds the store; session delete is +one of the app decisions still open (#18, chapter XIX). + +## Who may persist + +`persist: true` is allowed only in `local` objects and `lifetime: "host"` objects: identities without a principal. A +`remote` object's identity contains the principal key, which changes with the access path and the grant revision. Its +stored documents would scatter across access paths and be orphaned after a reconnect. The rule is enforced at the type +level (`self.state` in a remote factory accepts only non-persisted definitions) and at declaration ("persist needs a +stable identity: put the documents into a local shared store"). This is the second reason for the shared store. + +What survives a host teardown (a crash, a restart) is exactly what is persisted. In-memory documents live as long as +their object: they survive reloads and native restarts of a live object, and go when it ends. Everything else +(natives, derives, routines, references) is rebuilt on activation. + +## Facet documents + +A facet has its own identity (its plugin's name) and may hold documents too, for plugin-wide data such as +preferences. Objects the facet serves may read and write their facet's documents, and one transaction may span the +object's own documents and its facet's (lock order: own line, then the facet's). The facet never touches a served +object's documents. + +## Ownership across plugins + +No plugin reads or writes another plugin's documents, Chord or Durable. Data crosses plugins only through state and +methods. Data that core enforces lives in a format core owns and is handed to core through its API when the data is +created (as with Durable's `createConversation({ init })`), never written into core's documents. + +## Versions and migration + +A change of shape or meaning raises `version` and adds a pure `migrate(old, from)`: + +```ts +const NoteList = defineState("notes.list", { + schema: obj({ items: arr(NoteV2) }), // NoteV2 adds `pinned` + version: 2, + migrate: (old, from) => ({ items: old.items.map((n) => ({ ...n, pinned: false })) }), + initial: () => ({ items: [] }), + persist: true, + checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64, +}); +``` + +Documents migrate on access by the new definition: singletons during activation (hydration counts as access), keyed +ones on their next access. The migrated base is persisted with the next commit; right after a bump, a read-only +transaction writes it. + +Chord does not walk values against schemas. Instead, each definition computes a **schema fingerprint** once +(descriptions and titles ignored), and every stored record keeps the fingerprint it was written with. On access Chord +compares them: the same version with a different fingerprint fails the object `broken` ("schema changed without a +version bump"). So every structural change needs a bump, an additive one with a trivial `migrate`. The one exception +to "no value checks" (R8): the result of `migrate` and of `initial()` is validated against the new schema once, when +produced. A failed check, a throwing `migrate`, or old code meeting a newer document refuses the access, keeps the +stored data and fails the owning object. A changed meaning without a schema change cannot be detected; that is on +you. + +## Storage + +Behind the store is a `DocumentStorage`, passed as `createHost({ storage })`; memory by default. The server has one +shared storage, each worker one for its session. + +```ts +interface DocumentStorage { + commit(writes: readonly DocumentWrite[]): Promise; // create/change/retire, atomic + find(address: DocumentAddress): Promise; + load(id: string): Promise; // version, value, deltasSinceBase + scan(query: { owner: string; definition: string }, limit: number, cursor?: string): + Promise<{ keys: string[]; next?: string }>; + byTag(tag: string, value: string, limit: number, cursor?: string): Promise<{ ids: string[]; next?: string }>; + mintId(): Promise; +} +``` + +Chord ships a memory storage and a SQLite storage (over `@earendil-works/chord/sqlite`, the driver layer that moved +from Durable; Node and Durable Object drivers). Storages check only metadata (incarnation, version, sequence, address +occupancy, fingerprint) and write the supplied delta or base; they never rebuild or validate content on a commit. +Chord documents never commit together with Durable transcripts: Chord has its own storage, not an adapter over +Durable's. Your own storage can prove itself with `registerDocumentStorageConformance(runner, name, withStorage)` from +`@earendil-works/chord/testing`. + +Keyed persisted documents that nobody reads, writes or watches for 10 minutes leave memory and reload on the next +access. A storage error ends the store and closes the host; `onError` receives `{ kind: "storage_failed" }` and the +app decides (chapter I). + +## Kinds of state + +| Kind | For | API | +| --- | --- | --- | +| document | data the object owns | `self.state`, `self.tx` | +| derive | computed values | `self.derive` (chapter VIII) | +| source readable | external data: Durable views, files, logs | value plus ordered changes; feeds derives | +| paged method | large or queryable collections | a method returning pages (chat history) | +| plugin database | full-text search, large blobs | a native opened by its owner (chapter IX) | + +App state lives in Chord documents. Only transcript-bound data, which must fork and rewind with a conversation, lives +in Durable. + +:::review | Review: does an absent singleton read `undefined` or `initial()`? (Q11, open) +Design §9 says "whether a document exists is read outside the transaction (`handle.value` / `handle.get` give +`undefined` for an absent one)" and also "a singleton that does not exist yet reads `initial()` through its handle; +retiring resets it to `initial()`". Review round 3 proposes: singletons always read `initial()`, only absent keyed +documents read `undefined`, and singleton existence needs no API. This guide follows the proposal (`list.value` is +never `undefined`); it is not decided. +::: + +:::note | Built so far +Documents, transactions and storages are layer L3, a port of Durable's session store; not started. The frozen kernel +has a working version with a JSONL storage, which the design rejected. +::: diff --git a/packages/chord/docs/next/explainer/src/06-references.md b/packages/chord/docs/next/explainer/src/06-references.md new file mode 100644 index 000000000..b852916b5 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/06-references.md @@ -0,0 +1,185 @@ + +# Holding On {#references lede="References, the object lifecycle and ctx: what a caller sees while objects start, fail, restart and move, and why nobody writes retry code."} + +A tab shows the notes of S1 for an hour. During that hour the laptop sleeps, the worker is stopped for idleness and +started again, the notes plugin reloads twice and the store's storage hiccups once. The panel should show the notes the +whole time, say "reconnecting" when it cannot reach them, and recover by itself. In Chord this is what a +**reference** does; plugin code contains no retry loops and no request ids. + +## Three ways to hold + +| Where | API | Released | +| --- | --- | --- | +| a factory | `self.get(Type, input)` | when the object ends | +| a call | `await ctx.get(Type, input, { state? })` | when the call ends | +| outside objects | `await host.get(Type, input, { signal, state?, parkTimeout? })` | `ref.release()` | + +Use `self.get` for what an object needs its whole life (the view's store), `ctx.get` for what one call needs, and +`host.get` only in UI code, tests and entry points. `{ state: false }` binds for methods only, without receiving state. +A reference keeps its object alive while held. + +## The object lifecycle + +```text +(get) → activating → live → ending → ended + └→ failed (backoff) → activating … +``` + +**Activating:** Chord binds the declared references and waits until they are live and hydrated, loads singleton +documents, opens natives (one after another, in declaration order), computes derives and starts routines. The object +goes live when all of that succeeded. Calls wait behind its **gate** until then. + +**Ending:** routines stop, the object's calls and jobs are aborted as their rules say, natives close in reverse order, +references are released. An object with `lifetime: "referenced"` (the default) ends when its last reference goes, +after the type's `linger` if it has one. A `lifetime: "host"` object (only for types without a required input) lives +as long as the host. A running, finishing or blocked job counts as a reference (chapter XI). + +## `ref.state` + +Every reference reports what is going on as a tagged value; listeners get each new one: + +```ts +type RefState = + | { kind: "loading" } // first hold, reconnecting, restarting + | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // undefined: no automatic retry + | { kind: "released" }; +``` + +| Situation | Calls | Values | +| --- | --- | --- | +| link drop | park until reconnected | readable, `loading` | +| object loading or restarting | park until live | readable, `loading` | +| rollout: provider ahead or installing | park until installed | readable | +| object `failed` | reject at once with the cause | `failed`, with cause and `retryAt` | + +Values stay readable through all of this: the panel keeps showing the last notes. Parked calls share one deadline (60 +s by default, `parkTimeout` per `get`), then fail `unavailable`. Otherwise a call ends only through its caller's +signal; there is no per-call deadline. A call that was running on an instance that ends fails with an error and is +not resent: Chord cannot know whether it took effect. + +**Errors look like disconnects.** A consumer handles a failed call exactly as a dropped link: show the error, let the +user retry. Work that must not run twice carries its own idempotency key as data (Durable's `requestId`, a job's +`key`). + +:::steps | The worker restarts while a tab watches the notes ++++ Watching +The tab holds `Notes { S1 }`. `ref.state` is `live`; `notes.notes.value` has 40 notes. ++++ The worker goes away +The supervisor stops worker S1 (or it crashes). The server's IPC link to it drops. The server tells the tab that its +hold is gone; the tab's reference goes `loading`. The 40 notes stay readable. ++++ A click meanwhile +Ann's colleague clicks **Add**. The call **parks** on the reference; nothing is sent yet. The deadline is 60 s. ++++ Chord resolves again +Holds on S1 still exist, so Chord calls `route(Session).resolve({ id: "S1" })` again, with the per-address backoff. +The app forks a new worker: a new **incarnation** of `Session({ id: "S1" })`. ++++ Re-hold +The server re-holds every hold on S1 over the new link. The new worker activates `NoteStore { S1 }` (its notes load +from storage) and the views; it sends fresh snapshots. The tab's reference goes `live`. ++++ The parked call runs +The parked `add` is sent now and succeeds. The panel shows 41 notes. Nobody wrote recovery code. +::: + +## Failure and retry + +Any broken part fails the **whole** object: a half-working object could break rules that rely on its documents +changing together. Chord tags the cause by where it failed: + +```ts +type FailureCause = + | { kind: "transient"; error: ChordError } // an open threw, a dependency was unreachable: automatic backoff + | { kind: "broken"; error: ChordError }; // factory, derive, hydration or migrate threw: retried only by a trigger +``` + +**Transient** causes get a backoff per address: 1 s, doubling to 60 s; concurrent gets share one attempt; success +resets it. **Broken** causes would fail the same way again (a code bug, invalid data), so they are retried only by a +**trigger**: a plugin reload, `ref.retry()` (a Retry button), or a readable the factory declared with +`self.retryOn(...readables)`. Only an opener may declare its own failure broken, with `throw broken(error)`, for +example for invalid configuration. A failed reference stays held; on success it goes `loading`, then `live`. + +```ts +host.serve(EnvConnection, ({ env }, self) => { + const config = self.get(EnvConfig, {}); // a facet or registry value, outlives the attempt + self.retryOn(config.envs); // a config change retries a broken open + const ssh = self.open(async ({ signal }) => { + const c = config.envs.value[env]; + if (!c) throw broken(new Error(`unknown environment ${env}`)); + return connect(c, signal); + }, (s) => s.close()); + … +}); +``` + +`retryOn` sources must live independently of the attempt (facet documents, registry values); a member of one of the +object's own references is a type error, since it goes away with the attempt. + +## Waiting for dependencies + +An object whose references are not live yet stays `loading` and goes live the moment they are, bounded by +`readyTimeout` (60 s). The decision for the failure case (R18) is what makes recovery automatic: + +:::sequence animate | Dependency recovery. The view fails transient with the store's cause, keeps its reference, and is retried the moment the store goes live. +actor tab "tab" +actor view "Notes { S1 }" blue +actor store "NoteStore { S1 }" accent +tab -> view: hold +view -> store: self.get (activation) +store -> store: open fails: disk not mounted +store --> view: failed (transient) +view --> tab: failed: transient, cause inside +note view..store: the view keeps its reference; its state is the retry trigger +store -> store: backoff retry: open succeeds +store --> view: live +view -> view: retried at once (backoff cancelled) +view --> tab: live +::: + +A dependency that fails, or does not become live in time, fails the dependent **transient** with the dependency's +cause inside, never broken: the cause is the dependency's. The dependent keeps its references while failed, and Chord +uses their state as a retry trigger. Factory authors write `self.get` and know none of this. After an object is live, +a dependency leaving `live` does not change it: its calls that use the dependency get the dependency's error, or park +while it loads. + +**Cycles are rejected** (Q5). If A's activation gets B and B's gets A, neither would ever lose its last reference. +Every object instance has a uuidv7 instance id; on one host Chord follows the waiting edges of activating objects, and +across hosts a hold made during an activation carries the chain of instance ids. A cycle fails the activation +`broken`, naming it ("A → B → A"), and keeps no references. + +## No optional references + +`get` has no `optional` flag. When a dependency's plugin may be missing, turn the dependency around: the dependent +owns a registry and the other plugin contributes to it. Radius uses forwards from the ports plugin, but ports is +optional, so radius owns `radius.forwards` and ports contributes its forwards; without ports the table is empty and +radius works (chapter XIII). + +## `ctx` + +Every call gets a `ctx` as its last argument: + +| Field | Meaning | +| --- | --- | +| `signal` | fires when the caller cancelled or the object ended; pass it to everything you wait on | +| `principal` | the caller (remote types only) | +| `incarnation` | the calling host's incarnation, for calls that arrived over a link (chapter X) | +| `get` | references released when the call ends | +| `telemetry`, `trace(name, attributes, fn)` | the current span; a child span with a child `ctx` (chapter XVII) | +| `jobId`, `progress` | in a job's `run` (chapter XI) | + +```ts +exportText: async ({}, ctx) => { + const catalog = await ctx.get(Catalog, {}); // released when the call ends + const title = catalog.sessions.value[session]?.title ?? session; + const res = await fetch(renderUrl, { method: "POST", body: render(title, list.value), signal: ctx.signal }); + return await res.text(); +}, +``` + +There are no code versions and no request ids in `ctx`. Durable's `Context` is the same module (`chord/context`): +`ctx` extends it, so passing `ctx` into pi-durable needs no conversion. Code outside objects gets a `ctx` from +`host.root(fn, { principal?, signal? })`; inside Durable tools it uses `chord(fn, context)` (chapter XIV). + +:::note | Built so far +References, `ctx` and the lifecycle are layers L4 and L5. `ChordError` and `FailureCause` are specified in the L1 +draft (`layer-1.md`), with codes `invalid`, `contract_mismatch`, `forbidden`, `gone`, `not_ready`, `released`, +`too_large`, `unavailable` and `aborted`, and a finer `reason` such as `invalid/args`. +::: diff --git a/packages/chord/docs/next/explainer/src/07-revocation.md b/packages/chord/docs/next/explainer/src/07-revocation.md new file mode 100644 index 000000000..1cacf921b --- /dev/null +++ b/packages/chord/docs/next/explainer/src/07-revocation.md @@ -0,0 +1,113 @@ + +# Taking Access Away {#revocation lede="Revocation: how removing a guest ends their objects on every host at once, and why reconnecting does not bring access back."} + +The owner removes Ann from S1. Her tab still has a WebSocket open and a `Notes { S1 }` view on worker S1. Both must end +now, on the server and on the worker, and her tab's automatic reconnect must not restore anything. Chord provides the +mechanism; the app decides who is blocked and remembers it. + +## Labels, not predicates + +The app declares, once, which **revocation labels** each principal carries. Revoking means revoking one label; +everything carrying it is hit. + +```ts +// studio/src/principal.ts +export type Revocation = { grant: string } | { device: string } | { session: string }; + +export const revokedBy = (p: P): Revocation[] => + p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] + : p.kind === "owner" && p.access === "device" && p.device ? [{ device: p.device }] + : []; + +const server = createHost({ address: Server(), principal: { kind: "server" }, revokedBy }); +// every worker is created with the same revokedBy +``` + +| Principal | Labels | +| --- | --- | +| guest Ann (grant g1) in S1 | `{ grant: "g1" }`, `{ session: "S1" }` | +| guest Bob (grant g2) in S1 | `{ grant: "g2" }`, `{ session: "S1" }` | +| owner on the paired phone | `{ device: "phone" }` | +| owner, local | none | + +| Call | Hits | +| --- | --- | +| `server.revoke({ grant: "g1" })`: remove Ann | Ann | +| `server.revoke({ session: "S1" })`: archive S1 | Ann and Bob | +| `server.revoke({ device: "phone" })`: sign out the phone | the phone | + +A label is plain data because revocation is an event that must travel from the server to the workers; a predicate +function could not be sent. + +## What `revoke` does + +`host.revoke(r)` ends every object and closes every link whose principal `p` has `r` in `revokedBy(p)` (compared by +canonical JSON; never the host's own principal), and passes `r` to its children, which apply their own `revokedBy`. + +:::sequence animate | Removing guest Ann. The label travels down; her objects end on every host; her reconnect is refused by the app's own record. +actor owner "owner tab" +actor server "server" accent +actor worker "worker S1" blue +actor ann "Ann's tab" "guest g1" muted +owner -> server: Sharing.remove({ grant: "g1" }) +server -> server: app: mark g1 blocked (document) +server => server: revoke({ grant: "g1" }) +server -> ann: close link +server => worker: revoke { grant: "g1" } +worker -> worker: end Notes { S1 } as g1 +ann -> server: reconnect +server -> server: authenticate: g1 blocked +server --> ann: refused (4401) +::: + +On worker S1, for `revoke({ grant: "g1" })`: + +| Object | Principal | Hit | +| --- | --- | --- | +| `Notes { S1 }` | owner, local | no | +| `Notes { S1 }` | guest g1 | **yes**: ends | +| `Notes { S1 }` | guest g2 | no | +| `NoteStore { S1 }` | none (local) | no | + +Ending Ann's view releases its reference to the store, which keeps serving the others. + +## Chord persists nothing here + +`revoke` cuts live access only. Lasting denial lives in the app's data (the sharing plugin's guest records), which +`authenticate` checks on every new connection. After a server restart nothing live exists to revoke; every client +authenticates again and blocked grants are refused there. + +**No re-stamping.** A changed grant (Ann moves from read to write) is a revocation of the old grant. Her link closes, +her tab reconnects and authenticates with the new grant revision, which gives a new principal key, and her references +re-hold by themselves, now as an editor. Reconnecting never restores revoked authority. + +**Revocation beats jobs** (R11). Normally a running job keeps its object alive after the last reference is released +(a closed tab does not abort an import). Revoking a principal ends its objects at once regardless; their jobs get the +abort signal and the normal close timeout. + +## Work in shared stores + +Revocation ends objects whose principal carries the label. A shared store has no principal, so work it runs *on +behalf of* someone (a question waiting for Ann's answer, a shell command a guest started) is not ended by `revoke`. +Such a store keeps the grant as data with the work and reconciles from the grant owner's live state with a routine +(chapter X): + +```ts +// in a shared store that runs work for guests +const sharing = self.get(Grants, {}); // the sharing plugin's live grants +self.routines.on(sharing.live, async (live, ctx) => { + for (const run of shell.running.value) + if (run.args.grant && !live[run.args.grant]) shell.abort(run.id); // idempotent +}); +``` + +There is no revocation hook: a hook can be missed (a crash between revoke and hook), a routine over current state +cannot. The window between the revocation and the routine's run is accepted (R10); a step started in it still runs. + +## Who does what + +| Chord | The app | +| --- | --- | +| stamps principals on links | what a grant is; share links, cookies, `authenticate` | +| keeps one object per principal key | defines `Revocation` and `revokedBy` | +| on `revoke(r)`: ends matching objects and links, passes `r` down | decides when to revoke; stores who is blocked | diff --git a/packages/chord/docs/next/explainer/src/08-derives.md b/packages/chord/docs/next/explainer/src/08-derives.md new file mode 100644 index 000000000..720542ef5 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/08-derives.md @@ -0,0 +1,114 @@ + +# Computed Values {#derives lede="Derives, the one footgun that makes them expensive, mapShared, and presence: who is looking."} + +The reader's notes without authors, a note count, the number of people viewing: values computed from other values. +Storing copies would go stale; recomputing them on every read would be wasteful and could not be watched. A +**derive** is computed once per change and sent to viewers as a difference. + +## `self.derive` + +```ts +const count = self.derive("count", [list], ([l]) => l.items.length); +const summary = self.derive("summary", [list, store.pinned], ([l, pinned], previous) => ({ + total: l.items.length, + pinned: pinned.length, +})); +``` + +`self.derive(name, sources, (values, previous) => value)`. Sources can be documents, readables, other derives and +state members of references. The value is recomputed when a source changes and diffed into operations for the wire. +Derived values are adopted before ordinary observers run and share their source commit's wire batch, so a viewer +never sees a note list and a count from different commits. + +`previous` is the derive's last value, kept across factory re-runs (reloads) by name. It is a cache, empty after a +host teardown, so a derive must be correct without it. A derive that throws fails its whole owner `broken`. + +## How the difference is found + +Chord compares the new value with the previous one using Delta's `diffRevisions`. Parts that are the **same objects** +as last time cost one `===` each. Parts that are new objects must be compared by content. + +**The footgun.** A derive that builds new objects for unchanged data makes every commit cost the size of the whole +value, per view. The reader list from chapter IV, written naively: + +```ts +self.derive("public", [list], ([l]) => ({ items: l.items.map((n) => ({ id: n.id, text: n.text, at: n.at })) })); +// one note added to 2,000: 2,001 new objects, every one compared by content +``` + +**`mapShared(f)`** keeps one result per source element: `f` runs only for new or changed elements, unchanged ones +return their previous result object. + +```ts +import { mapShared } from "@earendil-works/chord/delta"; + +const stripAuthor = mapShared((n: Note): PublicNote => ({ id: n.id, text: n.text, at: n.at })); +self.derive("public", [list], ([l]) => ({ items: l.items.map(stripAuthor) })); +// one note added: one new object, 2,000 identical references; the diff emits one insert +``` + +:::review | Review: mapShared is planned, not available +Design §12 introduces `mapShared` as coming "from `@earendil-works/chord/delta`", but nothing in `packages/chord/src` +defines it yet, and Delta does not yet treat bytes as atomic leaves in `diffRevisions`. Design §28 lists both as the +step "Delta in pi", before the layered rebuild. Until then, keep a `Map` from source element to result yourself, which +is what `mapShared` will do. +::: + +## Cookbook + +| Case | How | +| --- | --- | +| select a field, look up by key, a small aggregate | compute it; the value is small | +| filter or map a list | `mapShared` over the elements | +| redaction | preferably separate documents per visibility; otherwise `mapShared`, computed once in the shared store, never per view | +| join two sources | `mapShared` plus the looked-up source's revision | +| a window over a large list | take a section of the source elements without rebuilding them; cache per-element measurements by identity; keep the window's start stable so changes arrive as appends | +| a large view combined from several sources | re-path the sources' operations, as Durable's `ConversationView` does; application code never interprets ops | + +**Container width** (R24): structural sharing is O(change) along a path, not across a container's width. Changing one +item of a flat array copies the array's slots, and `mapShared` visits every element once (one identity comparison +each). Large collections are keyed documents or bounded windows. A benchmark of long arrays is to set the documented +limit. + +## Readables that are not documents + +Three small readables round out the toolkit: + +- `constant(value)`: never changes. Start data uses it (chapter IX). +- `self.writable(initial)`: a host-local readable of any in-process value (functions included), compared by identity, + with `set`. It never goes on the wire; it exists for registry contributions that change (chapter XIII). +- Source readables from elsewhere: a Durable view or document watch is already a Chord readable, because Durable and + Chord v2 share the readable type (`chord/src/state.ts`), so it can feed a derive or be exposed as state directly. + +:::review | Review: exposing a writable (Q14, open) +A `writable` may hold functions; nothing yet stops a factory from returning one as a state member, which would reach +the wire encoder. Review round 3 proposes that only members with a JSON schema in the contract are exposable, checked +when the object is served. Undecided. +::: + +## Presence: who is looking + +`host.references(Type)` is a readable of who holds references to objects of a type on this host: input, principal and +client. Client holds carry an opaque origin id, so two tabs are two clients. + +```ts +host.serve(NoteStore, ({ session }, self) => { + const viewers = self.derive("viewers", [host.references(Notes)], ([refs]) => + refs.filter((r) => r.input.session === session).length); + return { …, viewers }; +}); +``` + +A link drop is a departure: accurate and simple. Smoothing (a phone switching networks makes the count flicker) is the +UI's job, for example a debounced count. Counts per person are derived by the app from principals and origins. There +is no consumer-side linger: a closed tab closes its link and keeps nothing. The provider's `linger` keeps a +just-closed object warm, and that also keeps the worker alive through `host.activity`. + +**"Leaving" is never derived from an unmount.** A UI reload unmounts and remounts components; a component unmounted by +a reload has not left. Leaving is explicit navigation, or presence (a dropped reference). This replaces p2's presence, +viewer and login-abandon leases with one mechanism. + +:::note | Built so far +Readables and derives are layer L2 (a readable is today's `replicatedState`, derives reuse today's derived cells over +it). Not started in the rebuild. +::: diff --git a/packages/chord/docs/next/explainer/src/09-natives.md b/packages/chord/docs/next/explainer/src/09-natives.md new file mode 100644 index 000000000..ed7dacbbf --- /dev/null +++ b/packages/chord/docs/next/explainer/src/09-natives.md @@ -0,0 +1,160 @@ + +# Things the OS Gives You {#natives lede="Start data, host-lifetime objects and natives: how a plugin opens a file watcher or a socket so that it is closed exactly once, also across reloads."} + +Studio's notes get an inbox: a text file dropped into the session's `inbox/` directory becomes a note. That needs a +file watcher, a **native**: an OS-level value (a file handle, a socket, a process, a database connection) that cannot +be JSON, cannot be copied, must be opened, and must be closed exactly once. It also needs the session's data +directory, which the plugin does not know. This chapter covers both. + +## Start data is an object + +A worker knows things before any plugin loads: its session id, the data directory, init data from the server's start +message. There is no special mechanism for them. Start data is an ordinary `local` object with `lifetime: "host"` and +constant state, served by base code's core facet: + +```ts +// studio/src/core/contract.ts +export const Init = obj({ id: str(), dataDir: str(), cwd: str(), env: opt(str()) }); +export const SessionInfo = local("studio.session-info", { + version: 1, + input: obj({}), + members: { init: Init }, +}); + +// studio/src/core/worker.ts +export const coreWorkerFacet = (init: Static) => defineFacet((host) => { + host.serve(SessionInfo, () => ({ init: constant(init) }), { lifetime: "host" }); + // … the Harness and other core objects (chapter XIV) +}); +``` + +Any plugin on the worker declares a reference and reads it **after activation**: in an opener, a derive, a routine or +a method, never synchronously in the factory. + +```ts +const info = self.get(SessionInfo, {}); +// later: info.init.value.id, info.init.value.dataDir +``` + +Paths are derived by the plugin from generic start data (`join(dataDir, id, "notes-inbox")`), never provided per +plugin. Platform bindings work the same way: on a Durable Object, a core object's `self.open` returns the binding +(`state`, `env`) and its close does nothing. + +## `self.open` + +```ts +const watcher = self.open( + async ({ signal }) => watchDir(path, { signal }), // open: during activation + (w, reason) => w.close(), // close: when the object ends or restarts + { closeTimeout: 10_000 }, // optional; host default 30 s +); +watcher.value; // the watcher; reading it before activation throws not_ready +``` + +`self.open` returns a handle with `.value` (type `Native`), not a proxy. The rules: + +- **Natives live only in objects.** Not at module level, not in the facet. A module-level watcher would survive a + reload with nobody to close it. +- **Order:** natives open one after another in declaration order (a later opener may read an earlier handle's + `.value`) and close in reverse. The object goes live after all opens succeeded. +- **The value never changes during an attempt.** No replacement, no versions, no stale-handle logic. To change a live + native, the object restarts. +- **A failed open** makes the object `failed`: transient with backoff by default, broken if the opener says so + (`throw broken(error)`). A partly failed activation closes the natives already open, with reason `end`; the next + attempt opens all of them anew. +- **`close` gets a reason:** `{ kind: "reload" }` (the plugin reloads and this native object restarts), `{ kind: + "stop" }` (the host shuts down) or `{ kind: "end" }` (the last reference ended, or the type was removed). Each + `close` is bounded by `closeTimeout`, then reported as a possible leak while the rest continues. A throwing `close` + does not stop the others; `host.close()` collects the errors, background ends report to `onError`. +- **Behaviour around a native goes into routines** (chapter X). Listeners on natives outside routines are forbidden. + +An object with at least one `self.open` is a **native object**; one without is a **pure object**. The difference +decides what a reload does (chapter XV): pure objects are swapped in place, native objects restart. + +## The inbox + +The inbox must run as long as the worker, whether anyone looks at the notes or not. So it is a `lifetime: "host"` +object. The facet cannot `self.get` a type it serves, and the inbox has no meaningful input, so it is an object +without input that reads its session from the start data: + +```ts +// notes/contract.ts +export const NoteInbox = local("notes.inbox", { version: 1, input: obj({}), members: {} }); + +// notes/worker.ts +host.serve(NoteInbox, (_input, self) => { + const info = self.get(SessionInfo, {}); + const watcher = self.open(async ({ signal }) => { + const { id, dataDir } = info.init.value; // read in the opener: after activation + return watchDir(join(dataDir, id, "notes-inbox"), { signal }); + }, (w) => w.close()); + + self.routines.loop("import", async (ctx) => { // chapter X + const store = await ctx.get(NoteStore, { session: info.init.value.id }); + for await (const file of watcher.value.files(ctx.signal)) { + await store.add({ text: await readFile(file, "utf8"), by: "inbox" }, ctx); + await rm(file); + } + }); + return {}; +}, { lifetime: "host" }); +``` + +Note `ctx.get(NoteStore, …)` in the routine rather than `self.get` in the factory. The factory runs before the +session id is known (start data is read after activation), so a host-lifetime object without input gets +session-scoped objects with `ctx.get` in a routine or method (R22). A `get` whose input is computed after activation is +to be added only if a real case needs it. + +## Natives with work in flight + +A restart is sequential (R16): close the old native, then open the new one. When the old one still has work running, +use one of two patterns: + +- **Drain in the background.** `close` detaches the old resource and returns at once; it disposes of it when its work + finishes, or after `closeTimeout` at most. Chord opens the new one immediately. Code mode's sandbox does this: a + script running during a reload finishes on the old sandbox (for up to 30 s, Q7; a longer one is aborted and its turn + fails), new scripts start on the new one. +- **Swap through a registry.** The resource is a registry contribution (chapter XIII). Readers always look up the + current one at use; a reader that must wait for one uses a cancellable "wait until present" with the park deadline. + +Chord aborts only the object's own calls and jobs on close. Other work that uses the native (Durable tool calls on a +sandbox) is the owner's business: `close` may drain instead of tearing down. What an old native does after its owner +restarts is also the owner's `close`: a model runtime keeps working until collected, a closed SSH connection fails its +in-flight `exec`. + +## Natives that follow a relocation + +A session can move to another directory. The inbox watcher must follow. Restarting the object on every move would be +heavy-handed; instead the native is a stable manager opened once, and a routine on the session location switches its +internal resource and bumps a generation, so late events from the old location are dropped: + +```ts +const watcher = self.open(async ({ signal }) => new SwitchableWatcher({ signal }), (w) => w.close()); +self.routines.on(location.cwd, (cwd) => watcher.value.switchTo(join(cwd, ".notes-inbox"))); +``` + +## Call-scoped natives + +A native that lives exactly as long as one call (a temporary SSH session for a directory listing, a file handle for a +download) is opened and closed inside the method with `try/finally`. There is no `ctx.open`: + +```ts +list: async ({ path }, ctx) => { + const session = await ssh.open(config, ctx.signal); + try { return await session.readDir(path, ctx.signal); } + finally { session.close(); } +}, +``` + +## Core vs. plugin + +`SessionInfo` comes from base code, `NoteInbox` from a plugin; both use the same primitives. Where code lives decides +which it is. **Base code** (the app's `src/`, imported statically by the start code under a fixed name such as +`@studio/core`) is always there, not switchable, and a change restarts the hosts. **A plugin** (its own directory, +found by listing, named by its `package.json`) is switchable and reloads. Core is what the app always needs and +everything builds on: the harness, the session list, access, the session location. Optional features are plugins; +code mode is a plugin. + +:::note | Built so far +Natives and host-lifetime objects are layer L4. `constant` is decided (R23), not built. +::: diff --git a/packages/chord/docs/next/explainer/src/10-routines.md b/packages/chord/docs/next/explainer/src/10-routines.md new file mode 100644 index 000000000..4191f21d0 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/10-routines.md @@ -0,0 +1,145 @@ + +# Standing Behaviour {#routines lede="Routines, reconciling instead of notifying, and how two owners keep a rule between them without a shared transaction."} + +Notes older than 30 days should go every hour. Inbox files should be imported as they arrive. Notes of an archived +session should become read-only. Each of these is behaviour that stands, not a call someone makes. Written as bare +`setInterval`s and event listeners, it would keep running old code after a reload and touch state from nowhere. Chord +has one declared form for it: the **routine**. + +## Four kinds + +```ts +self.routines.every("cleanup", HOUR, (ctx) => dropOlderThan(30 * DAY, ctx)); // due time survives reloads +self.routines.on(settings, async (value, ctx) => applyLimit(value, ctx)); // a readable or event source +self.routines.loop("import", async (ctx) => { // runs until the object ends + while (!ctx.signal.aborted) await importNext(ctx); +}); +self.routines.custom("ticker", ({ signal, run }) => { // anything else + const timer = setInterval(() => run(tick), 1000); + return () => clearInterval(timer); // cleanup undoes the setup +}); +``` + +Routines start when the object goes live, stop when it ends, and restart with the new code on every reload of their +plugin. Each run gets its own `ctx`; errors are caught and reported, never uncaught. Routines are named (`on` +optionally by its source) so Chord recognizes them across reloads; that is how `every` keeps its due time. + +## Serialized per routine + +The new instance of a routine starts only after the old handler in flight has settled; only that routine waits, for at +most the routine handover timeout (10 s), after which it starts anyway and the stuck handler is reported. + +The reason, as a trace: a supervisor routine reconciles pending worker restarts. The old handler is midway through +`restart(W1)` when the plugin reloads. Without serialization the new routine starts at once, sees W1 still pending, and +restarts it a second time. + +## Reconcile, don't track + +A routine **reconciles from the current value**: it makes the world match what it sees now. It writes no "in +progress" flags, and it starts a job for long work (chapter XI). A handler that only cares about the latest value +compares with the current value itself; there is no `overlap` option. Subscribing to the object's own documents is +gapless: every commit is seen. + +No bare timers or callbacks that touch state outside routines, jobs or calls. That is what lets Chord stop all of an +object's behaviour when it ends or reloads. + +## Rules between two owners + +Every thing has exactly one owner, the smallest object it belongs to. Some rules span two: "notes of an archived +session are read-only" involves the catalog (which knows archiving) and `NoteStore` (which holds notes). Two tempting +designs are wrong: + +- **A cross-owner transaction** (archive and lock the notes atomically) would make the catalog know every plugin and + couple their storage. +- **A notification** (`catalog` calls `notes.archive(session)`) can be lost: the worker is not running, the link drops, + the process crashes between the archive and the call. + +The design's rule: **the owner of the dependent data enforces the rule** by reconciling from the other owner's state +with an idempotent routine. + +```ts +// NoteStore: an archived session's notes are read-only +const flags = self.state(NoteFlags); // { archived: boolean }, NoteStore's own document +const catalog = self.get(Catalog, {}); // the server's session list: another owner +self.routines.on(catalog.sessions, async (sessions, ctx) => { + const archived = sessions[session]?.archived ?? false; + if (flags.value.archived !== archived) await flags.change((d) => { d.archived = archived; }, ctx); +}); +// add and remove refuse while flags.value.archived +``` + +After a crash the routine runs again with the current value. The catalog does not know the notes plugin exists. The +same pattern closes a forward of an archived session in the ports plugin, aborts a job whose grant was revoked +(chapter VII), and keeps any invariant between plugins. + +## Reading without starting a worker + +The session list in the sidebar shows each session's note count. Getting `NoteStore { S1 }` for every session would +start every worker. The answer is plain publishing between two objects of one plugin, no Chord mechanism: the worker +side reports a summary to a server-side object by a method call from a routine, and clients read the server object. + +```ts +// notes/contract.ts +export const NoteCounts = remote("notes.counts", { + version: 1, input: obj({}), placement: () => Server(), + variants: { + owner: { state: { counts: map(int()) } }, + worker: { methods: { report: method(obj({ session: str(), count: int() })) } }, + }, +}); + +// notes/server.ts +const Counts = defineState("notes.counts", { + schema: obj({ bySession: map(obj({ count: int(), incarnation: str() })) }), + version: 1, initial: () => ({ bySession: {} }), persist: true, // survives a server restart + checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64, +}); + +host.serve(NoteCountStore, (_input, self) => { // local, lifetime "host" + const counts = self.state(Counts); + return { + counts: self.derive("counts", [counts], ([c]) => mapValues(c.bySession, (e) => e.count)), + report: ({ session, count, incarnation }, ctx) => + counts.change((d) => { + const last = d.bySession[session]; + if (last && last.incarnation > incarnation) return; // uuidv7: an older worker, ignore + d.bySession[session] = { count, incarnation }; + }, ctx), + }; +}, { lifetime: "host" }); + +host.serve(NoteCounts, (_input, principal, self) => { + const store = self.get(NoteCountStore, {}); + if (principal.kind === "worker") return { + variant: "worker", + report: ({ session, count }, ctx) => { + if (session !== principal.session) throw forbidden(); // a worker reports only itself + return store.report({ session, count, incarnation: ctx.incarnation }, ctx); + }, + }; + return { variant: "owner", counts: store.counts }; +}); + +// notes/worker.ts, inside NoteStore +const counts = self.get(NoteCounts, {}); // a reference to the server +self.routines.on(count, (n, ctx) => inVariant(counts, "worker") && counts.report({ session, count: n }, ctx)); +``` + +The server object keeps the latest value per session (in a document if it must survive a server restart) and +**ignores reports from an older worker incarnation** (incarnations are uuidv7, so they order by start time). A worker that was replaced may still deliver a late report; the +fence is the caller's incarnation, which Chord stamps on every call from the link handshake (R20): `ctx.incarnation`, +next to `ctx.principal`, not part of the principal or its key, never sent as data, so it cannot be forged. Session +status, subagent lists, unread marks and artifact lists in p2 all follow this pattern. The store is local and has no +caller incarnation of its own, so the view passes it on as data. + +:::review | Review: `ctx.incarnation` across a relay (Q13, open) +A worker's call to the server arrives directly, so `ctx.incarnation` is the worker's. But a call relayed through the +server (a tab's call to a worker) carries the server's incarnation on the worker's side, because the worker's +handshake is with the server. Review round 3 proposes that relays stamp the origin host's incarnation, as they stamp +the principal. Undecided; the pattern above only needs the direct case. +::: + +:::note | Built so far +Routines are layer L8. The design treats `every` as one of four forms over one runner; review round 3 proposed cutting +to `on` and `loop` only (C4), not decided. +::: diff --git a/packages/chord/docs/next/explainer/src/11-jobs.md b/packages/chord/docs/next/explainer/src/11-jobs.md new file mode 100644 index 000000000..e506c05df --- /dev/null +++ b/packages/chord/docs/next/explainer/src/11-jobs.md @@ -0,0 +1,171 @@ + +# Work That Outlives the Call {#jobs lede="Jobs: owned work started by a method that keeps running after its caller leaves, with cancel, dedupe and a recorded outcome."} + +Importing a 50 MB text file into notes takes a minute. The tab that started it may close; the import must go on, be +cancellable, not run twice when the tab retries after a link blip, and its result must be recorded. A method cannot do +this (it dies with its caller), a routine is standing behaviour rather than a request, and a Durable task is for work +that must resume after a crash. Chord's form for it is the **job**, built in pi-durable's sandwich: **intent** (a +transaction), **effect** (anything), **outcome** (a transaction in the current code). + +## Defining and running a job + +```ts +// notes/worker.ts +const ImportJob = defineJob("notes.import", { + version: 1, + args: obj({ path: str() }), + progress: obj({ lines: int() }), + result: obj({ added: int() }), + concurrency: 1, // one import at a time per NoteStore +}); + +const ImportResults = defineState("notes.import-result", { + schema: obj({ outcome: union([obj({ added: int() }), obj({ error: str() })]) }), + version: 1, initial: () => ({ outcome: { added: 0 } }), keyed: true, + checkpointWhen: () => true, +}); + +host.serve(NoteStore, ({ session }, self) => { + const list = self.state(NoteList); + const results = self.state(ImportResults); + const imports = self.job(ImportJob, { + run: async ({ path }, ctx) => { // 2. effect + let lines = 0; + for await (const line of readLines(path, ctx.signal)) { + await addNote({ id: `${ctx.jobId}:${lines}`, text: line, by: "import" }, ctx); // keyed by job id + ctx.progress({ lines: ++lines }); + } + return { added: lines }; + }, + finish: async (outcome, tx) => { // 3. outcome, current code + (await tx.draft(results, outcome.id)).outcome = + outcome.kind === "done" ? outcome.result : { error: outcome.error }; + }, + }); + return { + …, + imports: imports.running, // live list of runs + startImport: ({ path, requestId }, ctx) => + imports.start({ path }, ctx, { key: requestId }), // 1. intent (refined below) + abortImport: ({ id }) => imports.abort(id), + }; +}); +``` + +`defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })`; `progress` and the outcome +are typed by the schemas. + +:::sequence animate | The job sandwich. The intent commits before anything runs; the effect runs outside any transaction; the outcome commits in the current code. +actor tab "tab" +actor store "NoteStore { S1 }" accent +actor run "run" "effect" blue +tab -> store: startImport({ path, requestId }) +store => store: 1. intent: busy check, dedupe by key, commit +store --> tab: { id } +store -> run: start after the commit +run -> run: read lines, add notes (keyed by jobId) +run --> store: progress { lines } (throttled) +note tab..store: the tab may close here: the job counts as a reference +run --> store: settles: { kind: "done", result } +store => store: 3. record outcome, finish(outcome, tx), ack +::: + +## The three steps + +**1. Intent.** `job.start(args, ctx, { key? })` is shorthand for `tx.start(job, args, { key? })` inside a +transaction, which may contain other writes. The busy check (`concurrency`) and deduplication (the same key returns the +existing run) are atomic and in the same commit; `run` starts only after the commit. Run ids are uuidv7. + +**2. Effect.** `run(args, ctx)` may do anything: processes, network, other objects. It writes no document itself; it +returns the outcome. External writes carry a dedupe key derived from `ctx.jobId` (above: note ids), so an interrupted +attempt cannot apply them twice. On a reload `run` keeps its old code. + +**3. Outcome.** When `run` settles, Chord records the outcome in its own record, then calls the **current** code's +`finish(outcome, tx)` and acknowledges the run in the same commit. The outcome is tagged: + +```ts +type Outcome = { id: string } & ({ kind: "done"; result: R } | { kind: "failed"; error: "threw" | "aborted" | "lost" }); +``` + +A throwing `finish` changes nothing and is retried with a backoff (1 s to 60 s), and after the next reload. `finish` +is optional and follows Durable's transaction rule. + +## Dedupe ends with the run + +Deduplication covers **unsettled runs only** (Q9). Once `finish` has committed, Chord drops the run's record and key +and releases its waiters' references; it keeps nothing beyond a run's lifetime. Dedupe after completion is the +owner's. Studio records which run a request started, in the same commit as the intent, and `finish` records the +result by run id (`ImportResults` above): + +```ts +const Requests = defineState("notes.import-request", { + schema: obj({ run: str() }), version: 1, initial: () => ({ run: "" }), keyed: true, persist: true, + checkpointWhen: () => true, +}); + +startImport: async ({ path, requestId }, ctx) => { + const prior = await requests.get(requestId); // read outside the transaction (R13) + if (prior) return { id: prior.run }; // started before, maybe finished long ago + return self.tx(async (tx) => { + const run = await tx.start(imports, { path }, { key: requestId }); // dedupes a concurrent retry + (await tx.draft(requests, requestId, { tags: [SessionTag(session)] })).run = run.id; + return { id: run.id }; + }, ctx); +}, +``` + +Two concurrent retries both miss `prior`, but the key makes `tx.start` return the same unsettled run. The client +watches `imports` for progress and reads the result by run id once it is gone from `running`. + +Why Chord does not keep completed runs itself: either 100,000 keyed imports would stay in memory forever, or a +retention limit would let a retried key rerun its effect. The owner knows which results matter. + +## Watching, waiting, aborting + +- **`running`** is a readable of `{ id, version, args, progress, phase }`. `phase` is `running`, `finishing` (outcome + recorded, `finish` not yet committed) or `blocked` (with the reason). `concurrency` counts only active runs. The + panel shows a progress bar from it. +- **Progress** is in memory, throttled and sent as deltas. For output, use `OutputBuffer` and `Progress` from + `@earendil-works/chord/output`: bounded windows, adaptive throttling, no write amplification. +- **Waiting:** `start` returns a run with `outcome(ctx)`. The caller's abort ends only its wait, never the run. +- **Abort is a request:** `job.abort(id)` fires the run's `ctx.signal`; the run decides what it returns (a result + that already arrived stays a success). A run that ignores the signal keeps its slot until it ends. `finish` runs in + Chord's own context. + +## Lifetime, persistence, revocation + +**Jobs keep their owner alive.** A running, finishing or blocked job counts as a reference to its object, which ends +only once all its jobs are settled. Starting an import and closing the tab does not abort it. Unsettled runs also +count as activity for the worker's keep-alive. + +**Revocation takes precedence** (R11). Revoking a principal ends its objects at once, whatever jobs remain; the jobs +get the abort signal and the normal close timeout. A job runs as its object's principal. + +**Persistence:** jobs are in memory by default and gone with the process. `persist: true` (only for stable +identities) keeps the record: a run live at a crash ends `lost` at the next activation and `finish` records it. +Nothing is rerun automatically. + +**Versions:** a run started under an older job version keeps its version; `finish` and `running` see it through pure +`accepts` adapters, and stored outcomes are never rewritten. A version bump without an adapter does not refuse the +reload: the old outcome is `blocked`, kept and reported until a later reload brings the adapter. + +**Native objects:** a job in a native object ends `aborted` when the object restarts on a reload (chapter XV). In a +pure object it keeps running on its old `run` code. + +## What is not a job + +| Work | Use | +| --- | --- | +| request and response that may die with its caller | a method | +| input into running work (a pasted code, an approval) | a flow (chapter XII) | +| work that must resume after a crash | a Durable task | +| standing behaviour | a routine | + +:::review | Review: jobs are the largest open simplification (#9, C3) +Decision #9 (deferred until login is ported): a login or a long relocation is a job whose state consumers want to +address and watch (`LoginFlow { id }`). Options: (a) jobs stay records inside their owner, watched through `running`; +(b) a job is an object of its own, addressed by its id, living until done. Separately, review round 3 proposed cut C3: +shrink jobs to a runner over documents (no job versions, `accepts`, `blocked`, `persist`) or move them out of the +kernel. Not decided. `chord-next-walkthrough.md` concept 17 still shows the older outcome shape `{ ok: true, result }`; +the design tags outcomes by `kind`. +::: diff --git a/packages/chord/docs/next/explainer/src/12-flows.md b/packages/chord/docs/next/explainer/src/12-flows.md new file mode 100644 index 000000000..cf6a986c1 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/12-flows.md @@ -0,0 +1,141 @@ + +# Asking a Person {#flows lede="Flows and the Questions service: how running work waits for an answer from someone who may be on any tab, may reconnect, and may answer twice."} + +A provider login asks for a pasted code, then an account choice. A dangerous tool call waits for the owner's +approval. The agent asks everyone present a question. In each case running work waits for a person, who may be on any +tab, may lose the connection halfway, may answer late or twice. Two patterns cover this: a **flow** kept as its +owner's state, and the **Questions** service, a shared store for questions. + +## A flow is state, not a call + +The tempting design is a long-held server call, `step()`, that blocks until the next step. A link drop fails it, and +the consumer cannot tell whether it missed a step. Chord's answer is that the owner keeps the flow as **state**, which +re-holds by itself after any drop: + +```ts +// models/contract.ts (sketch from the design) +start: method(obj({ provider: str() }), obj({ id: str() })) // uuidv7 +answer: method(obj({ id: str(), step: int(), value: str() }), Answered) // ok | "step no longer active" +cancel: method(obj({ id: str() })) + +// LoginFlow { id }: state +type FlowState = + | { kind: "running"; step?: { id: number; prompt: LoginPrompt }; notices: Notice[] } + | { kind: "done" | "failed" | "cancelled"; notices: Notice[] }; +``` + +The UI watches the state and renders the current step. A late or second answer to a step that is gone is refused. A +promise-based callback inside the work (pi-ai's `prompt` during a login) stays **inside** the owner: its job keeps the +resolver, and nothing crosses an object boundary. + +Imperative consumers (tests, scripts, a CLI, an agent tool) use a generic client helper over the same state, not a +second protocol: + +```ts +async function* steps(flow: LoginFlowRef, ctx: Ctx) { + let last = 0; + for (;;) { + const s = await until(flow.state, (v) => v.kind !== "running" || (v.step !== undefined && v.step.id > last), ctx); + if (s.kind !== "running") return s.kind; + last = s.step!.id; + yield s.step!; + } +} + +for await (const step of steps(await ctx.get(LoginFlow, { id }), ctx)) + await login.answer({ id, step: step.id, value: answerFor(step) }, ctx); +``` + +`last` keeps steps from being lost or repeated across re-holds. + +:::review | Review: where a flow lives (#11, deferred) +Decision #11, deferred until login is ported: (a) a keyed document of its owner, or (b) an object `LoginFlow { id }` +living until done; tied to #9 (jobs as objects). The `LoginFlow { id }` address above assumes (b). Also undecided is +how "leaving" ends a login (p2 cancels when the starting owner leaves everywhere); with presence it is a dropped +reference, never an unmount (chapter VIII). +::: + +## The Questions service + +The Questions service (`apps/p2/docs/questions.md`) is a p2 design, not a Chord feature, built from Chord's +primitives: a shared store with per-principal views. The asker gets `ask`; viewers get the open questions and +`answer`. + +```ts +export type Question = + | { kind: "text"; message: string; secret?: boolean; help?: string } + | { kind: "select"; message: string; options: readonly { id: string; label: string }[] } + | { kind: "confirm"; message: string; allow: string; deny: string }; + +type Policy = { mode: "first" } | { mode: "quorum"; count: number } | { mode: "all"; join?: "live" | "snapshot" }; + +ask(question, { + audience, // a readable or a list of principals + policy, + key?, // stable per asking flow: a re-ask reattaches to the pending question + deadline?, // then resolve with what is there; `missing` names who did not answer +}, ctx): Promise; +``` + +Its rules: + +- **The store is the truth.** Pending questions and their resolvers live in the store; `open` is a readable over it, + filtered per viewer. Nothing is a document, so a reload or migration cannot leave a ghost question. +- **One settlement point.** Register the waiter and the abort listener before publishing the question; an answer, an + abort or the deadline atomically takes the entry out before resolving. A late or second answer gets "no longer + active". Answers are validated against the stored question (a select id must be one of the options). Never await an + answer inside a transaction. +- **Policies over a live audience.** `first` resolves with the first answer; `quorum(n)` after n answers; `all` when + every member of the audience has answered. With `join: "live"` (the default), someone who joins becomes required and + someone who leaves drops out. +- **Authorization.** The asker is stamped by the host (a worker asks only in its own session's scope; secret questions + only from server askers). Each question stores who may answer it; a guest's send access does not imply approval + rights. Secret answers never appear in `open`, documents or logs. +- **Link drops and crashes.** A rejected `ask` is not a denial. Askers retry with the same `key` until their own signal + aborts; a pending `ask` counts as an active call for keep-alive. +- **Where it lives.** On the host where the asking work runs: session questions on the session's worker, login + questions on the server. UIs reach both through the usual relay. + +The store in Chord v2 terms: + +```ts +host.serve(QuestionStore, ({ scope }, self) => { + const pending = self.open(async () => new PendingQuestions(), (p) => p.rejectAll("store closed")); + const open = self.derive("open", [pending.value.readable], ([all]) => all); // data only, no resolvers + return { open, ask: (args, ctx) => pending.value.ask(args, ctx), answer: (args, ctx) => pending.value.answer(args, ctx) }; +}); + +host.serve(Questions, ({ scope }, principal, self) => { + const store = self.get(QuestionStore, { scope }); + if (isHost(principal)) return { variant: "asker", ask: (args, ctx) => store.ask({ ...args, asker: principal }, ctx) }; + if (!mayView(principal, scope)) throw forbidden(); + return { + variant: "viewer", + open: self.derive("mine", [store.open], ([all]) => openFor(all, principal)), + answer: ({ id, value }, ctx) => store.answer({ id, value, by: principal }, ctx), + }; +}); +``` + +The pending map with its resolvers is in-process state with promises, so it is a native of the store (opened and +closed with it). Its `close` rejects every waiter; askers retry with their key against the new store after a reload. + +:::review | Review: questions.md predates the v2 rules +`apps/p2/docs/questions.md` writes the store as `self.resource(...)` with a `replace` hand-over, which the design +rejected ("resources replaced in place"), and opens Chord call scopes in Durable hooks with `host.root`, which Q6 +replaced with `chord(fn, context)` (chapter XIV). Design §20 says the Questions store "must live in an object (it +predates natives only in objects)". The sketch above is this guide's translation, with the derive per viewer; filtering +per viewer in the store, once per variant, may be cheaper for large audiences. The `ask` signature passes typed +objects as arguments, which is fine (data), but the `audience` readable form cannot cross a link: across links, +audience must be data (principals) or computed by the store. +::: + +## Choosing + +| Case | Use | +| --- | --- | +| a multi-step login in one owner | a flow: the owner's job keeps the resolver, state shows the steps | +| a tool needs the owner's approval | Questions, `first`, audience the owner | +| the agent asks everyone present | Questions, `all`, audience from presence | +| deploy needs 2 of 3 approvals | Questions, `quorum(2)` | +| a notice (auth URL, device code, progress) | not a question: the asking work's own progress state | diff --git a/packages/chord/docs/next/explainer/src/13-registries.md b/packages/chord/docs/next/explainer/src/13-registries.md new file mode 100644 index 000000000..22efc7019 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/13-registries.md @@ -0,0 +1,185 @@ + +# Extension Points {#registries lede="The facet, registries and the HTTP component: how plugins add behaviour to each other without callbacks, and how the owner stays in charge."} + +The notes plugin wants a download route, `GET /s/:session/notes.txt`. A logs plugin wants to time every request. +Several plugins add model providers, tools for the agent, commands for the TUI. Someone must combine these +contributions, decide conflicts, and replace one plugin's part as a whole when it reloads. Between plugins exactly two +things pass: object types and **registries**. + +## The facet, completed + +The facet has appeared since chapter II as the place that serves types. It is also itself a **pure object** with the +host's lifetime: + +```ts +// notes/server.ts +export default defineFacet((host, self) => { + const prefs = self.state(NotesPrefs); // plugin-wide documents + self.contribute(Http, { routes: { "GET /s/:session/notes.txt": exportNotes } }); + host.serve(NoteCountStore, …, { lifetime: "host" }); + host.serve(NoteCounts, …); +}); +``` + +- It may hold documents, references, routines and contributions; **no natives** (those need an object with its own + lifetime). +- Its identity is the plugin's name, so its persisted documents survive restarts and reloads. +- Served objects go live after the facet and may read and write its documents (chapter V). +- `host.serve` and `self.contribute` are valid only while the setup runs. The facet never `self.get`s a type it + serves. + +## Registries + +A registry is an extension point owned by someone (a Chord component, the app, a plugin). Each contributor +contributes **at most one value** per registry; the table is an immutable, ordered list of `(contributor, value)`; +the registry's `compose` turns it into the value consumers use. + +```ts +// owned by the HTTP component +export const Http = defineRegistry("chord.http", { + version: 2, + accepts: { 1: (old: HttpContributionV1): HttpContribution => upgrade(old) }, + compose: (table, { report }) => buildRouter(table, report), // pure; always delivers a value +}); + +self.contribute(Http, { routes: { … }, wraps: [ … ] }); // a contributor +const router = host.registry(Http).value; // a consumer: live +``` + +**`compose` runs at runtime, whenever the table changes:** after a reload's commit, when a contributor goes live or +ends, when a contribution changes. It is not a staging check: contributions can be dynamic (known only once a native +is open), so no check before a reload could be complete. `compose` always delivers a value with its fallback built in +(e.g. the first by plugin name wins) and reports conflicts through `report`, which become owner notices and telemetry +events. Readers get a plain value and never handle conflicts. Only if `compose` throws is the registry in an error +state, and readers get an error when they use it. Work that started earlier keeps the value it read. + +**Rules:** + +- **Who contributes** (R21): only facets and `lifetime: "host"` objects; only their `self` has `contribute` (a type + error elsewhere). Every object may read registries. +- **A contribution is in the table exactly while its contributor is live:** a facet's while its plugin is loaded; an + object's from `live` until it ends, fails or restarts. +- **Order** is sorted by plugin name: deterministic on every host and across reloads, and deliberately meaningless. + Meaningful order comes from the owner's composition (HTTP phases, below). Helpers: `keyed(table, entriesOf)`, + `ordered(table, itemsOf, orderOf?)`. +- **One authoritative declaration** (Q8): only the defining plugin's facet declares the registry to the host + (`compose`, version, `accepts`). Other plugins' tokens only identify it by name and types. A reload of the defining + plugin replaces the declaration atomically and the table recomposes. If the defining plugin is not loaded, the + registry has no value (readers get an error like a failed reference) and contributions wait in the table. +- **Versions** (Q8): a contribution carries the registry version its plugin was built against; older ones are adapted + by the defining facet's pure `accepts`. One without a matching version or adapter is left out and reported naming + plugin, registry and versions; its plugin keeps working. Contributions are in-process values (often functions) and + are not schema-checked. +- **Use sites guard contributions.** A contribution is untrusted code (third-party plugins are a goal). Every use site + catches throws and rejections when calling an entry, reports naming the contributor, and continues with the others. +- **The owner decides** conflicts, order and selection. Registries add behaviour; they never replace an owner's checks. + +## Natives through registries + +Registries are the only way a native reaches another plugin. An entry lives exactly as long as its contributor; +consumers look up at each use (or follow the composed value) and keep nothing from an entry beyond a use. The models +plugin's host-lifetime object contributes its runtime; the harness looks it up at every request. After a models reload +the next request gets the new runtime, while a running stream finishes on the old one. + +A handler that stores `const r = entry()` in a module variable and uses it for days has a bug: after the contributor +restarts it holds a closed native. + +## Contributions that change + +Some contributions are known only at runtime: MCP tools appear when a server connects. A contribution is either fixed +for its contributor's life (dynamic parts behind handles read at use) or a **readable**, which Chord subscribes to; +on every change Chord replaces that entry and runs `compose` again. For in-process values, `self.writable(initial)` is +a host-local readable of any value (no JSON, no wire, compared by identity) with `set`: + +```ts +const ext = self.writable(mcpExtension([])); // a Durable Extension, no tools yet +self.contribute(HarnessExtensions, ext); +self.routines.on(mcp.tools, (tools) => ext.set(mcpExtension(tools))); +``` + +There is no remove call. An entry leaves when its contributor ends, or after a reload whose new code no longer +contributes. A live contributor with nothing to offer sets its readable to the registry's empty value (an extension +without tools, empty routes); the table keeps exactly one entry per live contributor. + +## Optional dependencies + +Registries also replace optional references (chapter VI). Radius forwards connections to ports the ports plugin +manages, but ports is optional. So radius owns `radius.forwards` and ports contributes its forwards; without ports the +table is empty and radius works. + +## The HTTP component + +HTTP is an optional Chord component (`@earendil-works/chord/next/http`) built on a registry. A route gets a plain +request summary (method, path params, query, principal, provenance such as local or relayed, selected headers, parsed +and capped JSON bodies, uploads in chunks) and a `ctx` whose `principal` is the request's, so `ctx.get(Notes, …)` gives +exactly the view that person may see. The route returns a description; the component does the HTTP work once (ranges +with 206 and 416, HEAD, download headers with a safe filename, `nosniff`, CSP and its nonce, caching, streaming, abort +on disconnect): + +```ts +type HttpResponse = + | { kind: "json"; value: JsonValue } | { kind: "page"; title: string; body: string } + | { kind: "bytes"; name: string; mime: string; size: number; read(offset: number, length: number): Promise } + | { kind: "redirect"; to: string } | { kind: "error"; status: number }; + +// notes/server.ts +self.contribute(Http, { + routes: { + "GET /s/:session/notes.txt": async (req, ctx) => { + const notes = await ctx.get(Notes, { session: req.params.session }); // the requester's view + const text = new TextEncoder().encode(render(notes.notes.value)); + return { kind: "bytes", name: "notes.txt", mime: "text/plain", size: text.byteLength, + read: async (offset, length) => text.subarray(offset, offset + length) }; + }, + }, +}); +``` + +Routes compose with `keyed`: two contributors with one pattern is a reported conflict, and the first by plugin name +wins. **Wraps** are middleware in phases the owner defines (`observe`, `headers`, `transform`, outermost first), +within a phase in contributor order. No numeric priorities: plugins have no single registration order, so Fastify's +lifecycle phases and encapsulation are the model, not Express. + +```ts +// logs/server.ts +self.contribute(Http, { + wraps: [{ + name: "access-log", phase: "observe", scope: "all", // default scope "own": only own routes + wrap: async (req, next, ctx) => { + const t = Date.now(); + const res = await next(req); + log(req, res, Date.now() - t); + return res; + }, + }], +}); +``` + +`route` narrows a wrap to one route; a `route` that does not exist is a reported conflict. Wraps may add a security +boundary on top of an owner's checks, never instead of them. A request uses the router as it was when it started. +Request resolvers (who sent a request: cookie, loopback, relay) are hooks over the summary. + +## Channels and big bytes + +**Channels** are WebSocket byte channels (`send`, `onData`, `close`) for terminals, separate from object links and +with their own acknowledged windows, so a large state message never holds up keystrokes. + +**Big transfers are chunked by the owner,** not by Chord: bytes are a schema type (`bytes()`), carried natively +(never base64), and a 50 MB upload is a protocol of plain calls: + +```ts +beginUpload({ name, size }, ctx) → { upload } +append({ upload, offset, bytes }, ctx) // about 1 MiB per chunk; the awaited call is the backpressure +finishUpload({ upload }, ctx) → { id } +read({ id, offset, length }, ctx) → bytes +``` + +After a link drop the `append` in flight fails like any call; the writer asks the owner for the stored offset and +resumes. Unfinished uploads expire through the owner's routine. Chord has no stream type: it would need its own rules +for drops, cancellation, relays and restarts, which this offset protocol already survives. + +:::note | Built so far +Registries are layer L9. The HTTP component, channels and the chunked-transfer helper are deferred until after the +kernel and a first real proof (the harness, one tool with an approval pause, a code mode reload); until then the +playground serves its pages with plain app code. Their design stays the target. +::: diff --git a/packages/chord/docs/next/explainer/src/14-agent.md b/packages/chord/docs/next/explainer/src/14-agent.md new file mode 100644 index 000000000..40b69b555 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/14-agent.md @@ -0,0 +1,192 @@ + +# The Agent {#agent lede="pi-durable inside Chord: the harness as a core object, models, environments and tools through registries, and how a tool reaches Chord objects and people."} + +Each Studio worker runs a pi-durable harness for its session. The harness must run for the worker's whole life, +whether anyone watches or not. Its models come from the models plugin, its execution environments from the envs +plugin, its tools from every plugin, and all of those reload independently. This chapter assumes you know pi-durable +and shows only how it plugs into Chord. + +## Shared foundations + +Durable and Chord v2 share modules: Durable imports Chord's `context`, `delta`, `json`, `output` and `sqlite`, and v2 +imports the same, so `Context`, `Draft` and `JsonValue` are one type each. A v2 readable is today's +`replicatedState` (`chord/src/state.ts`), so Durable's `viewState()` and document watches can be exposed as Chord state +members without conversion. Only plain data crosses between Durable and Chord objects. + +## The harness is a core object + +```ts +// studio/src/core/harness.ts (base code) +import { Harness as DurableHarness } from "@earendil-works/pi-durable"; + +export const ChordHostKey = createContextKey>("studio.chord-host"); +export const HarnessModels = defineRegistry("studio.harness-models", { version: 1, compose: firstModels }); +export const HarnessEnvironments = defineRegistry("studio.harness-envs", { version: 1, compose: firstEnv }); +export const HarnessExtensions = defineRegistry("studio.harness-extensions", { + version: 1, + compose: (table, { report }) => extensionSnapshot(table, report), // one Durable Extension per plugin +}); +const HarnessRuntime = defineRegistry<() => DurableHarness, () => DurableHarness>("studio.harness-runtime", { + version: 1, compose: onlyEntry, // core-private: one contributor +}); + +host.serve(Harness, (_input, self) => { + const info = self.get(SessionInfo, {}); + const harness = self.open(async ({ signal }) => { + const { id, dataDir } = info.init.value; + const base = withContextValue(ChordHostKey, host, withAbortSignal(signal, BACKGROUND_CONTEXT)); + return DurableHarness.open(await openNodeSqliteStorage(join(dataDir, id, "durable.sqlite")), { + models: () => host.registry(HarnessModels).value, // read at each use + env: (target) => host.registry(HarnessEnvironments).value(target), + settings: () => currentSettings(), + registry: durableRegistry(host.registry(HarnessExtensions)), // a RegistryReader over the composed value + }, base); + }, (h) => h.close(BACKGROUND_CONTEXT)); + self.contribute(HarnessRuntime, () => harness.value); // for core's conversation objects + return { … }; +}, { lifetime: "host" }); +``` + +- **Lifetime `"host"`:** it runs for the worker's whole life, also while nobody holds it, because Durable tasks + continue. +- **Base code:** it lives in the core facet, so a plugin reload never restarts it; only a base-code change (a host + restart) does. +- **Registries, looked up at each use:** Durable's `models` and `settings` accept functions (pi `7ec1f648e`), so each + generation gets the current value. After a models reload the next request uses the new runtime; a running stream + finishes on the old one. +- **Durable's storage is its own.** Chord documents never commit together with Durable transcripts. + +The design writes the call as `Durable.open(storage, { … })`; pi-durable's actual entry point is +`Harness.open(storage, options, context)`, used here as `DurableHarness` to keep it apart from the core object +`Harness`. + +## Start order + +At host start Chord loads every facet and waits until every facet and every `lifetime: "host"` object **other than the +harness** is live or failed: the **start barrier**. Then the registries hold the contributions of every plugin that +loaded, and the harness opens and resumes scheduling (R17). An object that must contribute at start, like the model +runtime, is therefore `lifetime: "host"`. Nothing waits for specific contributions; what is missing shows where it is +used: + +| Missing | What happens | +| --- | --- | +| a tool | its call gets an error result the model sees | +| a model | the generation reports "model unavailable" and waits, not fails, until a model appears | +| a task definition | the task stays blocked (`missing_task`) until the plugin is installed or the task is aborted | +| a plugin that failed | reported with its error; a facet in backoff contributes later and the registries recompose | + +The same holds when a session's database is copied to an instance with other plugins: missing tools, sections and +hooks are skipped, documents of missing plugins stay untouched, stored selections are never rewritten. The UI must +show blocked tasks with their reason and offer to abort them; a turn waiting on one hangs until then. + +:::review | Review: the harness exception to the barrier (C17, R17) +R17 makes the harness start after the barrier, a special case in the barrier's rule. Review round 3 proposed cut C17: +no harness-specific exception; the harness is an ordinary object the app's start code holds after `load` resolves. +Not decided. `chord-next-walkthrough.md` concept 28 still says the harness's activation *is* the start barrier, which +R17 replaced. +::: + +## Tools that call Chord + +Durable tools, hooks and sections receive Durable's `ToolExecutionApi` (or `HookApi`) and a `Context`, never a Chord +`ctx` (Q6). Core put the host into the harness's base context under `ChordHostKey`, and every phase context inherits +it. A core helper opens a Chord call scope from any Durable context: + +```ts +// studio/src/core/chord.ts +export function chord(context: Context, fn: (ctx: Ctx) => Promise): Promise { + const host = context.value(ChordHostKey); + if (!host) throw new Error("not inside the Studio harness"); + return host.root(fn, { signal: context.abortSignal, telemetry: context.telemetry }); +} +``` + +The scope runs as the host principal; attribution is data; references taken with `ctx.get` are released when `fn` +settles. There is no wrapping of tools and no app-specific `defineTool`. The notes plugin gives the agent a tool: + +```ts +// notes/worker.ts +const addNote = defineTool({ + name: "notes_add", // tools are named _ + description: "Add a note to this session's notes panel", + parameters: Type.Object({ text: Type.String({ maxLength: 2000 }) }), + execute: (args, api, context) => chord(async (ctx) => { + const info = await ctx.get(SessionInfo, {}); + const store = await ctx.get(NoteStore, { session: info.init.value.id }); + const id = await store.add({ text: args.text, by: `agent:${api.conversationId}` }, ctx); + return { output: [{ type: "text", text: `added note ${id}` }] }; + }, context), +}); + +export default defineFacet((host, self) => { + self.contribute(HarnessExtensions, defineExtension({ name: "@studio/notes", tools: [addNote] })); + … +}); +``` + +A reload of the notes plugin replaces its extension in the table; Durable pins a registry snapshot per task phase, so +a tool call already running finishes on the old code and the next phase uses the new one. + +## Tools that need a person + +Clients are not addressable, so a tool cannot call the user's tab. A tool that needs the user goes through an owner's +state: the Questions service (chapter XII). The model-facing `ask_user` tool and an approval hook both call +`Questions.ask`; tabs render the open questions and answer them. + +```ts +hook(ToolTask, { + beforeTool: async (call, api, context) => { + if (!needsApproval(call)) return undefined; + const decided = await api.memo("approval", context); // a rerun does not ask again + if (decided !== undefined) return decided ? undefined : { block: "Denied" }; + const allowed = await chord(async (ctx) => { + const q = await ctx.get(Questions, { scope: { session } }); + for (;;) { + try { + const r = await q.ask( + { kind: "confirm", message: `Run ${call.name}?`, allow: "Allow", deny: "Deny" }, + { audience: approvers, policy: { mode: "first" }, key: api.taskId }, + ctx, + ); + return r.answer; + } catch (error) { + if (ctx.signal.aborted) throw error; // a link blip is not a denial: retry + await backoff(ctx); + } + } + }, context); + await api.memo("approval", allowed, context); + return allowed ? undefined : { block: "Denied by the user" }; + }, +}); +``` + +Durable runs `beforeTool` in the tool task's call phase; after the execute checkpoint a crash retries execution +without hooks, so the stored decision covers that retry. The approval must cover the final arguments: put the +approval hook after hooks that rewrite arguments. + +## Conversations for the UI + +Chat panels need a conversation's live view, submit, steer, stop. The design leaves p3's conversation API to the p3 +design; one construction that follows the rules: a core `ConversationStore { conversation }` (local) opens the Durable +watch as its own native, getting the Durable harness from the core-private `HarnessRuntime` registry contribution (a +native may reach another object only through a registry entry), and exposes a bounded window as a derive. Per-principal +`Conversation { session, conversation }` views in front of it check who may send (chapter XIX). + +## Code mode + +Code mode is a plugin. It owns its `pi-codemode` sandbox as a native; its tool entry looks up the **current** sandbox +when `execute()` starts, so a call pinned by a phase before a reload (waiting in an approval hook, say) starts on the +new sandbox. On `reload` and `stop` the old sandbox is sealed and drained in `close`: running scripts finish or hit +their deadline, for at most 30 s (Q7). After a worker stop Durable replays as it does today. + +## What lives where + +| Data | Where | +| --- | --- | +| transcript-bound: entries, documents that fork and rewind with a conversation | Durable | +| app state: session list, notes, drafts, grants, settings | Chord documents | +| data core enforces on conversations (read-only one-shots, cwd followers) | a core per-conversation policy document, written when the conversation is created (`createConversation({ init })`) | + +**Telemetry:** pass the host's adapter as `HarnessOptions.telemetry`; Durable stores trace carriers on submissions and +tasks (chapter XVII). diff --git a/packages/chord/docs/next/explainer/src/15-reload.md b/packages/chord/docs/next/explainer/src/15-reload.md new file mode 100644 index 000000000..09ab349de --- /dev/null +++ b/packages/chord/docs/next/explainer/src/15-reload.md @@ -0,0 +1,131 @@ + +# Reload and Rollout {#reload lede="What happens when you press Reload: per host, per object, and across the server, the workers and every tab."} + +Back to the preface. Mario changed the notes plugin's sort order and pressed Reload. Everything the earlier chapters +asked of plugin authors (synchronous factories, natives only in objects, routines instead of timers, data in +documents) exists so that this moment is safe. This chapter shows what Chord does with it. + +## On one host + +```text +1. load new bundles; an import or syntax error rejects +2. staging facet setups and every live object's new factory run with a recording self + (declarations only, nothing activated); any throw rejects +3. commit the host switches to the new code atomically +4. after per object: pure objects swap, native objects restart +``` + +**Rollback is all-or-nothing per host.** Any load or staging error leaves the whole host on its previous code, so a +host is always completely at one revision and its status is one value. Runtime states (a registry conflict, a failed +open, a migration) are not reasons to roll back; after the commit, errors stay with one object. There is no factory +trial, no migrate dry run, and no consumer contract check in staging: an incompatible consumer fails at bind with +`contract_mismatch` and a notice naming plugin and type. + +:::steps | Reloading @studio/notes on worker S1, object by object ++++ Load and stage +The new bundles load. The facet setup and the factories of `Notes` (three views), `NoteStore { S1 }` and `NoteInbox` +run with a recording `self`: they declare, nothing activates. Nothing throws, so the host commits to the new code. ++++ NoteStore: pure, swapped in place +`NoteStore` has no `self.open`. Its new methods and derives take over; `NoteList`, the drafts, its references and the +running import job stay. Routines restart, serialized: the new `cleanup` waits for the old handler in flight. The gate +opens. ++++ Running calls finish on old code +An `add` that was running when the swap happened finishes on the old code with its old handles; its transaction still +commits. Only new calls get the new handles. Exception (R15): if the reload raised a document's version, old code that +touches the migrated document throws, the document stays intact, and that call fails once. ++++ The import job keeps its old run +The job's `run` keeps the code it started with. When it settles, the **new** `finish` records the outcome. ++++ The views: pure, swapped +The three `Notes` views swap in place. Owner, Ann and Bob keep their references; the reader derive is recomputed with +the new sort and the difference is sent. Nobody sees a reconnect. ++++ NoteInbox: native, restarts +`NoteInbox` has a watcher. Its running work ends `aborted`, the watcher closes with reason `reload`, the new factory +runs, the watcher reopens. Its documents would stay. Whether a file dropped during the restart is seen is the +watcher's business: Studio's `watchDir` scans the directory once when it opens. +::: + +**Why natives restart instead of being handed over:** a native has no line, no transactions and no schema. If old and +new code shared it, they would interleave on it unchecked. In practice restarts are cheap: the Durable harness is base +code, terminal PTYs live in a separate terminal host process, and model runtimes, sandboxes and a search index rebuild +in milliseconds. Only reloading the plugin that owns SSH connections (running remote commands die) or the relay socket +(relayed connections drop) hurts. + +**Removing a type:** its objects end (routines stop, natives close, references end with an error); in-memory documents +go with them; persisted ones stay until retired. **Consumers elsewhere** see a restarting object like any object that +briefly goes away: `loading`, parked calls, then `live`. + +**UI facets reload too.** A reload unmounts and remounts components. That is why "leaving" (abort a login, close an +overlay) is never derived from an unmount (chapter VIII). + +## Across hosts + +A **revision** is (server incarnation, sequence) and names the complete desired code set: every facet with its +digest, not a batch of changes. Every install reconciles everything missing, and a host reports a revision only when it +runs that complete set. A revision also carries the code-set digest (a hash over the facet bundles' content hashes), +so a server restart with unchanged code triggers no reinstall. + +The order is **server, workers, clients**. Each host reports `installed`, `installed_failed` (with the objects that +failed) or `rejected` (with the reason; it stays on its previous code). A server that rejects stops the rollout. A +host that starts or reconnects installs the current revision before anything is routed to it. Installs are serialized +per host and bounded (60 s). + +Calls keep flowing during a rollout. When a call reaches a provider on another revision, the mismatch response +carries the provider's revision and status: + +| Provider | The call | +| --- | --- | +| ahead, or behind and installing | `updating`: values readable, calls park (with the park deadline) | +| rejected | fails at once; retried on the next revision | +| same revision | `contract_mismatch`: a real mismatch, final | + +A host checks its own install status first: if its own install was rejected, a call fails at once with the reason +instead of waiting on a provider. Calls parked on a binding that is released end with an error and are not resent. +There are no cross-host transactions and no votes. + +:::sequence animate | A tab calls into a worker that is still installing. The call parks and runs once the worker reports the revision. +actor tab "tab" "revision 8" +actor server "server" "revision 8" accent +actor worker "worker S1" "installing 8" blue +server => worker: install revision 8 +tab -> server: call Notes.add (new contract) +server -> worker: relay +worker --> server: updating (rev 7, installing) +server --> tab: updating +tab -> tab: park (deadline 60 s) +worker --> server: installed 8 +server --> tab: provider at 8 +tab -> server: call Notes.add (resent from park) +server -> worker: relay +worker --> tab: result +::: + +## Start barrier + +At host start and first load there is one more rule (chapter XIV): load every facet (the core facet with its start +data among them), wait until every facet and `lifetime: "host"` object other than the harness is live or failed, then +resume dependent work. Plugin reloads need no barrier. + +## Loading and the dev loop + +- **Roots:** plugins come from built-in, global (`/plugins`), package (npm or git, installed with + `--ignore-scripts`) and project roots, found by directory listing; no source parsing. The server selects which load. + Dependencies between plugins resolve by contract name at runtime. Names are prefixed: objects `.`, + tools `_`. +- **Loader D:** one base bundle per surface (host packages and the app runtime) plus one script per facet, with host + packages passed in (no module map). Old generations are reclaimed by the garbage collector; a stable filename per + (facet, digest), and V8's compilation cache off on hosts that reload in place. The loader is an adapter: static + imports (tests, Durable Objects) work the same, with the app passing the plugin name to `load`. +- **Dev loop:** a source watcher marks plugins dirty and never applies anything. A reload control (the playground's web + shell, a CLI, an agent tool such as p2's `reload_plugins`) reloads the changed plugins, or restarts all hosts for + base code. Reloads are always explicit. + +:::review | Review: project plugins (#14, deferred) +Per-directory plugins with trust prompts and snapshots (`apps/p2/docs/plugin-loading.md`, still the reference for +roots, trust and scopes) need their own pass over this design. Until then, treat the project root as designed but not +settled. +::: + +:::note | Built so far +Reload and rollout are layer L10, the last before telemetry. The frozen kernel has a reload path with host values and +a development freeze, both removed from the design. +::: diff --git a/packages/chord/docs/next/explainer/src/16-contracts.md b/packages/chord/docs/next/explainer/src/16-contracts.md new file mode 100644 index 000000000..51e0126cb --- /dev/null +++ b/packages/chord/docs/next/explainer/src/16-contracts.md @@ -0,0 +1,310 @@ + +# Changing Contracts {#contracts lede="How a contract gets a new version while old tabs, old workers and old plugins are still talking to it."} + +Chapter XV ended with a tab that calls into a worker on another revision. That case is short-lived: the rollout +finishes and both sides run the same code. Other cases last longer. A phone kept a tab open overnight with yesterday's +web bundle. A plugin from another author was built against `Notes` version 1 and is not updated yet. In both, a +consumer holds a contract that the provider no longer has. This chapter is about what the provider does then. It is +also the one part of Chord v2 that is built: everything in the sections on `evolve`, `checkAccepts` and fingerprints +runs today in `packages/chord/src/next/contracts/`. + +## What has a version + +Everything that is stored or crosses a boundary has a version. Each surface handles an old version its own way: + +| Surface | Version | When an old one shows up | Chapter | +| --- | --- | --- | --- | +| object type (contract) | `version` plus fingerprint | `evolve` and `checkAccepts` on the provider; otherwise `contract_mismatch` | this one | +| document | `version` plus schema fingerprint | pure `migrate` on access; a schema change without a bump fails the object `broken` | V | +| job | `version` | `accepts` on outcomes and `running`; `blocked` without an adapter | XI | +| registry | `version` | `accepts` adapts contributions; one without a match is left out and reported | XIII | +| native | none | the object restarts on reload | IX | +| Durable tasks and documents | Durable's `version` | Durable's `migrate` | XIV | +| link protocol | protocol version | refused; hosts update together, a web page reloads | XVII | +| UI components, Chord and app APIs | none yet | exposed as they are; an SDK will version them later | | + +The rule behind every row is the same as for documents: **every change bumps the version**, additive ones included. +Chord never guesses compatibility from the shape of two schemas. + +## A history, not a file of versions + +A contract history starts with a **base**: a full `remote()` or `local()` declaration with an explicit `version`. The +`Notes` contract from chapters II and III is a base at version 1. Studio now wants pinned notes: notes get a `pinned` +flag and editors get a `pin` method. Instead of editing the declaration, the plugin continues the history: + +```ts +// notes/contract.ts +import { bool, evolve, nul } from "@earendil-works/chord/next"; + +export const NoteV2 = obj({ id: str(), text: str({ max: 2000 }), by: str(), at: int(), pinned: bool() }); +export const PublicNoteV2 = obj({ id: str(), text: str(), at: int(), pinned: bool() }); + +export const NotesV2 = evolve(Notes, (e) => + e.state("editor", "notes", obj({ items: arr(NoteV2) })) + .state("reader", "notes", obj({ items: arr(PublicNoteV2) })) + .addMethod("editor", "pin", obj({ id: str(), pinned: bool() }), nul()), +); +``` + +`evolve(prev, (e) => …)` appends one version to `prev`'s history: `NotesV2.version` is 2, and `NotesV2` still knows +version 1 exactly as it was declared. Every version is **materialized and fingerprinted at declaration**. That is what +makes old code safe: an old plugin bundle that contains its own copy of `Notes` version 1 computes the same +fingerprint as the history inside `NotesV2`. There is no build step, no generated file and no fingerprint copied by +hand. + +The edits: + +| Edit | Changes | Kind | +| --- | --- | --- | +| `e.state(variant, member, schema)` | one state member of one variant | remote | +| `e.addState(variant, member, schema)` / `e.removeState(variant, member)` | adds or removes a state member | remote | +| `e.state(member, schema)`, `e.addState(member, schema)`, `e.removeState(member)` | the same, flat | local | +| `e.args(name, schema)` / `e.result(name, schema)` | one side of a method | both | +| `e.method(name, args, result)` | both sides of a method | both | +| `e.addMethod(variant, name, args, result)` | adds a method (or an existing one to another variant, same signature) | remote | +| `e.addMethod(name, args, result)` | adds a method | local | +| `e.removeMethod(name)` | removes a method from every variant | both | + +Edits chain. The `method(...)` wrapper appears only in base declarations; edits take the schemas directly, so a +method without a result is added with `nul()`. The rules of chapter III still hold across versions: a method has one +signature in every variant, and a member's kind never changes, not even after it was removed and added again. + +**What `evolve` cannot do:** change the input or the set of variants. Both change what an object *is* (its identity, +who gets which members), and no adapter can turn one into the other. That needs a new history base, below. + +## The provider says what it accepts + +`NotesV2` says what changed. It does not say how version 1 consumers are served; the **provider** decides that, with +mappings that live in its own code, never in the contract. The worker facet declares them with `checkAccepts`: + +```ts +// notes/worker.ts +import { checkAccepts } from "@earendil-works/chord/next"; + +const toV1 = ({ pinned, ...note }: Static) => note; +const toPublicV1 = ({ pinned, ...note }: Static) => note; + +const notesAccepts = checkAccepts(NotesV2, { + 1: { + state: { + editor: { notes: (current) => ({ items: current.items.map(toV1) }) }, + reader: { notes: (current) => ({ items: current.items.map(toPublicV1) }) }, + }, + methods: {}, // add and remove did not change; pin is new + }, +}); +// notesAccepts.versions: [{ version: 1, fingerprint: Notes.fingerprint }] + +host.serve(NotesV2, notesFactory, { accepts: notesAccepts }); // working name for the objects-layer option +``` + +`checkAccepts` is **pure**: it checks the mappings without calling them and returns only the version metadata of the +old versions it lists. A provider that lists nothing accepts only its current version. The objects layer later +compares a consumer's version and fingerprint against this list. + +The mappings point in the direction the data flows: + +| Member in the old version | Mapping | Direction | +| --- | --- | --- | +| state, changed | `(currentMember) => oldMember` | provider's current value to what the old consumer expects | +| state, removed | `(currentVariantState) => oldMember`, or `"refuse"` | receives the whole current state of that variant | +| method arguments, changed | `args: (old) => current` | old caller's arguments to current ones | +| method result, changed | `result: (current) => old` | current result back to the old shape | +| method, changed or removed | `{ call: async (oldArgs, ctx) => oldResult }`, or `"refuse"` | replaces the method for old callers | +| member added since | nothing | an old consumer never asks for it | + +A changed state member cannot be refused: the old consumer still has that member and must get a value. A removed one +can be computed from what the variant has now, or refused. `e.method(name, args, result)` replaces both sides and +therefore needs both mappings, even if one side stayed the same (then an identity mapping). + +**Required, at type level and again at runtime.** For every member that exists in the old version and changed or was +removed since, the mapping is required by the type of `checkAccepts`. Because plugins are bundled without type +checks, the same check runs at declaration: a missing mapping, a key for a member that needs none, an unknown version +or a wrong shape throws, naming the member's path (for example `accepts[1].state.reader.notes`). An absent `state` or +`methods` section counts as empty and may be left out only when nothing there needs a mapping. + +**Conservative with no-op edits.** An edit that sets a schema to what it already was, or changes a member and then +changes it back, still requires an identity mapping. TypeScript erases constraints such as `{ max: 2000 }` from the +static type, so two schemas that look the same to the compiler may differ; Chord does not try to tell. + +### Old versions map straight to the current one + +Studio later replaces `remove` with `archive`, which keeps the note in a trash list: + +```ts +export const NotesV3 = evolve(NotesV2, (e) => + e.removeMethod("remove").addMethod("editor", "archive", obj({ id: str() }), nul()), +); + +const notesAccepts = checkAccepts(NotesV3, { + 1: { + state: { editor: { notes: (c) => ({ items: c.items.map(toV1) }) }, reader: { notes: (c) => ({ items: c.items.map(toPublicV1) }) } }, + methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } }, + }, + 2: { methods: { remove: { call: async ({ id }, ctx) => { await archiveNote(id, ctx); return null; } } } }, +}); +``` + +Each entry maps **one old version directly to the current one**, not through the versions in between. Version 1 +needs the state mappings (its notes have no `pinned`) and the `remove` replacement; version 2 needs only the +`remove` replacement, and leaves `state` out because its state is unchanged. A member that an old version never had +needs nothing for that version, even if it was added, changed and removed later. Dropping support for version 1 is +deleting its entry: version 1 consumers then get `contract_mismatch`. + +## Binding an old consumer + +A hold carries only the consumer's contract **version and fingerprint**, never a description of the contract (R6). +The provider decides with its own schemas: + +:::flow wide animate | What a provider does with a hold. The decision uses only the provider's own history; a peer's schema is never an input. +row "The hold arrives" +box hold "hold" "version 1, fingerprint F1" blue +box cur "equal to current?" "version and fingerprint" +box acc "accepted old version?" "F1 from the provider's own history" +row "Outcome" accent +box live "bind" "current members" +box adapted "bind adapted" "through the mappings" accent +box mismatch "`contract_mismatch`" "or updating, during a rollout" muted +arrow hold -> cur +arrow cur -> live "yes" +arrow cur -> acc "no" +arrow acc -> adapted "yes" +arrow acc -> mismatch "no" +note "A mismatch fails only that consumer's binding. The provider object and its other bindings are not affected." +::: + +Once a version 1 tab is bound, it is served as if version 1 were current: + +:::steps | A version 1 tab on a version 3 worker ++++ Bind +The tab holds `Notes { S1 }` with version 1 and fingerprint F1. The worker's history computes F1 for version 1, and +`notesAccepts` lists version 1, so the hold binds adapted. The tab gets the `editor` variant as version 1 defined it: +`notes`, `add`, `remove`. It never sees `pin` or `archive`. ++++ State through the mapping +Chord serves the adapted state like a derive of the provider's state: it applies the mapping to the current value +and keeps the result per binding. The snapshot the tab receives has no `pinned` fields. ++++ A commit +The owner pins a note from a current tab. The worker sends the current tab the draft's ops. For the version 1 tab, +Chord applies the mapping to the new value and sends the difference to its last adapted value (`diffRevisions`). The +mapping produced nothing new for a pin, so the version 1 tab receives no change at all. A large list is never resent +whole on every commit. ++++ A call +The tab calls `remove({ id })`. Its arguments are checked against the provider's own version 1 schema, then the +`call` replacement runs `archiveNote` on version 3 code. Its `null` result is what version 1 expects. Had the method +only changed, `args` would have mapped the arguments forward and `result` the result back. ++++ The worker moves on +A later revision drops the version 1 entry. When the tab re-holds (or the worker restarts and the tab binds again), +the hold gets `contract_mismatch`, and Studio's panel shows "reload to update" instead of failing silently. +::: + +## A new history base + +Some changes cannot be adapted. Studio decides that notes belong to a project, not a session: the input changes from +`{ session }` to `{ project }`. That is a new base, written in full, with an explicit version that continues the +published numbering: + +```ts +export const NotesV4 = remote("notes", { + version: 4, // explicit: the published history used 1 to 3 + input: obj({ project: str() }), + placement: () => Server, + variants: { + editor: { state: { notes: obj({ items: arr(NoteV2) }) }, methods: { add: method(…), pin: method(…), archive: method(…) } }, + reader: { state: { notes: obj({ items: arr(PublicNoteV2) }) } }, + }, +}); +// checkAccepts(NotesV4).versions is empty +``` + +A new base has no history before its version and **cannot accept older versions**, not even for an additive change. +Every consumer built against versions 1 to 3 gets `contract_mismatch` and must update. Later `evolve` calls continue +from version 4. The same applies to a change of the variant set. Do not spread an older declaration into a new one or +copy its fingerprint; the contract objects are branded and `evolve` and `checkAccepts` reject copies. + +## Local contracts + +`local` contracts (chapter IV) evolve the same way, with a flat members map and no variants: + +```ts +export const SessionInfoV2 = evolve(SessionInfo, (e) => e.state("init", InitV2)); +checkAccepts(SessionInfoV2, { 1: { state: { init: (current) => toInitV1(current) }, methods: {} } }); +``` + +Same-host consumers of a `local` type are usually reloaded together with their provider, but not always: another +plugin on the same host may be built against an older version. The rules are identical; only the variant argument is +missing. + +## Implementing a contract + +`Implementation` is the type of everything a provider must supply for the current version: one state +getter per member and variant, and one asynchronous method per method, each receiving `(args, ctx)`: + +```ts +const impl: Implementation = { + state: { editor: { notes: () => list.value }, reader: { notes: () => publicList.value } }, + methods: { add: async ({ text }, ctx) => addNote(text, ctx), remove: async ({ id }) => removeNote(id), pin: async (a) => pin(a) }, +}; +``` + +Keep declaration types **inferred** (no annotation on `NotesV2`): the inferred type carries the complete history, and +`checkAccepts` needs it to know what changed. + +:::review | Review: L1's implementation shape and the factory's return value +`Implementation` in `src/next/contracts/types.ts` is `{ state: { variant: { member: getter } }, methods }`. The +design's factories (`design.md` §6, used throughout this guide) return `{ variant, member: value, method: fn }`. +The objects layer (L4) has to connect the two, and also define how `serve` receives the checked accepts (this guide +uses the working name `{ accepts }` as a third argument) and what `"refuse"` does at runtime. The natural reading, +used in the stepper above, is that a refused method call and a hold that needs a refused state member end with +`contract_mismatch`; no source says so. +::: + +## The fingerprint + +The fingerprint is a 64-bit hash (`hash64`, portable across runtimes) of a **normal form** of the contract's +schemas. The normal form decides what counts as a change: + +| Ignored | Significant | +| --- | --- | +| order of object keys, `required` and `enum` (sorted) | order of union and intersection branches (`anyOf`, `oneOf`, `allOf`) and of tuples | +| annotations on schema nodes: `title`, `description`, `examples`, `$comment`, `deprecated` | defaults, formats, patterns, bounds, `$ref` names | +| placement | input, variants, member kinds and every member schema | + +Union order is significant because it is meaning: TypeBox picks defaults and decodes by the first matching branch. +Annotations are dropped only on schema nodes; an object property that happens to be named `title` is data and counts. + +Only data schemas are allowed. One walker copies, validates and normalizes each schema against a **whitelist of +TypeBox keywords**: primitives, arrays and tuples, objects and records, unions and intersections, `$defs` and `$ref` +(use `$ref` for recursive types), constraints and annotations. Conditional and dependent schemas, `not`, +`definitions`, `contentSchema`, `unevaluated*`, `contains`, `propertyNames`, unknown keywords, executable or custom +TypeBox features, arbitrary `Unsafe` and refinements, accessors, symbols and cyclic objects are rejected at +declaration. Bytes are the one special leaf: `bytes({ min, max })` checks a `Uint8Array` and has the data marker +`{ type: "bytes", minLength?, maxLength? }`, so it hashes like any schema; TypeBox's checking still works on it. + +Schemas are deep-copied into a private, frozen history; your own schema objects are copied, not frozen. +`NORMAL_FORM_VERSION` is part of every digest, so a change of the normal form is a link protocol change. + +**Not a security boundary.** A fingerprint says "we were compiled against the same contract"; it proves nothing. A +hostile client can send any fingerprint it likes. Providers therefore always validate arguments with their own +schemas (chapter IV), whatever the fingerprint says. + +## Rules of thumb + +- Treat a published contract like a published API: every change is `evolve`, every `evolve` needs a decision about + old consumers. +- Keep mappings small and pure. A state mapping runs for every commit and every adapted binding; make it a map over + the value, not a query. +- Accept the versions your users still run (one or two back during a normal rollout), and delete entries when the + metrics show no old bindings. +- Change the input or the variants only with a new base, and only when updating every consumer is acceptable. +- Contract, document and job versions are separate and each is decided on its own. Pinned notes needed `NotesV2` + for the wire and `NoteList` version 2 with a `migrate` that adds `pinned: false` (chapter V), but no new + `ImportJob` version, because its outcome did not change. + +:::note | Built so far +This chapter's contract layer is built: `packages/chord/src/next/contracts/` (commit `7d2bb8ea8`, about 1,300 lines +with `remote`, `local`, `method`, `evolve`, `checkAccepts`, `Implementation`, the schema helpers, `fingerprintOf`, +`schemaNormalForm` and `NORMAL_FORM_VERSION`), with nine test files covering evolution, fingerprints, the hash, +local contracts, long chains, schemas, types and review findings. It stands alone: it hosts nothing, calls no +mapping and does not import the frozen runtime. Binding by version and fingerprint, adapted state and calls belong to +the objects and protocol layers, not built yet. +::: diff --git a/packages/chord/docs/next/explainer/src/17-wire.md b/packages/chord/docs/next/explainer/src/17-wire.md new file mode 100644 index 000000000..67bd21118 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/17-wire.md @@ -0,0 +1,251 @@ + +# On the Wire {#wire lede="What travels over a link, what a relay does with it, how big things are kept in check, what each hop costs, and how to see it in traces."} + +Most plugin code never thinks about the wire: it declares state, commits transactions and calls methods, and Chord +moves the bytes. You need this chapter when something is slow, when a value is large, when a link misbehaves, or when +you want to read a trace. It also explains why the earlier chapters insisted on small operations, shared structure and +no callbacks: each of those rules removes a cost on some hop. + +## What a state member sends + +Every exposed document, derive or readable is a **state member**. Over a hold it produces one of these: + +| Update | When | +| --- | --- | +| `snap` | on hold and on re-hold: the whole value, once | +| `ops` | per commit: the draft's recorded ops, no diff | +| `ops` from a diff | per change of a derive (and of an adapted binding, chapter XVI) | +| throttled `ops` | job progress | +| `reset` | a migration commit replaced the value (adapted for older consumers) | +| `error` | the member failed | + +**Commits are atomic for consumers.** On the same host, subscribers are notified once, after every document of the +commit is adopted. Remotely, one `state` message per hold carries a batch of member updates that is applied before +listeners run, and a commit that touches several holds on one link travels as **one** `state` message on that link. +A consumer never sees half a commit. + +## Messages + +A message is a **header plus opaque parts** (R1). There is one part per hold the message concerns; the header says +which hold each part belongs to and how large it is. + +| Direction | Kind | Parts | +| --- | --- | --- | +| consumer to provider | `hold`, `release`, `call`, `cancel` | `call` has one (the arguments) | +| provider to consumer | `bound` | one: the snapshot | +| | `state` | one per hold the commit touched on this link | +| | `result` | one | +| | `gone` | none | + +There are no request ids in application code. Hold ids and call ids exist only inside the protocol, per link. + +**One binary format on every link (Q4):** `u32 header length | header JSON | part bytes`, as one `Uint8Array`. A +WebSocket sends it as a binary frame, IPC passes it with `serialization: "advanced"`, a memory link delivers it +asynchronously. Encoding, framing, header checks, id resolution, flow control and size limits are one shared protocol +layer; link adapters only move bytes. Tests on memory links therefore run the production path. + +A part is encoded **once, at its origin**, with that hold's own Delta encoder (paths are interned per hold, end to +end between provider and consumer). A JSON part above a threshold (about 16 KiB) is compressed once at its origin, +whatever the link; byte segments stay raw; relays forward compressed parts as they are; the header names the codec. + +**The header is a TypeBox schema (R4)**, a union tagged by `kind`; the protocol types derive from it. Headers from +untrusted links (client WebSockets) are checked with the schema compiled once; trusted links (IPC between server and +workers) are not checked. **Unknown ids are ignored on both sides (Q1).** They are normal races: a `result` crossing a +`cancel`, a `call` crossing a `gone`, a state part crossing a `release`. The side that ended a hold or call settles +its own pending work: a consumer that receives `gone(h)` rejects its pending calls on `h`, and the order per +direction guarantees nothing else for `h` follows. Only a header that fails the schema is a protocol error, and it +closes that link. + +The link protocol has a version. Hosts with different protocol versions refuse each other; a web page reloads to get +the server's protocol. + +## Through a relay + +A tab never talks to a worker directly (chapter I). The server **relays**, and a relay reads and rewrites only +headers: it maps hold and call ids between its two links, stamps the principal on `hold` and `call` (a client's +principal field is always overwritten; only a trusted parent link may carry one), and groups parts per destination +link, so each link gets only its own holds' parts. It never parses a payload. + +:::sequence animate | The owner adds a note while Ann watches. The commit leaves the worker as one `state` message with two parts; the server splits it by destination without decoding either part. +actor owner "owner's tab" "hold h1" blue +actor ann "Ann's tab" "hold h2" muted +actor server "server" "relay" accent +actor worker "worker S1" "NoteStore, Notes" +owner -> server: call c7 Notes.add (1 part) +server -> server: map ids, stamp owner +server => worker: call c3 (part forwarded as is) +worker -> worker: method runs, self.tx commits +worker => server: state: parts for h1', h2' +server -> server: group parts per link +server --> owner: state (h1 part) +server --> ann: state (h2 part) +worker --> server: result c3 +server --> owner: result c7 +::: + +The `state` message reaches the owner's tab before the `result`, so when `await notes.add(…)` resolves, the new note +is already in `notes.value`. Ann's view is a different variant (reader) and gets its own part from its own derive; the +relay neither knows nor cares. + +## Flow control + +Each link has **one byte window**, counted from each message's `byteLength` (parts are already encoded bytes; nothing +is measured or stringified to count it). The window covers every message, first snapshots included. + +- **Snapshots are always admitted (R19)**, even when larger than the window; they use it up. Whether a hold + **overflows** is decided on the changes queued after the snapshot. +- **Under backpressure** a commit's ops are collected per hold and encoded into one part when credits return. The + consumer still gets the commit atomically. +- **After an overflow** the hold is dropped and established again once the previous snapshot's debt has drained + (credits back at the full window, Q3), and not more often than a minimum interval. So there is at most one snapshot + in flight per hold: no re-hold storm, and no stall for a snapshot larger than the window. +- **Calls** are throttled by call credits the provider grants (a generous window). Beyond them calls wait locally. + Nothing is refused for being "too many", and holds are not limited by count; authorization and input validation + protect them. +- **Byte-heavy traffic** such as terminals uses separate channels (chapter XIII), so a large state message never + holds up keystrokes. + +**Recovery:** after a link drops, the consumer re-holds every hold over the new link and receives snapshots; calls in +flight fail as on any disconnect, calls made meanwhile park. A relay whose window overflows releases the hold and +re-holds; the provider sends a fresh snapshot. After a routed link drops, Chord re-resolves the address (with the +per-address backoff) while holds on it exist, so a passive viewer recovers without anyone pressing anything. + +## Size limits + +Limits apply **per part** (R2, Q2), both to the compressed and the decoded size, and are checked from the sizes in the +header before anything is decoded; zlib's `maxOutputLength` enforces the decoded limit in the one decode function. + +| Situation | What fails | +| --- | --- | +| a part over the limit | only its own hold or call, with `too_large`; the other parts of the message apply | +| a snapshot over the limit | its hold ends `failed`, with no automatic re-hold | +| a WebSocket frame over `maxPayload` (e.g. 64 MiB) | the connection closes (code 1009); this is the transport's hard cap | + +Defaults for client WebSockets are 16 MiB compressed and 64 MiB decoded per part, configurable per link kind. The +decoded limit travels with relayed work: the server stamps the admitting link's limit on the header at admission +(never taken from the client), and the host that decodes enforces it. IPC therefore needs no limit of its own. + +Large data is never a large state member or a large argument. Files go through chunked transfers or the HTTP +component (chapter XIII); long lists are keyed documents or bounded windows (chapter VIII). + +## What each hop costs + +Chord v2's goal is as little compute and memory as possible while staying correct: **O(size of the change)** per +commit and per viewer, independent of the size of the value. A 500 KB value is touched as a whole only when a new +viewer takes its first snapshot. Delta, the value library under documents, was built for this, and its own primitives +are almost free: `diffRevisions` on shared subtrees takes 0.018 ms, `applyImmutable` of a small change 0.004 ms. + +| Hop | Work | +| --- | --- | +| commit | Delta's `prepare` (copies and checks only the values placed in this commit), adopt by pointer swap; storage writes the delta | +| reads in a transaction | the draft overlay, no copy; an unchanged draft writes nothing | +| subscribers on the same host | the same value and ops, by reference | +| derives | compute plus `diffRevisions`; unchanged parts stay the same objects (`mapShared`, chapter VIII) | +| send | one encoding per hold part at its origin, size taken from the encoded bytes | +| relay | header only: ids mapped, principal stamped, parts grouped; parts never parsed | +| receive | compiled header schema on untrusted links; arguments stripped and validated once at the provider (compiled schema); inputs normalized | +| apply | `applyImmutable`, structure shared | +| new hold | the value encoded once (unavoidable) | +| memory | one authoritative value per document per host; replicas and derived values share structure; one cached value per adapted binding | + +These are targets. The frozen kernel measured far worse, which is why the design is so strict here: + +:::stats +- 35 ms | one small change to a 500 KB value in the frozen kernel, no viewer +- +13 ms | per viewer on top (10 viewers: 130 ms) +- 17 / 33 / 101 ms | a tiny SQLite commit after 0 / 10 / 50 earlier deltas +::: + +Nearly all of that came from layers around Delta: whole-value validation on every access, whole-value freezing, +JSON-ness walks on every received frame, double stringification to measure a frame, snapshots cloned per viewer, and +a SQLite storage that replayed history, and schemas rebuilt for every check (a compiled TypeBox check takes +0.012 ms, an interpreted one 5.6 ms). The redesign removes each of them (chapters V, VIII and this one); the +before/after per hop is in `apps/p2/docs/p3-reviews/v2-cost-model.md`. + +:::review | Review: v2-cost-model.md is partly outdated +The measurements and the per-hop "after" targets in `apps/p2/docs/p3-reviews/v2-cost-model.md` are current. Its +closing section "Decisions this needs" is not: items 6 and 7 (validation by the caller's trust; contract descriptions +once per link and fingerprint) were replaced by R7 (validate for every caller) and R6 (no descriptions on the wire), +and §10 still describes the description fetch. +::: + +### Cost guards + +Chord reports **full-value walks, deep copies, encodes, schema validations and materializations** as ordinary +telemetry metrics, always on; with the NOOP adapter they cost an empty call where such work happens. Tests use the +in-memory adapter and assert them per hop. The reference case: one appended note to a 500 KB document with 10 viewers +does no walks, no copies, one encode per hold part at its origin and none at relays, and no validations. A +`vitest bench` suite measures the hops. + +### Ways to make it slow anyway + +| Mistake | Why it costs | Instead | +| --- | --- | --- | +| a derive that builds new objects for unchanged data | every commit diffs and sends O(value) | keep unchanged items identical: `mapShared`, `previous` (chapter VIII) | +| one long array that changes often (R24) | sharing is per path; each change copies one slot per element and visits each | keyed documents or a bounded window | +| bytes as base64 strings | inflated, parsed, compressed as text | `bytes()` leaves; chunked transfers for files | +| awaiting I/O inside `self.tx` | the object's line is blocked for every caller | do the I/O first, then a short transaction | +| storing registry entries or references in documents | stale after a reload, not data | look up the current value at use | +| opening natives at module level or in a facet | no owner: never closed, never restarted | `self.open` in an object (chapter IX) | +| callbacks as arguments | cannot cross a link, keep old code alive | registries, jobs, flows | +| one big object that everybody views | one large snapshot per viewer and per reconnect | split by viewer need; views of the same variant share one value | + +## Telemetry + +Chord reports traces, events and metrics through `@earendil-works/pi-telemetry`. Each host gets **one adapter**, +`createHost({ telemetry })`, NOOP by default; the adapter decides sampling, buffering, export and filtering. Only hosts +hold the adapter, to start root spans. Code sees spans: + +- `ctx.telemetry` is the current span (`Ctx` extends `Context`, so it is the same field Durable and pi-ai read). Every + span carries its adapter's metrics and propagation, so code with a `ctx` can record a metric or inject a carrier. +- `ctx.trace(name, attributes, fn)` runs `fn` in a child span with a child `ctx`. +- App values such as the principal stay out of Durable's context. + +**Across links**, `call` and `hold` carry the caller's trace carrier as a bounded header field (R5: limited key count +and string lengths); the provider runs the method under it, and each relay adds a `chord.relay` span. The call from the +sequence above, as a trace: + +```text +chord.call Notes.add [owner's tab] +└─ chord.relay [server] + └─ chord.method Notes.add [session:S1] principal=owner + · event chord.commit docs=2 ops=5 +``` + +Adapters trust typed input: no validation of malformed values and no try/catch around recording. Only carriers from +another process are checked (keys present, strings), and a bad one falls back to a new root. + +**Shared work** (one activation serving several `get`s, a worker start) is its own root span, linked to the trigger +that started it; each waiting span links to it. Routine runs, job runs, eviction, backoff retries and background +closes are root spans too, linked to their trigger where there is one. + +| Tier | What | +| --- | --- | +| spans | calls, methods, relays, holds, activations, opens, closes, loads, revocations, resolves, routine runs, job runs | +| events | commits inside methods, retries, parking | +| metrics | state messages, WebSocket bytes, commit durations, in-flight calls, active holds, open links, park durations, backpressure waits, the cost guards | + +Ids are attributes of spans, never of metrics. **Nothing is measured or walked for telemetry**; only sizes the code +already has are reported. Errors carry a fixed category and a message cut to 500 characters; cancelling is not an +error. Chord ships a typed telemetry schema, as pi-ai (`AI_TELEMETRY_SCHEMA`) and Durable +(`DURABLE_TELEMETRY_SCHEMA`) do. + +**Storing and reading:** `@earendil-works/chord/telemetry-sqlite` over `chord/sqlite` (Node and Cloudflare) stores one +row per settled span (events and links inline) and metrics aggregated per series per 10 s, flushes in batches every +second, keeps 7 days, and drops (and counts) beyond a bounded queue. The read interface `TelemetryStore` is shared: +`findTraces` (by name, attributes, errors, time window, with limit and cursor) returns complete traces, `getTrace` +one, `findMetrics` metrics. One renderer, `pi-telemetry/render`, prints traces as above; it prints links and never +follows them. The playground's trace viewer reads the same store. + +**Durable** already emits submit and phase spans, stores carriers on submissions and tasks (`TaskOptions.trace`), +and pi-ai emits `pi.ai.request` and `pi.ai.poll`. With the harness given the host's adapter (chapter XIV), an agent +tool that calls `Notes.add` through `chord(fn, context)` shows up as one tree from the model request down to the +commit. + +:::note | Built so far +Nothing of the protocol is built in v2: messages, relays, flow control and limits are layers L6 and L7, telemetry in +Chord is L11. Built in pi: `pi-telemetry` (propagation, links, metrics, the store interface, the renderer), +`chord/context` with `Context.telemetry`, pi-ai and Durable telemetry, and `chord/telemetry-sqlite`. Delta is built; +`mapShared` and bytes as atomic leaves in `diffRevisions` are still to do. The frozen kernel sends descriptions on the +wire and closes a link on an unknown id; both are replaced. +::: diff --git a/packages/chord/docs/next/explainer/src/18-p2-core.md b/packages/chord/docs/next/explainer/src/18-p2-core.md new file mode 100644 index 000000000..51925055d --- /dev/null +++ b/packages/chord/docs/next/explainer/src/18-p2-core.md @@ -0,0 +1,324 @@ + +# Building p2: the Core {#p2-core lede="From a list of what the app must do to a set of owners: the base code every p3 plugin builds on."} + +The notes plugin used every Chord concept. A real app is the same concepts many times over. This chapter and the +next two assemble **p3**, the Chord v2 version of p2, from the parts the guide introduced. p3 is built next to p2 in +`apps/p3`, fully isolated with its own data directory, and **without migrating p2's data** (no import of +`catalog.sqlite`, `shares.sqlite` or the subagents table). Its plugins are derived from what the app must do, not +ported one to one from p2's plugin layout. + +:::review | Review: p3 is not designed yet +The design fixes the rules (`design.md` §25) and lists the p3 decisions still open (#17, #18, #20 to #23); the +per-domain design of p3 has not started. What these three chapters show is one construction that follows the rules, +derived from the functionality map in `apps/p2/docs/p3-functions/`. Object names are working names. Where a domain +has an open decision, the text says which. +::: + +## Start from functions, not from plugins + +The functionality map in `apps/p2/docs/p3-functions/` lists what p2 does, in plain words, with four facts per +function: the data and its size, how durable it is, who may see or do it, and where it must be available. It groups +the functions into 22 domains: + +| # | Domain | # | Domain | +| --- | --- | --- | --- | +| 1 | sessions | 12 | access | +| 2 | session workers | 13 | models, credentials, settings | +| 3 | conversations and agent | 14 | prompt resources | +| 4 | shell commands | 15 | status and presence | +| 5 | chat mode | 16 | search | +| 6 | agents (subagents, one-shots, peer messages) | 17 | remote access and forwards | +| 7 | code mode | 18 | logs and notices | +| 8 | session files | 19 | dev and reload | +| 9 | environments and execution | 20 | web shell | +| 10 | workspace | 21 | TUI shell | +| 11 | terminals | 22 | transport | + +The four facts answer the four questions every Chord object needs, in order: + +:::flow wide animate | From a function in the map to code. Each fact in the map answers one question Chord needs answered. +row "The map says" +box data "data" "what, how big" +box dur "durability" "ephemeral · until restart · persisted · transcript-bound" +box who "who" "owner · guest read · guest write · worker" +box where "where" "server · worker · web · TUI · environment" +row "Chord needs" accent +box owner "the owner" "smallest object it belongs to" accent +box store "where it is kept" "Durable, persisted or in-memory document, tab" +box variant "variants" "and revocation" blue +box host "host and code" "placement; core or plugin" +arrow data -> owner +arrow dur -> store +arrow who -> variant +arrow where -> host +note "Durability decides storage: transcript-bound data is Durable's; persisted app data is a persisted document with a stable identity; until-restart data is an in-memory document or a native; tab-scoped UI state stays in the client." +::: + +**One owner per thing, the smallest object it belongs to.** A guest's grant belongs to the access owner, not to the +session; a draft belongs to one (conversation, author) pair, not to the chat plugin as a whole. Ownership nests, and +rules set at creation are written in the creating commit (Durable's `createConversation({ init })`). + +**Core or plugin.** Core owners are what the app always needs and everything builds on; they are always present, so +other code may depend on them directly. Plugin owners may be missing; anything that depends on an optional feature +does so through a registry it owns itself (chapter XIII). The risk to avoid is a **core god object**: each core owner +owns only what belongs to it. + +## The ownership map + +The result of that procedure for p3, as a starting point: + +| Domain | Owner object(s) | Host | Code | +| --- | --- | --- | --- | +| access | `Access` (grants, devices, share links), `authenticate` | server | core | +| sessions | `Catalog` (session records), lifecycle jobs | server | core | +| session workers | the supervisor (`route(Session)` + idle stop) | server | core (app code) | +| session location | `SessionLocation { session }` | worker | core | +| conversations and agent | `Harness`, `ConversationStore`, `Conversation` views | worker | core | +| status and presence | `SessionStatus` (server copy), reporters on workers | both | core | +| shell commands | `ShellJob` in the chat plugin | worker | plugin | +| chat mode | chat plugin | worker | plugin | +| agents | subagents plugin (Durable child conversations) | worker | plugin | +| code mode | code mode plugin (sandbox native) | worker | plugin | +| session files | `SessionFiles { session }` (proposed, #17) | server | core or plugin, open | +| environments | envs plugin (config, SSH pools #20) | server and worker | plugin | +| workspace | workspace plugin | worker | plugin | +| terminals | terminal host process (#23), terminals plugin | separate process | plugin | +| models, credentials | models plugin, login job + flow | server | plugin | +| settings | `Settings` (single writer, #21) | server | core | +| prompt resources | prompt plugin (watchers in the session's environment) | worker | plugin | +| search | search plugin (index on the server, feed on workers) | both | plugin | +| remote access, forwards | radius plugin, ports plugin | server | plugin | +| logs and notices | `Notices` (one channel), log capture | server | core | +| dev and reload | the reload control, `reload_plugins` tool | server | plugin | +| web and TUI shells | UI facets over the objects above | clients | shells | + +The rest of this chapter builds the core rows. Chapters XIX and XX build the plugins. + +## The base code + +```text +apps/p3/ + src/ + hosts.ts Server, Session host kinds + principal.ts P, Revocation, principalKey, revokedBy, can() + server/main.ts createHost, load, route(Session), WebSocket and Radius entry points, supervisor + worker/main.ts createHost with the start data, connect to the parent + core/ the core facets: Access, Catalog, SessionStatus, Settings, Notices (server); + SessionInfo, SessionLocation, Harness, ConversationStore (worker) + plugins/ chat, subagents, codemode, envs, workspace, terminals, models, login, + prompt, search, ports, radius, files, dev +``` + +`hosts.ts`, `principal.ts`, the `SessionInfo` start data and the `Harness` object are the ones the guide already +built for Studio (chapters I, III, IX and XIV). Base code restarts the hosts when it changes; everything in +`plugins/` reloads. + +## Who is calling: authentication + +Chord stamps principals; the app produces them. p3's server has four ways in, each turned into a principal by +`authenticate(req)` when a WebSocket connects (chapter III): + +| Way in | Principal | +| --- | --- | +| loopback without a cookie (TUI, CLI, first page load) | `{ kind: "owner", access: "local" }` | +| loopback owner browser cookie (a per-port secret file) | `{ kind: "owner", access: "local" }` | +| a device cookie (HMAC-signed, per port, 30 days), obtained with a one-time sign-in token | `{ kind: "owner", access: "device", device }` | +| a guest cookie from a share link | `{ kind: "guest", session, mode, grant, revision }` | +| the Radius relay channel | `{ kind: "owner", access: "relay" }` | + +The checks that are not Chord's stay in this function: origin checks and Fetch Metadata (a browser request from +another site gets no principal, except a top-level navigation), proxy trust, the sign-in token rules (10 minutes, +single use, at most 32 outstanding, memory only), cookie expiry and revocation lists. A failed check closes the socket +before Chord sees it. + +**One permission matrix.** Every function in the map names who may use it, and p2 checks those rules in several +places. p3 writes them once, in `principal.ts`, and every factory and `authorize` uses it: + +```ts +// src/principal.ts +type Action = "read" | "send" | "upload" | "stop" | "workspace" | "terminal" | "manage"; + +export function can(p: P, action: Action, session?: string): boolean { + if (p.kind === "owner" || p.kind === "server") return true; + if (p.kind === "worker") return action !== "manage" && p.session === session; + if (p.kind !== "guest" || p.session !== session) return false; + if (action === "terminal" || action === "manage") return false; // terminals: owner only + if (p.mode === "read") return action === "read"; // no workspace for guest read + return true; // guest write: send, upload, stop, workspace +} +``` + +Factories pick variants from it (`can(p, "send", session) ? "writer" : "reader"`), so the matrix is visible in one +file and the members a caller lacks are simply not there. + +## Access: grants, devices, share links + +`Access` is the core owner of everything that lets someone in: share links per session and mode, guest records, the +device list and the signing key. It lives on the server as the chapter IV pattern: a `local` store that persists the +documents (only stable identities may), and a `remote` view for the owner's settings screen: + +```ts +// src/core/access.ts (server, in the core server facet) +const Links = defineState("access.links", { schema: map(ShareLink), version: 1, persist: true, + initial: () => ({}), checkpointWhen: (_v, _o, info) => info.deltasSinceBase > 64 }); +const Guests = defineState("access.guests", { … }); // same shape: map(Guest), persisted +const Devices = defineState("access.devices", { … }); // map(Device), persisted + +host.serve(AccessStore, (_input, self) => { // local: may persist, acts for the host + const links = self.state(Links), guests = self.state(Guests), devices = self.state(Devices); + return { + links, guests, devices, + removeGuest: async ({ grant }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(guests))[grant].blocked = true; }, ctx); + host.revoke({ grant }); // live access ends now + }, + resetLink: async ({ session, mode }, ctx) => { + const blocked = await self.tx((tx) => rotateLink(tx, links, guests, session, mode), ctx); + for (const grant of blocked) host.revoke({ grant }); + }, + signOut: async ({ device }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(devices))[device].revoked = true; }, ctx); + host.revoke({ device }); + }, + … // shareLink, join, signOutAll + }; +}); + +host.serve(Access, (_input, principal, self) => { // remote view: owner only + if (!can(principal, "manage")) throw forbidden(); + const store = self.get(AccessStore, {}); + return { + variant: "owner", + guests: self.derive("guests", [store.guests], ([g]) => publicGuests(g)), // names, modes; no tokens + removeGuest: (args, ctx) => store.removeGuest(args, ctx), + resetLink: (args, ctx) => store.resetLink(args, ctx), + signOut: (args, ctx) => store.signOut(args, ctx), + }; +}); +``` + +`authenticate` is base code on the same host and checks `AccessStore`'s current documents on every new connection +(it holds a reference to the store, taken with `server.root` at start). Live access and lasting denial stay separate, as chapter VII explained: + +| Event | Lasting effect (data) | Live effect | +| --- | --- | --- | +| remove a guest | guest record blocked | `revoke({ grant })` | +| reset a share link | new token; its guests' grants blocked | `revoke({ grant })` for each | +| archive a session | grants of that session void while archived | `revoke({ session })` | +| restore a session | grants valid again; revision bumped | none: guests reconnect | +| sign out a device | device revoked until its expiry | `revoke({ device })` | +| sign out all devices | signing key rotated, tokens cleared | `revoke({ device })` for each | + +A guest who rejoins with the same browser keeps its identity; a removed guest cannot mint a new one, because the +block is in `Guests` and `authenticate` refuses it. Names are unverified, cleaned, at most 40 characters, unique per +session ("Ann (2)"). + +## The catalog + +`Catalog` owns the session records: id, title (and whether it was edited), location, model default, pinned, +archived, timestamps. It lives on the server, persisted, and it **never starts a worker**: listing sessions, showing +their status or their unread marks reads server-side copies only. + +```ts +export const Catalog = remote("p3.catalog", { + version: 1, + input: obj({}), + placement: () => Server(), + variants: { + owner: { + state: { sessions: obj({ items: arr(SessionRow) }) }, + methods: { + create: method(CreateArgs, str()), rename: method(RenameArgs), pin: method(PinArgs), + archive: method(obj({ id: str() })), restore: method(obj({ id: str() })), + }, + }, + guest: { state: { sessions: obj({ items: arr(SessionRow) }) } }, // only the guest's own session, if not archived + }, +}); +``` + +Behind it sits a persisted `local` `CatalogStore`; the guest variant is a derive over it, as in chapter IV. The lifecycle operations (archive, clone, fork, +relocate, delete) are chapter XIX; here only the rule that keeps the catalog small: **the catalog does not know which +plugins exist.** Archiving a session writes `archived: true` and revokes the session; every plugin that keeps +per-session things reconciles from `catalog.sessions` with a routine (chapter X), as the notes plugin did. The ports +plugin closes forwards of archived sessions that way, also after a crash. + +## On the worker: start data, location, harness + +Three core objects exist on every worker from its start: + +- **`SessionInfo`** (chapter IX): the start message as `constant(init)`, with the session id, data directory and the + location the server chose at start. +- **`SessionLocation { session }`** (working name), lifetime host: the current cwd and environment, with a location + revision. Relocation changes it while the worker runs (chapter XIX); natives that depend on the location follow it + with a routine and a generation (chapter IX), and conversations that follow the session's cwd are listed in the core + per-conversation policy document. +- **`Harness`** (chapter XIV): the Durable harness for the session's store, one SQLite database per session, opened + under an exclusive lock keyed by the real data directory and session id. It refuses to start when the saved root + model is not configured, and resumes interrupted Durable work on start. + +`ConversationStore` and the `Conversation` views in front of it are chapter XIX. + +## Workers: start, keep alive, stop + +Starting, supervising and stopping workers is app code (chapter I); Chord only routes and counts. p3's supervisor is +`server.route(Session, { authorize, resolve })` plus a small server object that keeps per-session worker state: + +:::steps | A worker's life, as the supervisor sees it ++++ Authorize, then start +A tab gets `Notes { S1 }`. `authorize` checks `can(p, "read", "S1")` and that the session exists and is not archived +(an archived session refuses, except an index rebuild that starts the worker without resuming work). `resolve` forks +`worker/main.ts` with the start data from the catalog, bounded by a 60 s readiness timeout. ++++ A start failure sticks +If the worker fails to start (an unconfigured environment, a missing model), the error is kept per session and +every `get` fails at once with it, until the owner presses Retry. That clears the error and the related notices. +Guests see that the session is unavailable, not the error text. ++++ Counting +The core facet on the worker reports `host.activity` (`{ references, calls, jobs, lingering }`) and the harness's +running Durable task count to the supervisor object, from a routine whenever either changes. The call carries +`ctx.incarnation`, so a report from an older worker incarnation is ignored (chapter X). ++++ Idle stop +When everything has been zero for five minutes (checked every second, as in p2), the supervisor stops the worker: +the worker closes its host (routines stop, natives close, storage flushes) and exits. Durable work stays persisted +and resumes on the next start. Detached work that must keep the worker alive (a shell command whose caller left) is +a job, so it counts. ++++ Crash +If the worker exits on its own, the link drops. The supervisor records the exit code and a redacted stderr tail +(at most 4,000 characters) for that incarnation and posts one notice. References that still exist make Chord call +`resolve` again with backoff; a passive viewer recovers without a click. ++++ Shutdown +On server shutdown the supervisor stops every worker, with a forced exit after three seconds. Durable work remains +persisted. +::: + +## Status without a worker + +The session list shows per session whether it is running, busy (which conversations), its task count, usage and cost, +and an error. None of that may start a worker. The pattern is chapter X's pub/sub: a reporter on each worker (a +routine over the harness's state) calls `SessionStatus.report(summary)` on the server; `SessionStatus` keeps the latest +value per session and ignores calls whose `ctx.incarnation` is older than the newest it has seen; clients read +`SessionStatus`, which lives on the server. Children lists, unread marks, search index rows and artifact lists use the +same pattern (chapter XIX). + +## Settings and notices + +**Settings** (default model and thinking, run settings, compaction, queue modes, code mode switch, owner name) have +**one writer**: a core `Settings` object on the server owns the file, validates writes and serializes them; every +other process follows its state through a reference instead of watching and rewriting the file itself. p2 watches +settings, models and auth in every process and writes them from two, which is decision #21. + +**Notices** are one channel: one core `Notices` object on the server, written by everything that fails or finishes +(worker crashes, reload results, model errors, login outcomes, grants), keeping one latest notice per source, at most +50. The owner's shell reads one list; dismissing is a method. p2 has five notice channels for the same thing. + +:::review | Review: settings and file formats (#21, #22) +Single writer for settings, models and auth (#21) and versioning on-disk formats such as `envs.json` like documents +(#22) are open p3 decisions in `design.md` §26. The construction above assumes "one owner on the server, others +follow its state" for #21. +::: + +:::note | Built so far +Nothing of p3 exists. The functionality map is complete (`apps/p2/docs/p3-functions/`, five area files plus the +condensed `README.md`), and the design's app rules (`design.md` §25) are decided. p3 starts after the layered rebuild +and the playground's first real proof. +::: diff --git a/packages/chord/docs/next/explainer/src/19-p2-sessions.md b/packages/chord/docs/next/explainer/src/19-p2-sessions.md new file mode 100644 index 000000000..9d40c2287 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/19-p2-sessions.md @@ -0,0 +1,276 @@ + +# Sessions and Conversations {#p2-sessions lede="The session lifecycle, status without a worker, chat, shell commands, subagents and search, each mapped to an owner and the Chord parts it uses."} + +This chapter builds the domains a person touches most: the session list and its operations, the conversation view +and composer, shell commands, chat mode, subagents, messages between sessions, search and unread marks. Each section +names the owner, the host, and the Chord parts it uses; code appears only where it shows something new. + +## The session lifecycle + +The catalog (chapter XVIII) owns the session records. Its operations, by the Chord part each needs: + +| Operation | Owner and form | Notes | +| --- | --- | --- | +| create | `Catalog.create`, a method | absolute directory, `~` expanded, must be a directory; on an environment, resolved there through the envs plugin; refuses an unknown model; model and thinking default to the last used | +| rename, pin | methods on `Catalog` | title 1 to 120 characters, whitespace collapsed; an explicit title is never replaced automatically | +| automatic title | a routine on the search feed | the first user message, line breaks to spaces, 100 characters | +| agent rename | an agent tool on the worker calling `Catalog` | root conversation only | +| archive, restore | methods; revocation; reconciling routines | below | +| clone, fork, relocate | jobs on the catalog store | outlive the call, cancellable, one outcome | +| delete | a job; decision #18 | below | + +### Archive and restore + +Archiving must stop the worker whatever it is doing, end guest access, and close the session's port forwards, while +keeping its storage. The catalog does none of the plugin-specific parts: + +:::steps | Archiving S1 ++++ One write +`Catalog.archive({ id: "S1" })` sets `archived: true` in the session record, in one transaction. ++++ Revoke the session +The catalog calls `host.revoke({ session: "S1" })`. Every guest principal of S1 has `{ session: "S1" }` in its +`revokedBy`, so their objects end and their links close on the server and on the worker (chapter VII). New +connections are refused by `authenticate`, because grants of an archived session are void. ++++ Stop the worker +The supervisor reconciles from `catalog.sessions` with a routine: an archived session's worker is stopped, and +`authorize` refuses to start it again. ++++ Plugins reconcile +The ports plugin's routine on `catalog.sessions` closes forwards of archived sessions. The search plugin marks the +session's rows archived. If the server crashed between the write and these steps, the routines run again with the +current value at the next start: nothing is a notification that can be lost. ++++ Restore +`Catalog.restore` clears the flag. Grants are valid again, so guests reconnect and authenticate as before; forwards +stay closed. +::: + +### Clone, fork, relocate + +Each is work that outlives its call, can be cancelled and has one outcome, so each is a **job** (chapter XI) on the +catalog's local store, keyed by the owner's request id. They differ in what they pause: + +- **Clone** copies the session's store, uploads, downloads and artifacts under a new id, with the source worker + paused (refused while work runs) and title "(copy)"; directories `0700`, files `0600`, no links, no overwrite; a + partial copy is removed on failure. +- **Fork** before a user message copies the active context as of the entry before it into a new session (title + "(fork)"), with the message waiting in the new session's editor. It needs two workers: the source reads history, + the new one is seeded empty. A failure archives the copy. +- **Relocate** (cwd, environment or both): refused while the session is busy (naming the running subagents) or when + a subagent has its own cwd and the move crosses machines. The job fences the worker (new admissions wait), updates + the record, and moves the running worker in place by updating `SessionLocation`: natives that follow the location + switch with a generation (chapter IX), and conversations that follow the session's cwd are rewritten in one Durable + commit before work resumes. After 60 s it restarts the worker instead; on failure it releases the fence. + +:::review | Review: jobs as objects (#9) +The design defers whether a long relocation or a login is a record inside its owner (watched through `running`) or an +object of its own (`Relocation { id }`). The construction above uses records. Decide when login is ported +(`design.md` §15, §26). +::: + +### Delete + +p2 has no session delete. p3 needs one that removes the session everywhere: every document tagged with the session +(`store.retireByTag(SessionTag, id)`, chapter V), the worker's databases (the Durable store), the session's files, the +search rows, read marks, children copies, forwards, guest records, and finally the catalog entry. The design commits to +this outcome (`design.md` §25); how plugins take part is open. + +:::review | Review: session delete (#18) +Options in `design.md` §26: per-session objects plus `SessionTag` plus one session directory (delete is retire by tag +and remove a directory, plugins need do nothing), or owner hooks contributed through a catalog registry. Also open: +whether the search index is per session (deleted with it) or global with a hook. Who holds the store for +`retireByTag` is not specified either. +::: + +## Status, children, unread: reports to the server + +The session list must show status, subagents and unread marks for sessions whose workers are not running, and must +not start them. All three follow chapter X's pub/sub pattern: a routine on the worker calls a server object with the +current summary, the server object keeps the latest value per session (persisted where it must survive a restart), +ignores calls from an older `ctx.incarnation`, and clients read the server object. + +| Summary | Source on the worker | Kept on the server | +| --- | --- | --- | +| status: running, error, task count, busy conversations, usage and cost | the harness's task graph and usage | `SessionStatus`, in memory: a stopped worker is "not running" | +| children: name, model, thinking, tools, cwd, last entry, archived | the subagents plugin's child list | `Children`, persisted | +| latest root answer | the search feed | `Unread`: compared with the owner's read mark, persisted, shared by all owner devices | + +Guests see their own session's status without the error text. The owner's read mark is written when the root +conversation is visible, and a new session starts as read. + +## Conversations + +The harness (chapter XIV) owns the Durable store. The UI needs, per conversation: a live window of entries, older +pages, usage numbers, the working indicator, the inbox (queued inputs), and the actions. Two core objects provide them: + +- **`ConversationStore { conversation }`** (local, on the worker): opens the Durable watch as its native, reaching + the harness through the core-private `HarnessRuntime` registry entry, and exposes a **window** as a derive: the + newest entries (at most 100 or 512 KiB, at least 20) plus the conversation's documents. The window's start stays + put while it is within 1.5 times the limits, so new entries arrive as small appends instead of a shifted list + (chapter VIII's window pattern). +- **`Conversation { session, conversation }`** (remote, placement `Session`): the per-principal view. Variants by + `can()`: `owner` (everything, including model, thinking and handoff), `writer` (guest write: send, steer, stop, + compact, shell), `reader` (the window only). + +```ts +host.serve(Conversation, ({ session, conversation }, principal, self) => { + const store = self.get(ConversationStore, { conversation }); + const policy = self.get(ConversationPolicy, { conversation }); // core: readOnly, followsSessionCwd + const author = authorOf(principal); // structured: kind, id, display name + if (!can(principal, "send", session)) return { variant: "reader", view: store.window, page: store.page }; + return { + variant: can(principal, "manage") ? "owner" : "writer", + view: store.window, + page: store.page, // older history: returns data + send: ({ text, attachments, mode, requestId }, ctx) => { + if (policy.value.readOnly) throw forbidden("read-only conversation"); + return store.submit({ type: "input", content: text, attachments, author, whenBusy: mode, requestId }, ctx); + }, + stop: (_args, ctx) => store.stop(author, ctx), // allowed even when read-only + … + }; +}); +``` + +What each function becomes: + +| Function | How | +| --- | --- | +| send, retried send | Durable `submit` with the author's `requestId`: a retry with the same id is admitted once (Durable dedupes, chapter XIV) | +| steer or queue while busy | `whenBusy: "steer"` or a follow-up; the inbox is `docs["pi.inbox"]` in the window | +| attribution | the view derives the author from the principal; the entry stores it structured, the model sees a speaker block; nothing parses text | +| wait for prompt resources | the first input of a fresh worker waits until the prompt plugin's resources are loaded (a registry readable the store follows) | +| stop, dequeue | withdraw queued inputs, abort the run; the caller's texts return to its editor, other authors' to their **restored drafts** | +| restored drafts | a keyed document per (conversation, author), tagged `SessionTag`, persisted in the store; claimed once, first tab wins | +| read-only conversations | the core per-conversation policy document, written at `createConversation({ init })` by whoever creates the conversation; send, dequeue, configure, compact and shell refuse; stop is allowed | +| history pages | a method returning a page (100 entries, 1 MB, 2,000 scanned); compaction and reset markers always included | +| model and thinking | owner variant only; validated against the models plugin's catalog; the root conversation's model is also reported as the session default | +| compaction, handoff | Durable operations through the store | + +**Drafts in the composer** are UI state, not Chord state: the web shell keeps one draft in `sessionStorage`, the TUI +keeps one per conversation across UI replacements, and pending sends keep their request id until acknowledged +("Sending when connected"). Only restored drafts, which the worker produces, are documents. + +## Shell commands + +`!cmd` runs a command in the conversation's cwd on its environment and records the result as an entry; `!!cmd` does +the same but keeps the output out of the model's context. One command at a time per conversation; the live tail is +at most 50 KB or 2,000 lines, sampled every 200 ms; the full output spills to a file; the result entry is placed at the +run's next boundary while the agent is busy and never starts a run. This is the design's job example almost as is: + +```ts +const ShellJob = defineJob("chat.shell", { + version: 1, + args: obj({ command: str({ max: 100_000 }), include: bool(), author: Author }), + progress: obj({ output: str() }), + result: ShellResult, + concurrency: 1, // one per conversation +}); + +const results = self.state(ShellResults); // keyed by run id: dedupe after completion +const shell = self.job(ShellJob, { + run: async ({ command, include, author }, ctx) => { + const out = new OutputBuffer({ maxBytes: 50_000, maxLines: 2000, retain: "tail" }); + const env = await environmentFor(ctx); // the conversation's environment, chapter XX + const r = await env.exec(command, { cwd: env.cwd, signal: ctx.signal, spill: spillPath(ctx.jobId) }, + (chunk) => { out.push(chunk); ctx.progress({ output: out.snapshot().text }); }); + const data = { command, output: out.snapshot().text, exit: r.code, cancelled: ctx.signal.aborted }; + await conversation.submit( + { type: "write", entry: shellEntry(data, author, include), requestId: `shell:${ctx.jobId}` }, ctx); + return data; + }, + finish: async (done, tx) => { + (await tx.draft(results, done.id)).outcome = done.kind === "done" ? done.result : { error: done.error }; + }, +}); +``` + +The request id `shell:${ctx.jobId}` makes the entry idempotent: an attempt interrupted after the write cannot write +it twice. Cancel is `shell.abort(id)`, allowed for the command's author and the owner. A retried `runShell` with the +same request id returns the running job (dedupe of unsettled runs), or the stored result (the owner's own dedupe after +completion, chapter XI). p2's "cancel when nobody watched for 30 s" is dropped: a job keeps its owner alive and does +not depend on viewers. + +## Chat mode + +Chat mode is a per-conversation switch that everyone in the session sees and only the owner changes; forks and new +subagents start with it off. Because it must fork and rewind with the conversation, it is **transcript-bound**: a +Durable conversation document, not a Chord document. Messages without `@pi` become display-only Durable `write` +entries (author, text, attachments, reply quote); a message with `@pi` becomes an input with a digest of the chat +since the last mention (at most 100 messages, 20 files, 4 images), recording which entries it passed on. The reply +quote is derived on the worker from the entry id the client names: the client never supplies author or quote text. + +## Subagents and one-shot agents + +Subagents are Durable child conversations owned by tasks, as in Durable's own background-subagent example: one +`subagent` tool spawns, steers, waits for, stops and lists them; each child is owned by a background anchor task, and +each message is delivered by a reporter task that posts the answer back to the parent once, with request ids so a +restart sends nothing twice. Chord adds only three things: + +- the **child list** for the sidebar is reported to the server (`Children`, above), so it shows without a worker; +- a child that follows the session's cwd, or a read-only one-shot agent created by a code-mode script, is marked in + the core **per-conversation policy document** when it is created, so chat and relocation enforce it without knowing + the subagents plugin (`design.md` §25 porting guideline); +- archive and restore of a child from the UI are methods on the subagents plugin's view, refused for one-shots. + +Subagents cannot spawn subagents; thinking levels are clamped to what the model supports, in one helper shared with +the session's model choice. + +## Messages between sessions + +The root agent can list other sessions and send one an attributed message. The tool runs on the sender's worker and +calls a server object through `chord(fn, context)` (chapter XIV): + +```ts +const peerSend = defineTool({ + name: "session_send", + execute: (args, api, context) => chord(async (ctx) => { + const peers = await ctx.get(Peers, {}); // server, worker variant + await peers.send({ to: args.session, text: args.text, mode: args.mode, requestId: `peer:${api.taskId}` }, ctx); + return { output: [{ type: "text", text: "sent" }] }; + }, context), +}); +``` + +`Peers` checks that the sender is a root conversation of a non-archived session, refuses sending to itself and +archived recipients, then gets the recipient's `Conversation` (which starts its worker) and submits with the +sender as a structured author and the same request id (the tool task's id, stable across restarts), so a retried +tool call delivers once. The sender's +incarnation is `ctx.incarnation`; a call from a replaced worker is refused. Only owner-initiated turns may use the +tool; guests never. + +## Search + +The search plugin indexes user and assistant messages of root and subagent conversations (no shell output, no +Durable internal inputs) for title, cwd and message search across sessions, active or archived. + +:::flow wide animate | The search feed. The worker pushes pages from a cursor; the server owns the index and never starts a worker to answer a query. +row "Worker S1" blue +box commit "Durable commit" "in any conversation" +box wake "wake" "a routine on the harness's commit signal" +box page "read from cursor" "200 entries per page" +row "Server" accent +box index "SearchIndex" "native: SQLite FTS" accent +box cursor "cursor per conversation" "moves after the whole range" +box query "query" "terms, scope, at most 41 hits" +arrow commit -> wake +arrow wake -> page +arrow page -> index "report(page)" +arrow index -> cursor +note "Reports carry ctx.incarnation. Indexing is idempotent: a page sent twice changes nothing." +::: + +The index is a native owned by a host-lifetime object on the server; cursors live with the index, so after a crash +the worker asks for the cursor and continues. Rebuilding one session's index drops its rows and rescans, starting +an archived session's worker without resuming its work. Queries are methods returning data: at most 41 hits, +snippets of 300 characters, queries up to 500 characters. The same feed sets automatic titles, moves "updated at" +forward and supplies the latest root answer for unread marks. + +## Presence + +Viewers per session are `host.references(Conversation)` on the worker, counted per client (chapter VIII), and +reported with the status. Connected guests (name and mode) are the references whose principal is a guest. Presence +is not keep-alive: the supervisor uses `host.activity`, which also counts calls and jobs. + +:::note | Built so far +None of this exists in p3. The parts it relies on are Durable (built: submit with request ids, steer and queue, +child conversations, background tasks) and the Chord layers L3 to L9 (not built). +::: diff --git a/packages/chord/docs/next/explainer/src/20-p2-machines.md b/packages/chord/docs/next/explainer/src/20-p2-machines.md new file mode 100644 index 000000000..51b9e7546 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/20-p2-machines.md @@ -0,0 +1,245 @@ + +# Machines, Files, Models {#p2-machines lede="Environments, workspace, terminals, session files, ports and Radius, models and logins, prompt resources, logs, reload and the shells: the rest of p3."} + +The last domains are the ones that touch the outside world: other machines over SSH, the file system and git, +terminals, the relay, model providers and their credentials. Most of them are native objects, and most of them are +places where p2 has the same thing several times (four SSH connections to one environment, five notice channels, +three copies of the environment config parser). The rule that fixes that is the one from chapter XVIII: one owner per +thing. + +## Environments + +The **envs plugin** owns the environment configuration: name, host, user, port, identity file, shell and its +variables, plus a managed ed25519 key per environment and its own `known_hosts` (kept apart from the user's, aliased +per environment name). + +| Function | Owner and form | +| --- | --- | +| list, add, edit, delete (refused while a session, archived too, uses it) | `Envs` on the server: a persisted document behind an owner-only view | +| validation (no leading `-`, no whitespace or control characters in host and user) | the view's methods, before the write | +| a hand-edited `envs.json` is picked up; an invalid file keeps the last valid set and shows an error | a native file watcher (poll fallback) in the envs store; one writer (#21) | +| environment revision: a hash of the config, part of its identity | a derive | +| connection test (connect, deploy the daemon, platform, latency, warnings) | a job; the result per revision in a document | +| unknown or changed host key: fingerprints, trust after confirmation, forget | a flow (chapter XII) on the server | +| browse directories (`~`, relative paths, at most 10k scanned, 1,000 shown) | a method with a **call-scoped** SSH session (chapter IX) or the pooled connection | +| a changed environment restarts its sessions' workers once idle | a routine on the envs store and the status reports | + +**Execution in the agent.** The worker's harness asks `HarnessEnvironments` for an environment per target (chapter +XIV). The envs plugin contributes it from its worker facet: local execution at the session's cwd, or a remote one over +SSH through the `pi-env` daemon, deployed per platform and connecting at the first operation. A remote session never +falls back to local tools. The environment follows `SessionLocation`: after a move, the old connection closes once +the move has committed and watchers reopen at the new location. + +```ts +// envs/worker.ts +export default defineFacet((host, self) => { + const location = self.get(SessionLocation, { session }); + const pool = self.get(EnvPool, {}); // one owner of SSH connections (#20) + self.contribute(HarnessEnvironments, (target) => + target.env === "local" ? localEnv(target.cwd) : pool.environment(target.env, target.cwd)); +}); +``` + +:::review | Review: connection pools (#20) +SSH connections to one environment exist four times in p2 (worker environment, browse connection, test connection, +terminal host). `design.md` §26 asks for one owner with a pool or deliberate per-use connections. `EnvPool` above is +the first option: a native object that closes idle connections after 10 minutes and invalidates them on a config or +key change. Reloading the plugin that owns the pool kills running remote commands (chapter XV), so it should change +rarely. +::: + +## Workspace and file index + +The **workspace plugin** serves the viewed conversation's working tree (a subagent's own cwd included) on the worker: +list a folder (at most 2,000 entries, `.git` hidden), read a file (1 MiB, binary detected), git status, diff, log and +single commits, literal text search, and all paths for name filtering. Every query is a **method returning data**, not +state: results are bounded and requested by the UI. Only change notifications are state: a counter in a document, +bumped by a native watcher of the tree, `HEAD`, the index and config (debounced 100 ms); viewers reload what they show +when it changes. + +Two rules live in one place each instead of three: + +- **Confinement:** path rules, a canonical root, no `.git`, the app's private data directory denied (also excluded + from search). One helper used by listing, reading, uploads and the file index. +- **Git hardening:** no optional locks, no fsmonitor, filters disabled, no prompts or pager, no system config, 10 s + timeout, output caps. + +Downloads (25 MiB, byte ranges for media) go through the HTTP component (chapter XIII) with a route on the server that +reads chunks from the worker; uploads into the project use the chunked protocol of chapter XIII (1 MiB chunks, temp +file then rename, never overwrite, writes serialized). Guest read gets no workspace: the view has no variant for it. + +The **file index** for `@` completion belongs to the same owner, so completion consumes the index instead of scanning +on its own: built from the environment's listing (else git or a walk, at most 50k paths), rebuilt at most once per +second while files change, discarded after a move. Review comments drafted on files and diffs are browser state until +sent; sending them is an ordinary `Conversation.send`. + +## Terminals + +Terminals must survive server and worker restarts, so their PTYs live in a **detached terminal host process**, one per +data directory, with its own process lock; it exits after 10 minutes with no terminals and no clients. The terminals +plugin on the server is the owner the UI talks to: list (title, cwd, environment, size, exit status), open at the +session's cwd and environment (refused for archived sessions), attach, close (asking first if a foreground job runs). +Terminals are **owner only**. + +Bytes do not go over object links. Attaching opens a **channel** (chapter XIII) with a single-use token (30 s): JSON +control on the object, bytes on the channel, with its own acknowledged window (256 KiB, acknowledgements per 64 KiB; a +viewer that falls behind gets a fresh screen snapshot; an 8 MiB backlog disconnects). Attaching gets the current +screen from a headless emulator with 10k lines of scrollback. + +:::review | Review: the terminal host process (#23) +The terminal host runs its own code. Is it part of the revision, reloaded or restarted with it (and running terminals +end), or deliberately outside it with a "host runs older code" notice as in p2? Open in `design.md` §26. +::: + +## Session files + +Attachments (25 MiB, at most 20 per message), sent-file snapshots made by the agent's `send_file` tool, versioned +artifacts (HTML, SVG, Markdown, text; 2 MiB; each update a new immutable version) and shell spill files are all +per-session bytes with caps, immutable ids and range reads; p2 has two upload routes, two caps and two directory +layouts for them. The proposal is one owner: + +```ts +export const SessionFiles = remote("p3.session-files", { + version: 1, + input: obj({ session: str() }), + placement: () => Server(), // readable without a worker + variants: { writer: { … }, reader: { … }, worker: { … } }, +}); +``` + +`SessionFiles { session }` on the server holds the index (per kind, readable without a worker), the chunked upload +and read protocol, the caps per kind, copying on clone and fork, and deletion with the session. Attachments are copied +into a remote environment before an input is admitted, with no fallback on failure. Artifact previews run in a +sandboxed frame with its own CSP and no app identity; ordinary file previews allow only safe types. + +:::review | Review: session files (#17) +Proposed in `design.md` §26, not decided. Open sub-points: one preview limit (p2 has 256 KiB for sent files and +512 KiB for artifacts), what a fork copies of the artifact list, and how attachments are copied into remote +environments. Related: #22, whether on-disk layouts such as downloads and uploads get versions like documents. +::: + +## Ports and Radius + +The **radius plugin** keeps the outbound relay connection, a native in a host-lifetime object on the server: sign-in +by device code from the CLI (credential in a private file), token refresh, a heartbeat in the link adapter (20 s ping, +60 s timeout; TCP does not notice a sleeping laptop), backoff from 1 to 30 s, a disable switch. The relay delivers two +kinds of channels: + +```ts +// radius/server.ts (the entry point: one line, as in chapter I) +radius.onChannel((ws) => host.accept(webSocketLink(ws), { principal: { kind: "owner", access: "relay" } })); +radius.onForward((channel) => proxyToForward(channel, host.registry(RadiusForwards).value)); +``` + +Object links become owner principals with `access: "relay"`: owner, but treated as remote (no localhost-only +powers). p2 had to fake a loopback request with a secret header for this; with `accept` that is gone. + +Forward channels go to `localhost:` (Host, Origin and Location rewritten; 502 when nothing listens; anonymous +visitors only to public forwards). Which forwards exist is not radius's business: it owns a registry, +`radius.forwards`, and the **ports plugin** contributes its forwards (at most 50, random ids, a public toggle that +needs confirmation, an agent tool `ports` for top-level owner-initiated conversations). Without the ports plugin the +table is empty and radius still works: an optional dependency is a registry owned by the dependent (chapter XIII). The +ports plugin closes forwards of archived sessions with a routine on `catalog.sessions` (chapter XIX). + +:::review | Review: may a plugin call `accept`? +`design.md` §3 shows the Radius line in the server's start code and calls entry points "app or adapter code". If +radius is a plugin, its facet needs the host's `accept`, which no other part of this guide gives a plugin. The +alternative is that the start code keeps the entry point and gets channels from the radius object. Not specified. +::: + +## Models and credentials + +The **models plugin** owns the model runtime (pi-ai's models with custom providers and the model cache) in a +host-lifetime object, and the credentials. It contributes the runtime to `HarnessModels` (chapter XIV), so after a +models reload the next request uses the new runtime while a running stream finishes on the old one. **Credentials +never go to clients**: the views expose providers, login methods and which type is signed in, never tokens. Every +secret is registered for redaction in one place, the log capture. + +Views: a model catalog for pickers (names, ids, context windows, thinking levels; at most 300 matches in the web), +logout (refused during a login), refresh a provider's model list. The default model and the cycling scope are +settings (chapter XVIII). + +### Login + +A login is the guide's flow example (chapter XII): a **job** on the server that runs pi-ai's login, and a flow that +shows its steps to the starting owner. + +```ts +const LoginJob = defineJob("models.login", { + version: 1, args: obj({ provider: str(), type: AuthType }), progress: LoginState, result: obj({ ok: bool() }), + concurrency: 1, // one login at a time on the server +}); + +const login = self.job(LoginJob, { + run: async ({ provider, type }, ctx) => { + await runtime.value.login(provider, type, { + signal: ctx.signal, // 10 minute deadline, cancel, leaving + prompt: (prompt) => steps.ask(prompt, ctx), // text, secret, select, manual_code + notify: (event) => steps.notice(event, ctx), // info, auth_url, device_code, progress + }); + return { ok: true }; + }, + finish: async (done, tx) => { /* record the outcome; refresh the model list */ }, +}); +``` + +`prompt` is a promise-based callback inside the work; the resolver stays in the job (`steps.ask` records the step in +the flow state and waits for `answer`), and nothing crosses an object boundary. On a relayed connection the device +code method is selected automatically. The flow is visible only to the owner principal that started it; others are +refused. Leaving ends it: when the starting owner's references are all gone (presence, not an unmount), a routine +aborts the job. Codes, URLs and typed secrets are redacted from logs. + +## Prompt resources + +`AGENTS.md`, skills and templates are read in the session's environment, watched (poll fallback, last valid state), +and expanded on the worker before sending. The **prompt plugin** owns them per session on the worker, contributes the +system prompt section and skills to `HarnessExtensions` as readable contributions (a change recomposes), and exposes a +readiness readable: the first input of a fresh worker waits until it is ready (chapter XIX). After a move the watchers +reopen at the new location. + +## Code mode + +Code mode is a plugin (chapter XIV): its `pi-codemode` sandbox is a native; scripts call tools as nested Durable calls +and replay after a restart; the tool entry looks up the current sandbox when it starts. A reload seals and drains the +old sandbox for at most 30 s. The on/off switch is a setting. + +## Logs and notices + +**Log capture** tees stdout and stderr of the server, workers and detached hosts without changing their behaviour, +redacts known secrets and credential patterns, and keeps bounded rings (at most 128 sources, 1,000 lines or 256 KiB +each). Detached hosts publish redacted snapshots to a private file. Logs are read without waking a stopped worker. +**Notices** are the one channel from chapter XVIII; dismissal stores only notice ids. + +## Dev and reload + +The dev plugin is chapter XV's reload control for p3: a source watcher marks plugins dirty and never applies anything; +the `reload_plugins` agent tool and a control in the owner's shell reload the changed plugins, or restart all hosts for +base code (through the supervisor, reopening the sessions that were running). UI bundles are published only after a +successful backend change; a web page detects a new manifest and reloads with its drafts and navigation state saved; +the TUI swaps its sandboxed UI code with a snapshot of its state. A local Unix socket (`0700`) accepts development +commands without a network admin port. + +## The shells + +The web and TUI shells are UI facets over the objects above. Two rules matter for Chord: + +- **UI state survives navigation, reload and reconnect on purpose.** Composer drafts, pending sends with their request + ids, reading positions and review drafts are kept in tab storage or the TUI's replacement snapshot. A reconnect + re-holds and gets snapshots; nothing is lost that was not in flight. +- **Leaving is never derived from an unmount.** A plugin reload remounts components (chapter XV). Cancelling a login, + closing an overlay or giving up a lease is an explicit action or a dropped reference. + +The TUI launcher keeps downloaded UI code in a permission-dropped worker with no access to files, network or +credentials; local side effects (copy, upload, download, open a URL, an external editor) need a real user gesture or a +confirmation. That is app code around Chord, not part of it. + +## Transport + +The last domain, transport, is chapter XVII: one message format, per-part limits (16 MiB compressed and 64 MiB decoded +on client WebSockets, instead of p2's 1 MiB input cap and 32 pending calls), one byte window per link, re-holds with +snapshots after a reconnect, and a protocol version that makes an outdated web page reload. + +:::note | Built so far +None of p3 exists. pi-ai's login API, pi-codemode and the `pi-env` daemon exist and are used by p2 today; the HTTP +component and channels are deferred until after the kernel's first real proof (chapter XIII). +::: diff --git a/packages/chord/docs/next/explainer/src/20a-compared.md b/packages/chord/docs/next/explainer/src/20a-compared.md new file mode 100644 index 000000000..aa4fcd2c2 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/20a-compared.md @@ -0,0 +1,146 @@ + +# Other Ways to Do It {#compared lede="Cordis and DeepSeek Harness attack the same problem, plugins that come and go while an agent runs, inside one process. What that buys, and what it costs."} + +Chord v2 is not the only answer to "replace plugin code while the app keeps running". The closest relative is +**Cordis**, a TypeScript plugin framework by Shigma that has been the plugin kernel of the Koishi chatbot framework +since 2019, and its most visible user, **DeepSeek Harness** (`dsh`), DeepSeek's open-source agent harness, whose +motto is "everything is a plugin". This chapter compares them with Chord v2 on the questions the guide cared about: +who owns side effects, how plugins depend on each other, what a reload keeps, what a crash keeps, and how far the model +reaches beyond one process. + +:::note | Sources and status +DeepSeek Harness at commit `687f282` (release 0.2.1-rc.2), which vendors Cordis as `@deepseek-ai/cordis` 4.0.5 (Cordis +RC7 with local changes, `vendor/README.md`): its Cordis tutorial (`docs/cordis-tutorial/`), `docs/architecture.md`, +`docs/subsystems/persistence.md`, `docs/subsystems/core.md` and `packages/boot/hmr/README.md`; the Cordis paper "A +Programming Paradigm for Spatiotemporal Composability" (arXiv 2608.25512). Cordis and dsh are built and in use; Chord v2 +is the design this guide describes. The comparison is between a working system and a design. +::: + +## Cordis in one example + +A Cordis plugin is a function (or an object with `apply`, or a `Service` subclass) that receives a **context**. It may +declare the services it needs with `inject`, and everything it does to the world goes through the context: + +```ts +export const name = "greeter"; +export const inject = ["tools"]; // activate only while a `tools` service exists + +export function apply(ctx: Context) { + ctx.tools.register(defineTool({ name: "greet", /* … */ })); // a registration is an effect: undone on unload + ctx.on("session/start", (s) => log(s.id)); // an effect-bound listener + ctx.effect(() => { // any other side effect, with its disposer + const timer = setInterval(work, 200); + return () => clearInterval(timer); + }); +} +``` + +Each plugin instance is a **fiber** with states `PENDING → LOADING → ACTIVE → UNLOADING → DISPOSED` (or `FAILED`). +Every effect, registration, listener, child plugin and provided service is recorded on the fiber, and unloading runs +the disposers. The paper calls this **temporal composability**: a component can be removed at any time and reverses its +own effects. Its counterpart, **spatial composability**, is the dependency side: `inject` is live, so a plugin starts +when its services appear, and unloads (and later reloads) when a service it depends on goes away or is replaced. The +paper's claim is that independently written components can therefore be combined and replaced without disturbing each +other. The next sections check how far that holds. + +**Services** are named capabilities on the context (`ctx.provide("x", value)` or `class X extends Service`). Names are +runtime strings in one flat namespace per application; TypeScript declaration merging makes `ctx.x` typecheck, but +nothing checks the service's shape at runtime, and optional dependencies are probed with `ctx.get("x")`. Scopes come +from the context tree: `ctx.extend()` derives a child context, `ctx.isolate(name)` gives one subtree its own binding of +a service, so two parts of an app can use two different `shell` providers. + +**Hot reload** follows from the two halves: the reload plugin unloads the old fiber (unwinding all its effects) and +loads the new module with the same config. Nothing held in the old instance's closures or fields survives; a plugin +that needs state across reloads must keep it in a service or on disk. Config edits are diffed by entry id and +mount, unmount or reconfigure only what changed; config fields declared volatile update in place without a remount. + +## DeepSeek Harness on Cordis + +dsh applies "everything is a plugin" to an agent: the model adapters (`ctx.llm`), the tool registry (`ctx.tools`), the +session log (`ctx.sessions`), the agents (`ctx.agents`) and the agent loop itself (`ctx.agentLoop`) are plugins and +services. A running dsh is a plugin tree composed at boot from a **profile** (`web`, `headless`, `sdk`, `acp`) that +stacks **bundles** of Cordis config rows, with user patches on top. There is no privileged core: replacing the agent +loop is a config patch. + +| Concern | How dsh does it | +| --- | --- | +| sessions | a durable, append-only event log per session (JSONL backend, single writer, `flush` as the durability barrier); model history is derived from the events | +| crash mid-turn | the log keeps everything appended; on resume the agent loop closes the open turn by appending missing tool errors and a synthetic `turn/end { reason: "interrupted" }` | +| tools | `ctx.tools.register(defineTool(…))`, an effect of the registering plugin; calls go through pre/execute/post policy waterfalls | +| approval | closed by default; `allowed-once` is the only grant; plugin installation needs full access or per-call approval | +| reload | a source watcher evicts changed modules; Loader entries running affected plugins stop and start again with the new code; a failed replacement is an activation failure, its previous code no longer runs | +| processes | one plugin tree in the Host process; the browser client and Electron's renderer talk to it over RPC streams | +| distribution | a Plugin Manager installs and enables bundles per profile; plugin markets are optional indexes | +| subagents | local subagents with their own durable sessions, and external ones | + +## What the guarantees are in practice + +Reading the Cordis source vendored in dsh and dsh's own documentation, the guarantees are narrower than the paper's +framing. An earlier review for p2 came to the same result (`apps/p2/docs/p3-reviews/dsh-compare.md`). + +- **Reversal is cooperative and unordered.** `_unload` runs all of a fiber's top-level disposers concurrently + (`Promise.all` in `vendor/cordis/src/fiber.ts`); order holds only inside one effect, and the primer tells authors to + "keep the related work in one effect" when teardown order matters. A disposer that throws is logged and skipped. A + disposer that never settles stalls the teardown; dsh's jobs service works around this by force-failing a job whose + cancel throws and warning that "the work may be orphaned" (`packages/jobs/jobs-local/README.md`). Reversal is as + good as every disposer, and nothing checks that a plugin registered all its effects. +- **Behaviour depends on registration order, and a reload changes it.** `emit`, `waterfall`, `serial` and `bail` + listeners run in registration order (`docs/cordis-primer.md`). Registering appends the listener (`push` in + `vendor/cordis/src/events.ts`, unless `prepend`), so when a plugin reloads, its listeners move to the end of the + list. A tool-policy listener that ran first runs last after its own plugin reloads; a waterfall that short-circuits + now sees a different decision first. +- **Dependencies are strings, and every provider reload restarts its dependents.** A fiber's activation key is the + concatenation of its injected providers' fiber ids (`_refresh` in `fiber.ts`), so replacing a provider unloads and + reloads every plugin that injects it, with their in-memory state. Service APIs are "pre-stable; update every + consumer" (`AGENTS.md`): there is no versioned contract between a provider and its consumers. +- **Reloading a core service ends live work.** Unloading the agent provider "stops and drains every live handle it + made" (`packages/core/agent/README.md`), so running agents stop; reloading the session persistence backend under + live sessions invalidates their write handles, and "writes fail loudly until the sessions restart" + (`.agents/notes/implemented/architecture/2026-08-27-handle-based-session-persistence.md`). +- **Registrations leave a gap.** A reloading plugin's tools and other registry entries are removed with its old fiber + and added again by the new one; anything that looks them up in between does not find them. + +None of this is a bug in the narrow sense: each point is documented, and dsh is built around it. But "components +interleave without disturbing each other" holds only for plugins that own nothing other plugins depend on, register +every effect, order nothing by registration, and hold no state that must outlive a reload. + +## Side by side + +| | Cordis and dsh | Chord v2 | +| --- | --- | --- | +| unit | a plugin instance (fiber) in one context tree | an object per (type, input, principal), placed on a host (chapters II, III) | +| side effects | recorded on the fiber with `ctx.effect` and friends; on unload the disposers run concurrently, each as good as its author made it | declared through `self` (documents, natives, routines, jobs, contributions); Chord activates and undoes them (chapters V, IX to XI) | +| dependencies | service names as strings in one flat namespace, via `inject`; a provider reload restarts every dependent | typed contracts via `get(Type, input)`; a reference parks and recovers through restarts (chapter VI); optional dependencies are registries (chapter XIII) | +| ordering | listeners run in registration order, which changes when a plugin reloads | registries compose with a declared `compose` function; nothing depends on load order (chapter XIII) | +| where code runs | one process; remote UIs over RPC to the Host | a tree of hosts (server, workers, tabs); objects reachable across links, relayed by the server (chapters I, XVII) | +| who is calling | not a framework concept | principals stamped by links, variants per caller, revocation (chapters III, VII) | +| state | the plugin's own business; dsh's session log is a service | documents written in transactions, persisted, versioned and migrated (chapter V) | +| reload | unload the old instance, load the new one: in-memory state and running work of that plugin end | pure objects swap their code in place with their documents and running work kept; only objects with natives restart (chapter XV) | +| a running agent turn during reload | entries of changed plugins stop; unloading the agent provider stops its live agents | the harness is base code; Durable pins tools per task phase, so a running call finishes on old code and the next phase uses the new (chapter XIV) | +| crash recovery of a turn | the turn is closed as `interrupted` | pi-durable continues the turn's tasks from their checkpoints; replay-safe tools rerun (chapter XIV) | +| contracts and versions | TypeScript types only, pre-stable APIs, every consumer updated together; config validated with a schema | contract versions with fingerprints and provider-side mappings for old consumers (chapter XVI) | +| status | in production in Koishi since 2019; dsh released | a design with the contract layer built | + +## What the comparison says + +**The core idea is the same.** "Factories declare, Chord activates" (chapter II) and Cordis' reversible effects are +two forms of one rule: a plugin may only change the world through its runtime, so the runtime knows what to undo. +Both reject the module-level timer and the stored callback for the same reason. + +**The difference is what survives a reload.** Cordis reloads at plugin granularity: the old instance unwinds and the +new one starts from nothing, together with every plugin that depends on it. That is simple, but a plugin's in-memory +state and its dependents' state and work in flight end with it, and correctness rests on every disposer and on +registration order. Chord v2 keeps state out of closures on purpose (documents, references, jobs as +records) so that it can swap the code of a pure object while its data and running work stay, and restarts only what +holds OS handles. That is more machinery: transactions, a store, versions on every document and contract. + +**The other difference is reach.** Cordis composes plugins inside one process, and dsh adds RPC to its own clients. +Chord v2 makes the process boundary part of the model: objects live on the host their placement names, callers are +principals, and a reload rolls out across the server, the workers and every tab. An app like p2, with a process per +session and guests in other browsers, needs that; a single-process agent does not. + +:::review | Review: what a reload does to a running tool call in dsh +The sources say that Loader entries running affected plugins stop and that unloading the agent provider stops its live +agents (`packages/boot/hmr/README.md`, `docs/subsystems/core.md`). They do not say what happens to a tool call in +flight when only the tool's plugin reloads; the comparison above does not claim either way. +::: diff --git a/packages/chord/docs/next/explainer/src/21-epilogue.md b/packages/chord/docs/next/explainer/src/21-epilogue.md new file mode 100644 index 000000000..987fb3c44 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/21-epilogue.md @@ -0,0 +1,104 @@ + +# What's Next {.epilogue lede="Where Chord v2 stands, what is still undecided, and where the written sources disagree with each other."} + +The guide built Studio from one object to a p2-sized app. Every piece used the same few rules: one owner per thing, +factories that only declare, state that changes only in transactions, long-lived behaviour in declared forms, and +mappings and migrations owned by the provider. Those rules are what let Mario press Reload on his phone in the +preface while the agent keeps running. This last chapter says how much of that exists and what is still open. + +## Status + +:::stats +- ~10k | lines in the frozen first kernel (steps 1 to 6, 443 tests), kept as a reference only +- 1 of 11 | layers of the rebuild started: L1 contracts, built and reviewed +- 6–7k | target size of Chord v2 without tests +::: + +- **The design is decided** except the points marked below. `packages/chord/docs/next/design.md` is the target; + `packages/chord/docs/next/decisions.md` is the log and wins where they differ. +- **The rebuild** goes layer by layer, each with a one-page spec the owner approves, a minimal build with counting and + garbage-collection tests, the cost-guard metrics, and a cleanup loop: L1 data and contracts; L2 readable and + derive; L3 documents and store; L4 objects; L5 references and `ctx`; L6 protocol and links; L7 relay, routes, + revocation; L8 routines and jobs; L9 registries; L10 reload and rollout; L11 telemetry in Chord. +- **Built:** the contract layer (`src/next/contracts/`, commit `7d2bb8ea8`, chapter XVI). In pi: Delta, + `pi-telemetry`, `chord/context`, `chord/telemetry-sqlite`, pi-ai and Durable telemetry, Durable's required + `checkpointWhen`. +- **Next, in order** (`apps/p2/docs/p3-handoff.md`): decide Q12 and Q15, which the L1 spec needs; write the spec for + the rest of L1 (values, inputs and identity, argument checks, principals, errors; draft in + `packages/chord/docs/next/layer-1.md`); then Q11, Q13, Q14 and the cuts C3 to C17 before the layers they affect. +- **In pi:** `mapShared` and bytes as atomic leaves in Delta's `diffRevisions`. +- **Then** the playground web shell (React, esbuild) with the reload control and a preview at `p3.mariozechner.at`, + and the first real proof: the harness, one tool with an approval pause, a code mode reload. Then the HTTP and + channel components, then p3, domain by domain. + +## Deferred decisions + +Kept with stable numbers in `design.md` §26, each waiting for a concrete case: + +| # | Topic | When | Chapter | +| --- | --- | --- | --- | +| 9 | jobs as objects: a record inside the owner, or an object addressed by its id | when login is ported | XI, XIX | +| 11 | where a flow lives: a keyed document of its owner, or an object `LoginFlow { id }` | when login is ported | XII | +| 14 | project plugins: per-directory plugins with trust prompts and snapshots | own pass over `plugin-loading.md` | XV | +| 16 | a Durable Object profile | when a DO target becomes concrete | | +| 17 | session files: one `SessionFiles { session }` owner on the server (proposed) | p3 design | XX | +| 18 | session delete: retire by tag plus a session directory, or owner hooks | p3 design | XIX | +| 20 | connection pools: one owner of SSH connections per environment, or per-use connections | p3 design | XX | +| 21 | config files with a single writer: one owner on the server, others follow | p3 design | XVIII | +| 22 | file formats (`envs.json`, downloads, uploads) as versioned interfaces | p3 design | XX | +| 23 | the terminal host process: inside the revision or deliberately outside | p3 design | XX | + +## Open from review round 3 + +Decisions (`apps/p2/docs/p3-reviews/v2-review3-summary.md`; Q1 to Q10 are decided): + +| # | Problem | Proposal | +| --- | --- | --- | +| Q11 | a singleton that does not exist yet: `undefined` or `initial()`? | singletons read `initial()`; `undefined` only for absent keyed documents | +| Q12 | stripping arguments (R7) against "nothing is copied" | copy on change: a new object only when the argument had extra fields; never mutate the caller's value (proposed in the L1 draft) | +| Q13 | `ctx.incarnation` on relayed calls | relays stamp the origin host's incarnation, as they stamp the principal | +| Q14 | a writable holding functions exposed as a state member | only members with a JSON schema in the contract are exposable, checked at serve | +| Q15 | arguments of `local` methods: validated or not | apply R7 to `local` too, or keep `local` as the one exception | + +Cuts the owner may choose (each removes a primitive this guide teaches): + +| # | Cut | What it costs | +| --- | --- | --- | +| C3 | shrink jobs to a runner over documents (no versions, `accepts`, `blocked`, `persist`), or move jobs out of the kernel | job versioning, or `self.job` for core-only users | +| C4 | routines: `on` and `loop` only | `every`'s due time across reloads | +| C5 | merge `lifetime: "host"` objects into the facet | a plugin's always-on natives share one failure domain | +| C6 | remove `broken()` for opens and `retryOn` | nothing; reload and Retry still work | +| C7 | remove provider-side `linger` | ten lines of app code | +| C8 | remove `state: false` (method-only references) | a method-only consumer pays one snapshot | +| C9 | remove `previous` from derives | derives that used it move to `mapShared` | +| C10 | one `cell(initial)` for `constant` and `writable` | nothing | +| C11 | remove `ctx.trace` (use `traceIn(ctx, …)`) | nothing | +| C12 | remove call credits (the byte window covers calls) | nothing | +| C13 | two timeouts (`closeTimeout`, `readyTimeout`) instead of eight | per-thing overrides | +| C14 | remove the `handle.change` and `handle.retire` shorthands | nothing | +| C15 | remove `inVariant` and `expectVariant` (keep `RefIn`) | nothing | +| C16 | remove facet-owned mutable documents and their transaction exception | atomic writes across an object and its facet move to one owner | +| C17 | no harness-specific start-barrier exception | factories before the barrier cannot depend on the harness | + +## Where the sources disagree + +This guide follows the decision log where it is newer than the design, and the design where it is newer than +everything else. The places that need a fix in the sources: + +:::review | Review: drift in the written sources +- `design.md` §26 lists only the deferred decisions; it omits the open review-3 items Q11 to Q15 and the cuts C3 to + C17. +- `design.md` §3 has a garbled sentence, "Values are method arguments are stripped". +- `apps/p2/docs/p3-handoff.md` §3 is stale: it says all R items are decided and, two lines later, that R2 to R5 and + R7 to R26 are open, and "resume R6 next". +- `kernel.md` and `step-*.md` describe the frozen code: host values, a development freeze, consumer-side linger, + JSONL storage, descriptions on the wire, and a link closed on an unknown id. None of these is the target. +- `chord-next-walkthrough.md` lags the log: validation by trust (now R7, every caller), dev mode and freeze (removed + by R9), descriptions once per link (gone with R6), an IPC frame-count window (now one byte window, Q4), job outcome + `{ ok }` (now `{ kind }`), harness activation as the start barrier (R17), registries "newer refused at load" (Q8), + and hooks using `host.root` (now `chord(fn, context)`, Q6). +- `apps/p2/docs/questions.md` uses `self.resource` and `replace` (both rejected) and `host.root` in hooks. +- `apps/p2/docs/p3-reviews/v2-cost-model.md`: the measurements are current, its "Decisions this needs" section is + not (chapter XVII). +- L1's `Implementation` shape differs from the factory return value used in the design (chapter XVI). +::: diff --git a/packages/chord/docs/next/explainer/src/22-appendix.md b/packages/chord/docs/next/explainer/src/22-appendix.md new file mode 100644 index 000000000..42eb965f4 --- /dev/null +++ b/packages/chord/docs/next/explainer/src/22-appendix.md @@ -0,0 +1,190 @@ + +# API Reference {.appendix #api lede="Every name the guide uses, grouped by where you meet it. Working names; only the contract layer is built."} + +## Hosts + +| Call | Meaning | Chapter | +| --- | --- | --- | +| `createHost({ address?, principal?, storage?, principalKey?, revokedBy?, telemetry?, onError?, timeouts })` | a host; with an address it is addressable and its own code acts as `principal`; without, a client | I | +| `host.load([{ name, facet }])` | load facets by plugin name; base code passes its core facet the same way | I | +| `host.connect(link, { principal })` | the link to the parent (at most one) | I | +| `host.accept(link, { principal })` | a link from a client; every request on it acts as `principal` | I | +| `host.route(Kind, { authorize, resolve })` | how to reach hosts of a kind; `resolve` returns `{ link, principal }` | I | +| `host.revoke(r)` | end matching objects and links here and below | VII | +| `host.root(fn, { principal?, signal?, telemetry? })` | a call scope with a `ctx` for code outside objects | VI | +| `host.get(Type, input, { signal, state?, parkTimeout? })`, `host.hold(…)` | a reference from outside objects; `release()` ends it | VI | +| `host.serve(Type, factory, { lifetime?, linger? })` | serve an object type (in a facet) | II | +| `host.registry(Registry)` | the composed value of a registry, a readable | XIII | +| `host.references(Type)` | who holds references to objects of a type on this host (presence) | VIII | +| `host.activity` | `{ references, calls, jobs, lingering }`, for keep-alive | I | +| `host.close()` | end the host: objects, natives, links | I | + +## Inside a factory: `self` + +| Member | Meaning | Chapter | +| --- | --- | --- | +| `self.state(Definition)` | a document handle | V | +| `self.tx(async (tx) => …, ctx)` | a transaction on the object's line | V | +| `self.derive(name, [sources], fn)` | a derived readable | VIII | +| `self.get(Type, input)` | a reference held for the object's life | VI | +| `self.open(open, close, { closeTimeout? })` | a native; `.value` once open | IX | +| `self.routines.on(readable, fn)`, `.every(interval, fn)`, `.loop(fn)`, `.custom(…)` | standing behaviour | X | +| `self.job(Job, { run, finish? })` | owned work that outlives its call | XI | +| `self.contribute(Registry, value)` | a contribution (facets and `lifetime: "host"` objects only) | XIII | +| `self.writable(initial)` | a host-local readable of any in-process value, with `set` | XIII | +| `self.retryOn(readable)` | a trigger that retries a broken activation | VI | + +**Document handles:** `value`, `get(key)` (keyed), `keys`, `change(fn, ctx)` / `change(key, fn, ctx)` and `retire(ctx)` / `retire(key, ctx)` (shorthands for +one-document transactions). **Transactions:** `tx.draft(handle, key?, { tags? })`, `tx.retire(handle, key?)`, +`tx.start(job, args, { key? })`. **Jobs:** `job.start(args, ctx, { key? })`, `job.abort(id)`, `job.running`, +`run.outcome(ctx)`. + +## References and `ctx` + +| Member | Meaning | +| --- | --- | +| `ref.` | a state member (readable: `.value`, subscribe) or a method | +| `ref.state`, `ref.onState(fn)` | `loading`, `live`, `failed` (cause, `retryAt`), `released` | +| `ref.retry()` | retry a failed reference now | +| `ref.release()` | end a reference taken outside objects | +| `ref.members` | the variant's members | +| `ctx.signal`, `ctx.principal`, `ctx.incarnation`, `ctx.get` | cancellation, caller, calling host's incarnation, call-scoped references | +| `ctx.telemetry`, `ctx.trace(name, attributes, fn)` | the current span; a child span with a child `ctx` | +| `ctx.jobId`, `ctx.progress(value)` | in a job's `run` | + +## Definitions + +| Name | What it declares | Chapter | +| --- | --- | --- | +| `hostKind(name, input?)` | a kind of host and its address | I | +| `remote(name, { version, input, placement, variants })` | an object type reachable across links | II | +| `local(name, { version, input, members })` | an object type on its own host | IV | +| `method(args?, result?)` | a method in a base declaration | II | +| `evolve(prev, (e) => …)`, `checkAccepts(contract, accepts)` | the next contract version; the provider's mappings (**built**) | XVI | +| `Implementation` | the type of a provider for the current version (**built**) | XVI | +| `str num int bool nul lit oneOf union obj arr map opt bytes unknown`, `Static` | schema helpers (**built**) | II | +| `defineState(name, { schema, version, initial, checkpointWhen, persist?, keyed?, ordered?, migrate? })` | a document | V | +| `defineTag(name)` | a tag; `store.retireByTag(tag, value)` | V | +| `defineJob(name, { version, args, progress, result, concurrency?, persist?, accepts? })` | a job type | XI | +| `defineRegistry(name, { version, compose, accepts? })` | an extension point | XIII | +| `bindFacet

()` | base code's `defineFacet`, with factories' principals typed as the app's `P` (working name) | III | +| `defineFacet((host, self) => …)` | a plugin's setup for one kind of host; imported from base code (`@studio/core`) | II, III | +| `constant(value)` | a readable that never changes (start data) | IX | +| `broken(error)` | a failure that is not retried by the backoff | VI | +| `inVariant(ref, …)`, `expectVariant(ref, …)`, `RefIn<…>` | narrowing a reference by variant | III | +| `mapShared(fn)` | an array map that keeps unchanged items identical (planned in Delta) | VIII | +| `OutputBuffer`, `Progress` | bounded output and throttled progress (`@earendil-works/chord/output`) | XI | +| `memoryLinkPair()`, `ipcLink(child)`, `webSocketLink(ws)` | link adapters | I | +| `DocumentStorage`, `registerDocumentStorageConformance` | the storage port and its shared test suite | V | + +# Timeouts and Defaults {.appendix #defaults} + +| Timeout or default | Bounds | Default | Override | +| --- | --- | --- | --- | +| `closeTimeout` | a native's `close`, then reported as a possible leak | 30 s | per native | +| call drain | an ending object's own call handlers | = `closeTimeout` | per type | +| `readyTimeout` | activation; an awaited `get` | 60 s | per `get` | +| park deadline | parked calls (link drop, loading, rollout) | 60 s | per `get` (`parkTimeout`); per type | +| routine handover | a new routine waiting for the old handler on reload | 10 s | per routine | +| `finish` retry | backoff of a throwing job `finish` | 1 s to 60 s | per job | +| rollout install | one host installing a revision | 60 s | none | +| linger | an object kept alive after its last reference | per type | per type | +| backoff | retries of transient failures, per address | 1 s doubling to 60 s | none | +| eviction | keyed persisted documents idle in memory | 10 minutes | | +| client WebSocket limits | per part | 16 MiB compressed, 64 MiB decoded | per link kind | +| compression | JSON parts compressed at their origin | above about 16 KiB | | +| worker keep-alive (p2, app code) | idle time before the supervisor stops a worker | 5 minutes | app | + +# Versions {.appendix #versions} + +| Surface | Version | On change | Chapter | +| --- | --- | --- | --- | +| object type | `version` plus fingerprint | `evolve` and `checkAccepts`; otherwise `contract_mismatch` | XVI | +| document | `version` plus schema fingerprint | pure `migrate` on access; no bump fails `broken` | V | +| job | `version` | `accepts` on outcomes and `running`; `blocked` without | XI | +| registry | `version` | `accepts` on contributions; unmatched ones left out and reported | XIII | +| native | none | restarts on reload | IX | +| Durable tasks and documents | Durable's `version` | Durable's `migrate` | XIV | +| link protocol | protocol version | refused; hosts update together | XVII | +| UI components, Chord and app APIs | none yet | exposed as they are | | + +# Errors and States {.appendix #errors} + +`ChordError` has a stable `code` and a finer `reason`; on the wire it is `{ code, reason?, message, details? }` +(specified in `layer-1.md`, not built). + +| Code | Meaning | +| --- | --- | +| `invalid` | input, arguments or placement rejected (`invalid/input`, `invalid/args`, `invalid/placement`) | +| `contract_mismatch` | the consumer's version and fingerprint are neither current nor accepted | +| `forbidden` | the caller may not do this | +| `gone` | the object or hold ended | +| `not_ready` | not activated in time | +| `released` | the reference was released | +| `too_large` | a part exceeded its size limit | +| `unavailable` | could not be reached before the deadline (`unavailable/locked` and others) | +| `aborted` | cancelled by a signal | + +| Type | Values | +| --- | --- | +| `FailureCause` | `{ kind: "transient"; error }` (retried with backoff) or `{ kind: "broken"; error }` (retried only by a trigger) | +| reference state | `loading`, `live`, `failed` (cause, `retryAt`), `released` | +| job outcome | `{ kind: "done"; result }` or `{ kind: "failed"; error: "threw" \| "aborted" \| "lost" }` | +| job phase | `running`, `finishing`, `blocked` | +| rollout status of a provider | `updating` (calls park), rejected (fail at once), same revision (`contract_mismatch`) | +| host install report | `installed`, `installed_failed`, `rejected` | + +# Rejected Designs {.appendix #rejected} + +From `design.md` §31, grouped. Each was considered and is not coming back without a new reason. + +| Area | Rejected | +| --- | --- | +| reload | drain-based handover; object restart for state changes; a factory trial; a migrate dry run; a consumer contract check during staging; reload votes; per-plugin rollback within a host; keeping natives across reloads by name; `restart_required`; host values and a development freeze | +| state | version fences without transactions; volatile documents; per-group lines; optimistic transactions; `tx.get`; a JSONL document storage; validating state and documents on commits or reads; copying values on same-host calls; op-rewriting projections and a view algebra; Durable's Session as the object store | +| objects and references | resources replaced in place (`replace`, `deps`, stale handles); revoking proxies on `.value`; consumer-side linger; detached references counting toward presence; an `optional` flag on references; `opaque` members; callbacks as method arguments; generic call or request ids; a stream type in Chord | +| registries | registries created at runtime; entries resolved through mutable indirection; generic registry keys with a `conflict` flag; numeric middleware priorities; registry composition as a staging check | +| principals and placement | principal field patterns and re-stamping for revocation; `storeKey`; placement as data naming an input field; host-name strings in app code; "magic" input-field matching for delete | +| wire | an outbox for transient delivery errors; cross-relay reset coordination; count limits that refuse calls or holds; contracts taken from the running app | +| process and tooling | a topology layer above the host API; loader variants A and B; Chord ending the process on a storage failure; `defaultProjectTrust: "always"`; ceremony (`native/` directories, `defineNative`, lints); weakening type checking | + +# Glossary {.appendix #glossary} + +**Adapted binding:** a consumer bound at an older accepted contract version, served through the provider's mappings. +**Base code:** the app's own code, imported statically; a change restarts the hosts. **Client:** a tab or TUI host, +not addressable. **Contract:** an object type's declaration (name, input, placement, members, version). +**Contribution:** a value a plugin adds to a registry. **`ctx`:** what every call receives. **Derive:** a value +computed from readables. **Document:** a named JSON value an object owns. **Facet:** a plugin's code for one kind of +host. **Factory:** the synchronous function that declares an object. **Fingerprint:** a hash of a contract's or +schema's normal form. **Flow:** input into running work, kept as state by its owner. **Host:** one participant of the +app, with an identity and an incarnation. **Incarnation:** the id a host gets on every start. **Input:** an object's +key among objects of its type; with the type and the caller's principal key it identifies the object. **Job:** owned work that outlives its call. **Link:** an ordered connection between +two hosts. **Local type:** an object type reachable only on its own host, without principals. **Native:** an OS-level +value opened with `self.open`. **Park:** a call waiting for a reference to become live again. **Placement:** the +function from input to host address. **Plugin:** a directory with a `package.json`, reloadable. **Principal:** who +calls, stamped by the link. **Principal key:** the string that identifies a principal in object identities. **Pure +object:** an object without natives; swapped in place on reload. **Readable:** a value plus its ordered changes. +**Reference:** what `get` returns. **Registry:** an owner's extension point. **Relay:** a host forwarding messages +between two links, rewriting only headers. **Reload:** swapping a plugin's code in running hosts. **Restart:** ending a +process. **Revision:** the complete code set the hosts should run. **Revocation:** ending live access for principals +matched by `revokedBy`. **Routine:** standing behaviour of an object. **Server:** the addressable root host. **Shared +store:** a local object that holds data once behind per-principal views. **Tag:** a typed label on documents, used to +retire them together. **Transaction:** an atomic set of document writes on the object's line. **Variant:** the set of +members a principal gets. **View:** a per-principal remote object in front of a shared store. **Worker:** a host the +server starts for one session. + +# Sources {.appendix #sources} + +| Source | Used for | Weight | +| --- | --- | --- | +| `packages/chord/docs/next/decisions.md` | every decision, review rounds 2 and 3 | wins where newer | +| `packages/chord/docs/next/design.md` | the target design, all chapters | authoritative otherwise | +| `packages/chord/src/next/contracts/` and its `README.md` | chapter XVI | built code | +| `packages/chord/docs/next/layer-1.md` | errors, inputs, argument checks | draft spec | +| `apps/p2/docs/p3-reviews/v2-review3-summary.md` | open Q11 to Q15, cuts C3 to C17 | current | +| `apps/p2/docs/p3-reviews/v2-cost-model.md` | measurements, per-hop costs | current except its last section | +| `apps/p2/docs/p3-functions/` | chapters XVIII to XX | current | +| `apps/p2/docs/p3-handoff.md` | status and order of work | §3 partly stale | +| `packages/durable/README.md` | the harness, submit, tasks, subagents | current | +| `chord-next-walkthrough.md`, `kernel.md`, `step-*.md`, `apps/p2/docs/questions.md` | background only | lag the decisions (see What's Next) | +| DeepSeek Harness at `687f282` (its Cordis tutorial, architecture, persistence, core and HMR docs), the Cordis paper (arXiv 2608.25512) | "Other Ways to Do It" | external, built systems | diff --git a/packages/chord/docs/next/explainer/walkthrough.md b/packages/chord/docs/next/explainer/walkthrough.md new file mode 100644 index 000000000..e4bfa5a9e --- /dev/null +++ b/packages/chord/docs/next/explainer/walkthrough.md @@ -0,0 +1,284 @@ +--- +title: Chord v2 +kind: walkthrough +kicker: Walkthrough · Chord v2 +subtitle: Build an app like p2 on Chord v2 and pi-durable, one concept at a time, with a tutor that draws the figures and asks the questions +model: anthropic/claude-sonnet-5-5 +sources: + - packages/chord/docs/next/explainer/book.md + - packages/chord/docs/next/design.md + - packages/chord/docs/next/decisions.md + - packages/chord/docs/next/layer-1.md + - packages/chord/src/next/contracts/README.md + - packages/chord/src/next/contracts/*.ts + - packages/durable/README.md +truth: treat Chord v2 as fully built as designed; ignore what the sources say about build status. redesign-decisions.md wins over design.md; design.md wins over the guide (book.md) +--- + +:::tutor +The learner knows pi-durable (harness, conversations, tasks, tools) and nothing about Chord. + +Teach Chord v2 as a finished, working runtime: never say a part is "not built", "planned" or "the target design", even +though the sources say so in places. Open decisions may be mentioned as "still being decided" when a concept depends on +one. + +The running example is Studio, introduced on the page before the first concept (see the introduction). Use it in every +concept and grow it step by step, as the guide book.md does chapter by chapter: read the matching guide chapter before +each concept, and check details in design.md and the decision log. Name the guide chapter in "Read more" lines. + +Keep code short and complete enough to read on its own. Prefer one figure per concept: a sequence for message orders, +a flow for structure, a stepper for lifecycles. +::: + +:::intro +#### What this is about + +Picture **p2** (Pi Mobile): a web app for working with pi agents from any device. It runs on your laptop as one +**server** process. Every chat session gets its own **worker** process, and in each worker a pi-durable agent works +through its turns. You use it from **browser tabs** on your desktop or phone, and you can invite a colleague to watch. +Most features (a notes panel, file browsing, terminals, provider logins) are **plugins**. + +Now you change one line in a plugin and press Reload, while an agent is twenty minutes into a task. The panel should +show the new code on every screen. The agent's turn, the running tool, the open connections and every bit of saved +data should simply continue. No restart, no lost work. + +**Chord v2** is the runtime that makes this normal. An app on Chord is a set of **objects** spread over the server, the +workers and the tabs. Each object owns its data and its OS handles, declares what it needs, and Chord creates, +connects, reloads and ends it. Because Chord knows every piece of state an object has, it can swap the object's code +while its data stays. + +#### The example: Studio + +This walkthrough builds **Studio**, a small p2, on Chord v2 and pi-durable. It has the same three kinds of processes: + +:::flow wide | Studio: one server, a worker per session with its agent, browser tabs as clients. The notes plugin is the first feature we build. +row "Clients" muted +box owner "owner's tab" "phone or desktop" +box guest "guest's tab" "read-only" +row "Server" accent +box server "server" "session list, access" accent +row "Workers" blue +box worker "worker S1" "agent + notes" blue +box worker2 "worker S2" "agent + notes" blue +arrow owner -> server "WebSocket" +arrow guest -> server "WebSocket" +arrow server -> worker "IPC" +arrow server -> worker2 "IPC" +::: + +The first feature is a **notes plugin**: a panel where the owner adds notes to a session, a guest reads them, and the +agent can add notes through a tool. Each concept adds one piece to Studio, until it has everything a p2 needs: who may +do what, saved data, live updates, background jobs, logins, extension points, the agent, and reload. + +#### How it progresses + +1. **The pieces:** processes, objects, callers, shared data, saved data. +2. **Living objects:** references that survive restarts, revocation, computed values, OS handles, standing behaviour, + background jobs, input from people. +3. **The app:** extension points between plugins, and the pi-durable agent inside Chord. +4. **Change:** reloading code while everything runs, evolving contracts, and what goes over the wire. +::: + +# What we're building {#goal part="The pieces"} +Guide preface ("Before we begin", "Ten things to know"). Must cover, briefly and concretely: the reload scenario (phone, +agent mid-turn, guest watching, one-line fix, Reload) and why restarting processes or hot-swapping modules fails; the +one rule behind Chord (every piece of state, OS handle and running work has one owner, an object, and objects only +declare what they need); Studio's pieces from the introduction. Do not go into APIs yet. + +:::check +1. Why is restarting the worker processes not a good way to load new plugin code? ++++ answers +1. It loses the agent's turn in flight, open terminals and connections (pi-durable would resume the turn, but the running tool and the connections are gone). +::: + +# The shape of an app {#shape part="The pieces"} +Guide chapter I. Must cover: hosts (server, workers, clients), links, host kinds and placement, createHost/load/connect/accept/route, +clients reach workers only through the server, incarnations, keep-alive is app code (host.activity). + +:::check +1. A tab wants an object that lives on worker S1, which is not running. What happens, in order? +2. Why does Chord have no listen() call? ++++ answers +1. The tab's call goes to the server; the address is not the server's, so it uses route(Session): authorize, then resolve starts the worker and returns a link plus principal; the server relays the hold, the worker builds the object and answers with a snapshot. +2. Entry points differ too much (loopback WebSocket, Radius channels, a Durable Object edge, memory links); each authenticates and calls accept. +::: + +# A first object {#first-object} +Guide chapter II. Must cover: contract (name, input, placement, variants, version): the input is the object's key +among objects of its type, `placement` computes the host from it; schema helpers, facet and factory (a facet imports +Studio's own defineFacet from @studio/core), serve and get, methods return data, placement checked on both sides. + +:::check +1. Two tabs of the owner ask for Notes of S1. How often does the factory run? +2. A caller with an outdated placement rule asks worker S2 for Notes of S1. What happens? ++++ answers +1. Once per principal key; the owner's two tabs on the same access path share the object, so once. +2. The provider runs the placement function itself and refuses with invalid/placement. +::: + +# Who is calling {#principals} +Guide chapter III. Must cover: the app's principal type, links stamp principals, principal key in the object identity, +variants per principal, narrowing with inVariant/RefIn, and how a facet knows P: base code exports defineFacet bound to +P (bindFacet

()), every facet imports it, and host.load accepts only facets typed for its P. + +:::check +1. Can a client send a principal field to claim it is the owner? +2. Why do guests not get an add method that throws "forbidden"? +3. The notes plugin's worker facet narrows principal.kind === "guest". Where does the type checker get Studio's principal union from? ++++ answers +1. No: a client's principal field is always overwritten by the link's stamp; only a trusted parent link may carry one. +2. Variants: the reader variant simply has no add member, and Chord refuses calls to members outside the caller's variant. +3. From Studio's base code: @studio/core exports defineFacet = bindFacet

(), the facet imports that one, and the worker's host.load accepts only facets typed for the same P. +::: + +# One store, many views {#shared-store} +Guide chapter IV. Must cover: local types, the shared store pattern, attribution as data, argument validation for every caller (R7, Q15 open). + +:::check +1. Where does "who wrote this note" come from if the store is local and has no principal? ++++ answers +1. The remote view knows the caller and passes the author into the store's method as data. +::: + +# Writing it down {#documents} +Guide chapter V. Must cover: defineState, self.tx, drafts and ops, persistence only for stable identities, versions with migrate, tags, storage. + +:::check +1. Why may a remote per-principal object not persist documents? +2. A schema changes but the version is not bumped. What happens? ++++ answers +1. Its identity includes the principal key, which changes (devices, grant revisions); persisted data needs a stable identity, so it lives in a local store or a host-lifetime object. +2. The stored fingerprint differs from the definition's at access, and the object fails broken ("schema changed without a version bump"). +::: + +# Holding on {#references part="Living objects"} +Guide chapter VI. Must cover: reference lifecycle, ref.state, parking, transient vs broken failures, backoff, ctx. + +:::check +1. The link to a worker drops while a tab calls a method. What does the call do? ++++ answers +1. A call already in flight fails like on a disconnect; calls made while the link is down park until the reference is live again (up to the park deadline, 60 s), then fail unavailable. +::: + +# Taking access away {#revocation} +Guide chapter VII. Must cover: revokedBy, host.revoke, propagation to children, no re-stamping, lasting denial in app data. + +:::check +1. After revoke({ grant: "g1" }), Ann reloads her tab. Why does she not get access back? ++++ answers +1. revoke cuts only live access; the app's guest record is blocked and authenticate refuses it on the new connection. +::: + +# Computed values {#derives} +Guide chapter VIII. Must cover: derives, the footgun (new objects for unchanged data), mapShared, container width, presence with host.references. + +:::check +1. Why is items.map(stripAuthor) in a derive expensive on every commit? ++++ answers +1. It builds new objects for every unchanged note, so the diff walks and sends O(value) instead of O(change); mapShared keeps unchanged items identical. +::: + +# Things the OS gives you {#natives} +Guide chapter IX. Must cover: natives with self.open, pure vs native objects, start data with constant, lifetime host, call-scoped natives. + +:::check +1. What happens to a native object on reload, and why? ++++ answers +1. It restarts: its natives close and reopen with the new code, because old and new code sharing an OS handle would interleave unchecked. +::: + +# Standing behaviour {#routines} +Guide chapter X. Must cover: routines (on, every, loop), serialized handlers, reconciling instead of notifying, pub/sub reports with ctx.incarnation. + +:::check +1. Why does archiving a session not call notes.archive(session) directly? ++++ answers +1. A notification can be lost (worker not running, link drops, crash); a routine on the catalog's sessions reconciles from the current value, also after a crash. +::: + +# Work that outlives the call {#jobs} +Guide chapter XI. Must cover: defineJob, the intent / effect / outcome sandwich, dedupe of unsettled runs (Q9), finish in current code, abort as a request. + +:::check +1. A shell job's run crashes after writing the result entry. How is a duplicate entry avoided? ++++ answers +1. The external write carries a key derived from ctx.jobId (requestId shell:jobId), so Durable dedupes it. +::: + +# Asking a person {#flows} +Guide chapter XII. Must cover: a flow is state, not a call; steps helper; the Questions service; decisions #9 and #11 open. + +:::check +1. Why not a long-held step() call that returns the next login step? ++++ answers +1. A link drop fails it and the consumer cannot tell if it missed a step; state re-holds by itself and shows the current step. +::: + +# Extension points {#registries part="The app"} +Guide chapter XIII. Must cover: registries, compose, who may contribute (R21), contributions that change, optional dependencies, HTTP component, channels, chunked transfers. + +:::check +1. Radius needs forwards from the ports plugin, which may be missing. How is that expressed? ++++ answers +1. Radius owns a registry (radius.forwards) and ports contributes to it; without ports the table is empty. +::: + +# The agent {#agent} +Guide chapter XIV. Must cover: the Harness as a core host-lifetime object, HarnessModels / HarnessEnvironments / HarnessExtensions registries, chord(fn, context) for tools, approval via Questions. + +:::check +1. How does a Durable tool get a Chord ctx? ++++ answers +1. Through chord(fn, context): the host is in Durable's context under a context key, and host.root gives fn a ctx. +::: + +# Reload and rollout {#reload part="Change"} +Guide chapter XV. Must cover: load, staging, commit, per-object swap or restart, rollback all-or-nothing per host, revisions, rollout order, updating vs contract_mismatch. + +:::check +1. A staging error happens in one of three plugins being reloaded on a host. What runs afterwards? ++++ answers +1. The whole host stays on its previous code; rollback is all-or-nothing per host. +::: + +# Changing contracts {#contracts} +Guide chapter XVI; the code is in src/next/contracts. Must cover: evolve, checkAccepts and mapping directions, binding by version and fingerprint, new history bases, the fingerprint normal form. + +:::check +1. Which way does a state mapping go, and which way an args mapping? +2. Why can an input change not be adapted? ++++ answers +1. State: current to old (the old consumer reads). Args: old to current (the old caller sends). +2. The input is the object's address; changing it changes what the object is, so it needs a new history base and old consumers get contract_mismatch. +::: + +# On the wire {#wire} +Guide chapter XVII. Must cover: messages and parts, relays rewrite only headers, one state message per commit per link, flow control (R19, Q3), size limits per part, the cost model, telemetry. + +:::check +1. What does the server do with a state part it relays from a worker to a tab? ++++ answers +1. It reads and rewrites only the header (maps ids, groups parts per destination link) and forwards the part without decoding it. +::: + +# Other ways to do it {#compared part="In context"} +Guide chapter "Other Ways to Do It" (after chapter XX). Must cover: Cordis (Koishi's plugin framework since 2019) and +DeepSeek Harness (dsh, everything is a plugin, built on Cordis): reversible effects on a plugin fiber, inject-based +services, reload by unload-and-load; dsh's durable session log and its "interrupted" crash repair. Then what the +guarantees are in practice (the chapter's section of that name): disposers run concurrently and are only as good as +their authors, listener order follows registration order and changes on reload, services are untyped strings and a +provider reload restarts all dependents, and reloading a core service stops live agents. Then the two differences that +matter: what survives a reload (Cordis: nothing of the plugin's or its dependents' in-memory state; Chord: a pure +object's data and running work) and reach (one process vs. a tree of hosts with principals). Say what Chord costs: +more concepts and machinery. A flow or table comparing a reload in both helps. + +:::check +1. A Cordis plugin keeps a list of notes in a field of its instance. What happens to the list when the plugin hot-reloads, and what would the same plugin do on Chord v2? +2. Two Cordis plugins register waterfall listeners on the same tool-policy event; the first one short-circuits. What changes after the first plugin is hot-reloaded? +3. Why does an app like p2 need Chord's multi-host model, when dsh runs its plugin tree in one Host process? ++++ answers +1. The old fiber unloads and the new one starts empty, so the list is gone unless it was kept in a service or on disk. On Chord the notes are a document of a pure object; a reload swaps the object's code and keeps the document. +2. Its listener is registered again at the end of the list, so the other plugin's listener now runs first and the decision can change, unless the listener was registered with prepend. +3. p2 has a worker process per session and clients (including guests) in other browsers; objects must live on the right host, callers must be stamped principals, and reload has to roll out across server, workers and tabs. A single-process tree does not cover that. +::: + diff --git a/packages/chord/docs/next/flow-control/book.md b/packages/chord/docs/next/flow-control/book.md new file mode 100644 index 000000000..0485fe6a0 --- /dev/null +++ b/packages/chord/docs/next/flow-control/book.md @@ -0,0 +1,333 @@ +--- +title: Flow control for Chord links +cover: Slow / Links +kicker: A design note · Chord v2 slice 4 · 2026-10-22 +subtitle: How a phone on a bad train connection keeps seeing the newest state of a streamed reply, and why the transport's own backpressure cannot do that +stats: + - 9.4 s | how old the newest state was with transport backpressure only + - 0.0 s | with an acknowledged window of two messages + - 345 ms | another client's p95 during a call flood, without a call window + - ~200 | lines for the converged design +note: | + Written from the measurements and the cross-checked reports of the reviewers Astra and Fable (2026-10-22): Node 24.4.1, + `ws` 8.22, separate server and client processes, a throttling proxy at 64 KiB/s. The numbers are single synthetic + runs: they show failure modes, not production capacity. Relays: first decided as per-hop acknowledgements (b) on + 2026-10-23, then replaced by flow control per origin on 2026-10-24 after the slice 4a benchmarks (chapter VI). +colophon: Flow control for Chord links · fourth edition · 2026-10-24 · based on the reviewers' experiments and design §3, §22 +--- + +# Before we begin {.preface} + +Mario is on a train. His phone shows a p3 session in which the agent is writing a long reply. Every token is a small +state change: the worker commits it, the server relays it, the phone applies it. The train enters a tunnel and the +phone's link drops to a trickle, 64 KB per second. The worker keeps producing tokens at full speed. + +Without flow control, every token still gets sent. The bytes pile up somewhere between the worker and the phone, and +when the train leaves the tunnel the phone has to work through a backlog of old tokens before it sees the current text. +It is not only slow: what it shows is minutes old, and nothing tells it so. + +This note explains what keeps the phone current, why the obvious answer (let the transport push back) fails, and the +design Astra and Fable converged on after measuring it. + +## Six things to know + +:::findings +1. **Transport backpressure arrives too late.** The kernel's and a proxy's buffers take megabytes before a WebSocket + reports anything buffered; the phone sees state 4 to 10 seconds old. +2. **Acknowledgements keep state fresh.** The receiving application acknowledges each message; at most two are in + flight; the newest state was 0.0 seconds old on the same 64 KB/s link. +3. **Count messages, not bytes.** Each message carries everything that piled up since the last acknowledgement, so + throughput adapts to the link by itself. +4. **Merge ops first, snapshot last.** 2,000 merged token appends cost 26 KB, sent one by one 46 KB; a snapshot of a + member with 500 KB of history would cost 508 KB. +5. **Calls need their own window.** A tab firing 100,000 calls pushed another client's p95 to 345 ms; with a window of + 64 respected by the caller it stayed at 0.66 ms and the flood waited in the buggy tab. +6. **Nothing is refused for being too many** (decision K16): extra state is merged, extra calls wait. +::: + +## Words used throughout + +A **link** connects two hosts. The **provider** owns an object and sends its state; the **consumer** holds a +reference to it. A **hold** is one reference over a link, with its own stream of **ops** (Delta's changes). A +**snapshot** is a hold's whole state. An **acknowledgement** (`ack`) is a tiny message the consumer sends after it +applied a state message. **In flight** means sent and not yet acknowledged. **Coalescing** means merging the ops that +wait to be sent. A **relay** is a host in the middle (the server between worker and tab). + +# Where the Bytes Hide {#hidden lede="Why the sender cannot see a slow phone."} + +A WebSocket in Node reports `bufferedAmount`: bytes the library still holds. It looks like the perfect signal: stop +sending when it grows. The experiment says otherwise. + +:::sequence animate | A flood of state messages to a slow phone through a proxy. The sender's buffer stays small; the kernel's and the proxy's do not. +actor worker "worker" "commits tokens" accent +actor ws "server WebSocket" "bufferedAmount" +actor kernel "kernel + proxy buffers" muted +actor phone "phone" "64 KB/s" blue +worker -> ws: token 1 … token 9,000 +ws -> kernel: written at once (bufferedAmount stays near 0) +kernel -> phone: drains at 64 KB/s +note kernel: megabytes waiting +worker -> ws: token 9,001 (the newest) +ws -> kernel: queued behind everything +kernel --> phone: arrives seconds later +::: + +| Flood, ~9,800 × 128 B per second, 64 KB/s link | Sender memory | Newest state the phone saw | +| --- | --- | --- | +| no flow control | 7.0 MB queued in the WebSocket, heap +18.5 MB | 9.4 s old | +| cap `bufferedAmount` at 64 KiB, merge pending state | +0.46 MB | **still 9.4 s old** | +| cap at 1 MiB (Fable's rerun) | small | 3.8 s, growing to 10.4 s | +| at most 1 to 4 acknowledged messages in flight, merge pending state | +0.4 MB | **0.0 s** | + +The transport's backpressure protects the sender's memory, but freshness depends on buffers the sender cannot see: +the operating system's socket buffer, autotuned to hundreds of kilobytes, and every proxy on the way. A normal +streamed reply (50 tokens per second, 23 bytes each) never fills them; a burst or a tunnel does. Only the receiving +application knows what it has applied. + +# Acknowledged Messages {#acks lede="A window of two messages in flight, and why it adapts to any link."} + +The rule: a provider has at most **K = 2** state messages in flight per link. The consumer sends `ack` (three bytes in +the binary codec) after it applied one. While two are in flight, new commits do not produce messages; their ops wait, +merged per hold, and go out in the next message once an `ack` arrives. + +:::steps | Two messages in flight on a slow link ++++ Fast link +Commits 1, 2, 3 each become a message; each is acknowledged before the next commit. Nothing waits: the window never +fills, so a normal stream behaves exactly as without flow control. ++++ The link slows +Messages 1 and 2 are in flight. Commits 3 to 400 arrive; their ops are folded into the pending queue of each member +they touch (chapter III). Memory: the net change, not the history; at most 10,000 ops per member. ++++ An ack arrives +`ack` for message 1. The provider sends message 3, carrying the folded ops of commits 3 to 400, encoded once. +The phone applies them as one batch and shows the newest text. ++++ Self-clocking +The slower the link, the more each message carries; the faster, the smaller and more frequent. Nobody measures +bandwidth: the acknowledgements pace the sender. +::: + +:::raw +

+
+ + + +
+ +
+
+ +::: + +The simulation above shows the same effect in miniature. Drag the link below the commit rate: the upper lane's queue +grows without bound and the phone falls further behind every second; the lower lane never has more than two messages +waiting, and the phone is at most a message behind. + +# Fold the Backlog {#merge lede="What a waiting hold keeps while the window is full, and why it never measures sizes."} + +While messages are in flight, each hold member keeps the ops of new commits in a **pending queue** and **folds** them: +a new op merges into the last pending op on the same path when that is safe, so a backlog keeps the net change, not +the history. When an acknowledgement frees a slot, the queue is encoded once into one message. + +| Last pending op | New op | Kept | +| --- | --- | --- | +| `a text "Hel"` | `a text "lo"` | one `a text "Hello"` | +| `s count 41` or an append | `s count 42` | the `s`: it overwrites what came before | +| anything | `r value` | only the `r`: a replacement makes everything before it moot | +| anything on a path in the index | a structural op (`p`, `m`, `d`, `t`, `s` of an object) | appended; the index is cleared | + +The merge partner is found through a **leaf index**: a map from path to the last pending append or primitive set on +that path. Leaf ops on different paths commute, so a new op may merge into an earlier one, not only into its +neighbour; a status field changing between two tokens no longer stops the reply from folding. Any structural op +clears the index, because after a splice `entries[7]` may name another entry. Past 10,000 pending ops the queue keeps +only "replace with the current value", which then costs nothing while the consumer stays slow. + +**Why no size rule.** The first idea was to send a snapshot once the merged ops outgrow the member. The reviewers +measured it on real p2 data and it failed three ways: "snapshot bytes plus sent op bytes" is no bound (an interned path +lets a value grow faster than its ops: 5,404 B of snapshot after 832 B of ops), it drifts 1,254 times too high under +churn, and an exact size needs a size on every node of the value. Folding alone came within about 10% of an exact +oracle, with no measuring at all. + +| 60 s stall, all workloads | Uncompressed | Compressed | +| --- | --- | --- | +| merged ops, no folding | 466 KB | 166 KB | +| adjacent folding | 399 KB | 147 KB | +| leaf index (the decision) | 372 KB | 145 KB | +| exact oracle (not buildable) | 335 KB | 136 KB | + +**One queue for every slow consumer.** The same queue serves Chord's shared watch handle: a listener still busy with +the last frame gets the commits made meanwhile as one call with the newest value and the folded ops, instead of +today's "after 100 frames, replace everything". With a listener taking one second, a conversation view's watch went +from 402 calls and 17.9 MB of ops to 312 calls and 257 KB. Durable's event stream keeps a queue of its own, because +events are not state: folding would drop them. + +:::raw +

Animation of the fold: How folding works.

+::: + +# Calls Get Their Own Window {#calls lede="Why acknowledged state does not protect against a call flood."} + +Calls are small and the provider acknowledges nothing until they finish, so a byte or message window on state says +nothing about how many handlers run. A tab with a bug: + +```ts +for (const note of notes) ref.add(note, ctx); // 100,000 calls, none awaited +``` + +| 100,000 calls, handlers awaiting 10 ms | Finished in | Provider memory | Another client's p95 round trip | +| --- | --- | --- | --- | +| no limit | 0.66 s | +56.5 MB, 43,824 running | 129 ms (345 ms with trivial handlers) | +| provider-side semaphore of 64 | 15.9 s | +42.5 MB, 96,736 queued on the provider | 0.38 ms | +| caller-side window of 64 | 17.1 s | **+4 MB, nothing queued on the provider** | 0.66 ms | +| handlers doing 10 ms of CPU work | | | blocked about 1 s, with or without a limit | + +The provider's `hello` announces a call window **N** (256 by default) per origin (chapter VI). A consumer keeps at +most N of its own calls unanswered per link; further calls wait in the link's call queue, in order. Nothing is refused (K16); a peer that sends +more than N is a protocol error. The flood waits in the tab that caused it. CPU-heavy handlers block the event loop +whatever the window: that stays the handler author's responsibility. + +# Relays {#relays lede="One link to a worker carries many tabs. Decided: flow control per origin."} + +The server relays every tab's holds to a session's worker over one link. Three ways to apply the acknowledged window +across that hop were considered: + +:::flow wide animate | (a) End-to-end acknowledgements: the server acknowledges the worker only after the tab did. +row "Worker" accent +box w "worker" "K = 2 per link" +row "Server" muted +box s "server" "acks upstream after downstream" +row "Tabs" blue +box fast "fast tab" "desktop" +box slow "slow tab" "phone in a tunnel" dashed +arrow w -> s "state" +arrow s -> fast "state" +arrow s -> slow "state, slow" +arrow slow -> s "ack (late)" dashed +arrow s -> w "ack only when every tab acked" accent +note "Simplest, but the slow phone holds the worker's one window, so the fast tab's updates arrive in coarser, later batches too." +::: + +:::flow wide animate | (b) Acknowledgements per hop: the server acknowledges at once and resets a slow tab. +row "Worker" accent +box w2 "worker" "K = 2 per link" +row "Server" muted +box s2 "server" "acks at once; window per tab" +row "Tabs" blue +box fast2 "fast tab" "desktop" +box slow2 "slow tab" "phone in a tunnel" dashed +arrow w2 -> s2 "state" +arrow s2 -> w2 "ack at once" accent +arrow s2 -> fast2 "state, current" +arrow s2 -> slow2 "window full: its parts dropped" dashed +arrow s2 -> w2 "reset: fresh snapshots" accent +note "The server cannot fold opaque parts, so it drops them and asks for snapshots. Built and measured in slice 4a: a 1 s stall cost three full snapshots (~1.5 MiB) and ~24 s to catch up under streaming." +::: + +:::flow wide animate | (c) Flow control per origin (decided 2026-10-24): the worker keeps a window per tab; the relay only forwards. +row "Worker" accent +box w3 "worker" "K = 2 per (link, origin)" +row "Server" muted +box s3 "server" "forwards; FIFO of senders" +row "Tabs" blue +box fast3 "fast tab" "origin 1" +box slow3 "slow tab" "origin 2, in a tunnel" dashed +arrow w3 -> s3 "state @1, state @2" +arrow s3 -> fast3 "state, current" +arrow s3 -> slow3 "state, slow" dashed +arrow slow3 -> s3 "ack (late)" dashed +arrow s3 -> w3 "ack @2 when the tab acked" accent +note "The relay numbers each tab (an origin) and forwards at once; each tab's ack goes back to the worker that sent the message. The worker folds the slow tab's commits in that origin's pending queues, exactly as for a direct tab; the fast tab is another origin with its own window." +::: + +How (c) works: + +- **Origins.** The relay gives each downstream link a number, never reused; every message upstream carries it + (`origin`), and the worker keys holds, calls, windows and acks by (link, origin, id). Ids are not renumbered. +- **What the relay keeps.** Per relayed hold its upstream link and its original `hold` message (to route calls and + hold again after a worker restart); per tab a FIFO of which worker sent each message in flight, so each in-order + `ack` goes back to the right worker. Nothing is parsed, folded, dropped or reset. +- **Calls.** N is per origin at the worker; the relay forwards calls at once, each tab bounded by its own window. +- **A tab leaves.** Its link closes; the relay sends `closed { origin }` to each worker it relayed to, which releases + that origin's holds and aborts its calls. +- **Cost.** A tab served by W workers may have 2W messages in flight (plus the relay's own 2). Two bytes of origin per + message on the worker link. + +# What Changes in the Design {.epilogue lede="The converged scheme, what it replaces, and the relay decision."} + +**Converged** (Astra and Fable, after cross-checking each other's experiments): at most K = 2 acknowledged state +messages in flight; pending ops folded per hold member through a leaf index, capped at 10,000; a call window N +announced in `hello`, extra calls queued by the caller; holds unlimited; size limits stay a separate check. + +**Replaced** from the design (§3, §22): the byte window and its credit messages, snapshot admission and debt (R19, Q3), +cut C12's call-credit accounting, any adapter backlog API. IPC and WebSocket behave the same, because acknowledgements +are messages. + +**Relays (owner):** per-hop acknowledgements (b) were decided on 2026-10-23 and built in slice 4a; the benchmarks +showed a slow tab recovering through resets far worse than a direct tab (24 s and three snapshots after a 1 s stall). +On 2026-10-24 they were replaced by **flow control per origin** (c): the worker keeps window, pending queues and calls +per (link, origin), the relay only forwards and returns acks. Measured after the rework: a slow tab through the relay +catches up exactly like a direct one. The circuit breaker stayed undecided; the heartbeat (15 s ping, 30 s close) +was built instead. + +# Measurements {.appendix} + +| Experiment | Setup | Result | +| --- | --- | --- | +| normal stream | 50 × 23 B/s through 64 KiB/s | no buffering, frames ≤ 2 ms old | +| flood, no control | ~9,800 × 128 B/s, 10 s | 7.0 MB in `bufferedAmount`, heap +18.5 MB, newest state 9.4 s old | +| flood, `bufferedAmount` ≤ 64 KiB + merging | same | heap +0.46 MB, newest state 9.4 s old | +| flood, 1 to 4 acked messages + merging | same | heap +0.4 MB, newest state 0.0 s old, 98 % fewer messages | +| client stops reading (loopback) | 20k × 90 B/s, 15 s | 23.4 MiB buffered, RSS +121 MB; merging above 1 MiB: flat | +| IPC to a busy child | 623k sends | `send()` false from ~64 KB, parent +1,035 MB; the callback acknowledges the write, not the handler | +| call flood | 100,000 calls, see chapter IV | caller-side window: +4 MB, other client p95 0.66 ms | +| slow tab through a relay, (b) resets | 64 KiB/s, 100 tokens/s into a 500 KB view, 1 s stall | ~24 s to catch up, three full snapshots (~1.5 MiB) | +| slow tab through a relay, (c) per origin | same, 1 s / 10 s / 60 s stall | caught up after 0.02 / 0.08 / 0.38 s, 485 B / 4.1 KiB / 23.7 KiB (same as direct) | diff --git a/packages/chord/docs/next/flow-control/fold/index.html b/packages/chord/docs/next/flow-control/fold/index.html new file mode 100644 index 000000000..3cb573bb5 --- /dev/null +++ b/packages/chord/docs/next/flow-control/fold/index.html @@ -0,0 +1,265 @@ + + + + + +How folding works + + + +
+

How folding works

+

What a hold does with commits while its link's window is full (two messages unacknowledged), and why folding needs no size measuring.

+ +

The rule

+

While the window is full, a hold keeps the ops of each new commit raw in a pending list. Before appending an op, it looks at the last pending op only:

+ + + + + +
last pendingnew opresult
a path "Hel"a path "lo"one a path "Hello" (text joined)
s path 41 or a path …s path 42one s path 42 (the set overwrites what came before)
anything on another pathanythingappended as it is
+

Applied in order, the folded list gives the same value as the unfolded one: only neighbours on the same path merge, so no op in between can see the difference. Cost: one path comparison per op. When an ack frees a slot, the pending list is encoded once into one message.

+
a = append texts = set valuep = insert into listsnapshot
+ +
+
+ + + + +
+
+ + + + +
+
+
link
window full: 2 unacked (tunnel)
+
commits
+
pending (raw)
+
sent
+
+
+ commits 0 + pending ops 0 + merged or dropped 0 + encodes 0 +
+
+
+ +

What the four cases show

+ + + + + + +
casepending after the stallwhy
streamed reply1 opevery token appends to the same path
counter, 100,000 sets1 opeach set overwrites the previous one
reply + status, interleavedgrowsthe neighbour is always on the other path: adjacent-only folding cannot merge
new list itemsgrowsinserts carry new data: nothing to fold, this is the real change
+

The last two cases are why a memory bound is still needed. It can be a plain count of pending ops after folding (say 1,000): past it the hold drops its list and marks “send the current value”, which then costs nothing more while the stall lasts.

+ +

Why not time since the last ack?

+

A time trigger (“no ack for T seconds: drop the list, send a snapshot later”) punishes the most common case: a phone in a tunnel while a reply streams into a long conversation. Folded, the 60 s stall costs one append of about 12 KB; a snapshot of the conversation costs its whole size (Astra's corpus: 4.6 KB of updates against 168 KB, recovery 0.2 s against 2.7 s). And before T, a fast producer on many paths still piles up rate × T ops: time bounds duration, not memory. With folding, the counter case (100,000 sets) is one op whatever the time, so time adds nothing there.

+
+ + + + diff --git a/packages/chord/docs/next/flow-control/index.html b/packages/chord/docs/next/flow-control/index.html new file mode 100644 index 000000000..a34b39a24 --- /dev/null +++ b/packages/chord/docs/next/flow-control/index.html @@ -0,0 +1,548 @@ + + +Flow control for Chord links + + + + +
+
+
A design note · Chord v2 slice 4 · 2026-10-22
+

Slow
Links

+

How a phone on a bad train connection keeps seeing the newest state of a streamed reply, and why the transport's own backpressure cannot do that

+
9.4 show old the newest state was with transport backpressure only
0.0 swith an acknowledged window of two messages
345 msanother client's p95 during a call flood, without a call window
~200lines for the converged design
+

Written from the measurements and the cross-checked reports of the reviewers Astra and Fable (2026-10-22): Node 24.4.1, +ws 8.22, separate server and client processes, a throttling proxy at 64 KiB/s. The numbers are single synthetic +runs: they show failure modes, not production capacity. Relays: first decided as per-hop acknowledgements (b) on +2026-10-23, then replaced by flow control per origin on 2026-10-24 after the slice 4a benchmarks (chapter VI).

+
+
+ + +
+

Before we begin

+

Mario is on a train. His phone shows a p3 session in which the agent is writing a long reply. Every token is a small +state change: the worker commits it, the server relays it, the phone applies it. The train enters a tunnel and the +phone's link drops to a trickle, 64 KB per second. The worker keeps producing tokens at full speed.

+

Without flow control, every token still gets sent. The bytes pile up somewhere between the worker and the phone, and +when the train leaves the tunnel the phone has to work through a backlog of old tokens before it sees the current text. +It is not only slow: what it shows is minutes old, and nothing tells it so.

+

This note explains what keeps the phone current, why the obvious answer (let the transport push back) fails, and the +design Astra and Fable converged on after measuring it.

+

Six things to know

+
    +
  1. Transport backpressure arrives too late. The kernel's and a proxy's buffers take megabytes before a WebSocket +reports anything buffered; the phone sees state 4 to 10 seconds old.
    2. +
  2. Acknowledgements keep state fresh. The receiving application acknowledges each message; at most two are in +flight; the newest state was 0.0 seconds old on the same 64 KB/s link.
    3. +
  3. Count messages, not bytes. Each message carries everything that piled up since the last acknowledgement, so +throughput adapts to the link by itself.
    4. +
  4. Merge ops first, snapshot last. 2,000 merged token appends cost 26 KB, sent one by one 46 KB; a snapshot of a +member with 500 KB of history would cost 508 KB.
    5. +
  5. Calls need their own window. A tab firing 100,000 calls pushed another client's p95 to 345 ms; with a window of +64 respected by the caller it stayed at 0.66 ms and the flood waited in the buggy tab.
    6. +
  6. Nothing is refused for being too many (decision K16): extra state is merged, extra calls wait.
    7. +
+

Words used throughout

+

A link connects two hosts. The provider owns an object and sends its state; the consumer holds a +reference to it. A hold is one reference over a link, with its own stream of ops (Delta's changes). A +snapshot is a hold's whole state. An acknowledgement (ack) is a tiny message the consumer sends after it +applied a state message. In flight means sent and not yet acknowledged. Coalescing means merging the ops that +wait to be sent. A relay is a host in the middle (the server between worker and tab).

+
+ +
I

Where the Bytes Hide

Why the sender cannot see a slow phone.

+

A WebSocket in Node reports bufferedAmount: bytes the library still holds. It looks like the perfect signal: stop +sending when it grows. The experiment says otherwise.

+
workercommits tokensserver WebSocketbufferedAmountkernel + proxy buffersphone64 KB/stoken 1 … token 9,000written at once (bufferedAmount stays near 0)drains at 64 KB/smegabytes waitingtoken 9,001 (the newest)queued behind everythingarrives seconds later
Figure 1. A flood of state messages to a slow phone through a proxy. The sender's buffer stays small; the kernel's and the proxy's do not.
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Flood, ~9,800 × 128 B per second, 64 KB/s linkSender memoryNewest state the phone saw
no flow control7.0 MB queued in the WebSocket, heap +18.5 MB9.4 s old
cap bufferedAmount at 64 KiB, merge pending state+0.46 MBstill 9.4 s old
cap at 1 MiB (Fable's rerun)small3.8 s, growing to 10.4 s
at most 1 to 4 acknowledged messages in flight, merge pending state+0.4 MB0.0 s
+

The transport's backpressure protects the sender's memory, but freshness depends on buffers the sender cannot see: +the operating system's socket buffer, autotuned to hundreds of kilobytes, and every proxy on the way. A normal +streamed reply (50 tokens per second, 23 bytes each) never fills them; a burst or a tunnel does. Only the receiving +application knows what it has applied.

+
+ +
II

Acknowledged Messages

A window of two messages in flight, and why it adapts to any link.

+

The rule: a provider has at most K = 2 state messages in flight per link. The consumer sends ack (three bytes in +the binary codec) after it applied one. While two are in flight, new commits do not produce messages; their ops wait, +merged per hold, and go out in the next message once an ack arrives.

+
Two messages in flight on a slow link

Fast link

Commits 1, 2, 3 each become a message; each is acknowledged before the next commit. Nothing waits: the window never +fills, so a normal stream behaves exactly as without flow control.

+

The link slows

Messages 1 and 2 are in flight. Commits 3 to 400 arrive; their ops are folded into the pending queue of each member +they touch (chapter III). Memory: the net change, not the history; at most 10,000 ops per member.

+

An ack arrives

ack for message 1. The provider sends message 3, carrying the folded ops of commits 3 to 400, encoded once. +The phone applies them as one batch and shows the newest text.

+

Self-clocking

The slower the link, the more each message carries; the faster, the smaller and more frequent. Nobody measures +bandwidth: the acknowledgements pace the sender.

+
+
+
+ + + +
+ +
+
+ +

The simulation above shows the same effect in miniature. Drag the link below the commit rate: the upper lane's queue +grows without bound and the phone falls further behind every second; the lower lane never has more than two messages +waiting, and the phone is at most a message behind.

+
+ +
III

Fold the Backlog

What a waiting hold keeps while the window is full, and why it never measures sizes.

+

While messages are in flight, each hold member keeps the ops of new commits in a pending queue and folds them: +a new op merges into the last pending op on the same path when that is safe, so a backlog keeps the net change, not +the history. When an acknowledgement frees a slot, the queue is encoded once into one message.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Last pending opNew opKept
a text "Hel"a text "lo"one a text "Hello"
s count 41 or an appends count 42the s: it overwrites what came before
anythingr valueonly the r: a replacement makes everything before it moot
anything on a path in the indexa structural op (p, m, d, t, s of an object)appended; the index is cleared
+

The merge partner is found through a leaf index: a map from path to the last pending append or primitive set on +that path. Leaf ops on different paths commute, so a new op may merge into an earlier one, not only into its +neighbour; a status field changing between two tokens no longer stops the reply from folding. Any structural op +clears the index, because after a splice entries[7] may name another entry. Past 10,000 pending ops the queue keeps +only "replace with the current value", which then costs nothing while the consumer stays slow.

+

Why no size rule. The first idea was to send a snapshot once the merged ops outgrow the member. The reviewers +measured it on real p2 data and it failed three ways: "snapshot bytes plus sent op bytes" is no bound (an interned path +lets a value grow faster than its ops: 5,404 B of snapshot after 832 B of ops), it drifts 1,254 times too high under +churn, and an exact size needs a size on every node of the value. Folding alone came within about 10% of an exact +oracle, with no measuring at all.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + +
60 s stall, all workloadsUncompressedCompressed
merged ops, no folding466 KB166 KB
adjacent folding399 KB147 KB
leaf index (the decision)372 KB145 KB
exact oracle (not buildable)335 KB136 KB
+

One queue for every slow consumer. The same queue serves Chord's shared watch handle: a listener still busy with +the last frame gets the commits made meanwhile as one call with the newest value and the folded ops, instead of +today's "after 100 frames, replace everything". With a listener taking one second, a conversation view's watch went +from 402 calls and 17.9 MB of ops to 312 calls and 257 KB. Durable's event stream keeps a queue of its own, because +events are not state: folding would drop them.

+

Animation of the fold: How folding works.

+
+ +
IV

Calls Get Their Own Window

Why acknowledged state does not protect against a call flood.

+

Calls are small and the provider acknowledges nothing until they finish, so a byte or message window on state says +nothing about how many handlers run. A tab with a bug:

+
for (const note of notes) ref.add(note, ctx);   // 100,000 calls, none awaited
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
100,000 calls, handlers awaiting 10 msFinished inProvider memoryAnother client's p95 round trip
no limit0.66 s+56.5 MB, 43,824 running129 ms (345 ms with trivial handlers)
provider-side semaphore of 6415.9 s+42.5 MB, 96,736 queued on the provider0.38 ms
caller-side window of 6417.1 s+4 MB, nothing queued on the provider0.66 ms
handlers doing 10 ms of CPU workblocked about 1 s, with or without a limit
+

The provider's hello announces a call window N (256 by default) per origin (chapter VI). A consumer keeps at +most N of its own calls unanswered per link; further calls wait in the link's call queue, in order. Nothing is refused (K16); a peer that sends +more than N is a protocol error. The flood waits in the tab that caused it. CPU-heavy handlers block the event loop +whatever the window: that stays the handler author's responsibility.

+
+ +
V

Relays

One link to a worker carries many tabs. Decided: flow control per origin.

+

The server relays every tab's holds to a session's worker over one link. Three ways to apply the acknowledged window +across that hop were considered:

+
WORKERSERVERTABSworkerK = 2 per linkserveracks upstream after downstreamfast tabdesktopslow tabphone in a tunnelstatestatestate, slowack (late)ack only when every tab ackedSimplest, but the slow phone holds the worker's one window, so the fast tab's updates arrive in coarser, later batchestoo.
Figure 2. (a) End-to-end acknowledgements: the server acknowledges the worker only after the tab did.
+
WORKERSERVERTABSworkerK = 2 per linkserveracks at once; window per tabfast tabdesktopslow tabphone in a tunnelstateack at oncestate, currentwindow full: its parts droppedreset: fresh snapshotsThe server cannot fold opaque parts, so it drops them and asks for snapshots. Built and measured in slice 4a: a 1 sstall cost three full snapshots (~1.5 MiB) and ~24 s to catch up under streaming.
Figure 3. (b) Acknowledgements per hop: the server acknowledges at once and resets a slow tab.
+
WORKERSERVERTABSworkerK = 2 per (link, origin)serverforwards; FIFO of sendersfast taborigin 1slow taborigin 2, in a tunnelstate @1, state @2state, currentstate, slowack (late)ack @2 when the tab ackedThe relay numbers each tab (an origin) and forwards at once; each tab's ack goes back to the worker that sent themessage. The worker folds the slow tab's commits in that origin's pending queues, exactly as for a direct tab; thefast tab is another origin with its own window.
Figure 4. (c) Flow control per origin (decided 2026-10-24): the worker keeps a window per tab; the relay only forwards.
+

How (c) works:

+
    +
  • Origins. The relay gives each downstream link a number, never reused; every message upstream carries it +(origin), and the worker keys holds, calls, windows and acks by (link, origin, id). Ids are not renumbered.
    • +
  • What the relay keeps. Per relayed hold its upstream link and its original hold message (to route calls and +hold again after a worker restart); per tab a FIFO of which worker sent each message in flight, so each in-order +ack goes back to the right worker. Nothing is parsed, folded, dropped or reset.
    • +
  • Calls. N is per origin at the worker; the relay forwards calls at once, each tab bounded by its own window.
    • +
  • A tab leaves. Its link closes; the relay sends closed { origin } to each worker it relayed to, which releases +that origin's holds and aborts its calls.
    • +
  • Cost. A tab served by W workers may have 2W messages in flight (plus the relay's own 2). Two bytes of origin per +message on the worker link.
    • +
+
+ +
∴

What Changes in the Design

The converged scheme, what it replaces, and the relay decision.

+

Converged (Astra and Fable, after cross-checking each other's experiments): at most K = 2 acknowledged state +messages in flight; pending ops folded per hold member through a leaf index, capped at 10,000; a call window N +announced in hello, extra calls queued by the caller; holds unlimited; size limits stay a separate check.

+

Replaced from the design (§3, §22): the byte window and its credit messages, snapshot admission and debt (R19, Q3), +cut C12's call-credit accounting, any adapter backlog API. IPC and WebSocket behave the same, because acknowledgements +are messages.

+

Relays (owner): per-hop acknowledgements (b) were decided on 2026-10-23 and built in slice 4a; the benchmarks +showed a slow tab recovering through resets far worse than a direct tab (24 s and three snapshots after a 1 s stall). +On 2026-10-24 they were replaced by flow control per origin (c): the worker keeps window, pending queues and calls +per (link, origin), the relay only forwards and returns acks. Measured after the rework: a slow tab through the relay +catches up exactly like a direct one. The circuit breaker stayed undecided; the heartbeat (15 s ping, 30 s close) +was built instead.

+
+ +
A

Measurements

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ExperimentSetupResult
normal stream50 × 23 B/s through 64 KiB/sno buffering, frames ≤ 2 ms old
flood, no control~9,800 × 128 B/s, 10 s7.0 MB in bufferedAmount, heap +18.5 MB, newest state 9.4 s old
flood, bufferedAmount ≤ 64 KiB + mergingsameheap +0.46 MB, newest state 9.4 s old
flood, 1 to 4 acked messages + mergingsameheap +0.4 MB, newest state 0.0 s old, 98 % fewer messages
client stops reading (loopback)20k × 90 B/s, 15 s23.4 MiB buffered, RSS +121 MB; merging above 1 MiB: flat
IPC to a busy child623k sendssend() false from ~64 KB, parent +1,035 MB; the callback acknowledges the write, not the handler
call flood100,000 calls, see chapter IVcaller-side window: +4 MB, other client p95 0.66 ms
slow tab through a relay, (b) resets64 KiB/s, 100 tokens/s into a 500 KB view, 1 s stall~24 s to catch up, three full snapshots (~1.5 MiB)
slow tab through a relay, (c) per originsame, 1 s / 10 s / 60 s stallcaught up after 0.02 / 0.08 / 0.38 s, 485 B / 4.1 KiB / 23.7 KiB (same as direct)
+
+ +

Flow control for Chord links · fourth edition · 2026-10-24 · based on the reviewers' experiments and design §3, §22

+
+ + diff --git a/packages/chord/docs/next/flow-control/window/index.html b/packages/chord/docs/next/flow-control/window/index.html new file mode 100644 index 000000000..52ac8c39c --- /dev/null +++ b/packages/chord/docs/next/flow-control/window/index.html @@ -0,0 +1,228 @@ + + + + + +A windowed conversation view + + + +
+

A windowed conversation view

+

Proposal for Durable: the view holds a window of the newest entries with a stable start; older entries are fetched in pages; a reconnect resumes after the last entry it saw.

+ + 
type ConversationView = {
+  conversation: ConversationRecord;
+  head?: EntryRecord;               // the current compaction summary, if any
+  entries: readonly EntryRecord[];  // the window: active entries with id >= from
+  earlier: boolean;                 // older active entries exist before it
+  docs: { "pi.live", "pi.agent", "pi.inbox", "pi.provider", "pi.usage" };
+};
+conversation.watch(context, { from?: EntryId, tail?: number });          // default: the last `tail`
+conversation.entries({ before, order: "descending" }, limit, cursor, ctx); // exists today
+ +

Watch it work

+

Shown with tail = 4 (the proposal: about 50). The window grows by appends until it holds twice tail; then the oldest half leaves in one splice.

+
+
+ + + + + +
+
+
store: every active entry
+
client: view window (blue), kept or fetched (purple)
+
+
+ window 0 entries + ops sent 0 + last snapshot - +
+
+
+
+ +

What each step costs

+ + + + + + + + +
stepwire
a new entryone append splice: p entries <len> 0 [entry]
window reaches 2 × tailone splice removing the oldest half: p entries 0 <tail> [], once every tail entries
streaming into the newest messageappends on docs["pi.live"].generation.message, unchanged
hold, re-hold, overflowa snapshot of at most 2 × tail entries plus the live documents, never the whole transcript
scroll upa page from entries(), outside the view: the client keeps it, the view never grows
reconnect with from: lastSeen + 1only the entries it missed plus the documents (issue #11327)
+

Why a stable start instead of “always the last 50”: a sliding window would remove one entry at the front for every entry appended at the back, two ops per entry, and every index in the window would shift. With a stable start, entries only append, and the front moves in one splice now and then.

+
+ + + + diff --git a/packages/chord/docs/next/layer-1.md b/packages/chord/docs/next/layer-1.md new file mode 100644 index 000000000..505fe7294 --- /dev/null +++ b/packages/chord/docs/next/layer-1.md @@ -0,0 +1,311 @@ +# L1: data and contracts (spec) + +Status: **approved (owner, 2026-10-22)**, rewritten 2026-10-22 as the single source for a from-scratch implementation +(the first implementation was discarded for code quality). Every rule here is decided; the decision log +(`packages/chord/docs/next/decisions.md`) has the reasons. Code style: the section "Chord v2 Code Style" of the root +`AGENTS.md`. + +## What L1 is + +Everything about **data** that the upper layers need and that does not depend on hosts, objects or links: how an +object type is declared and evolves (contracts), what a value is, how params become an identity, how arguments are +checked, what a principal key is, what an error is. Pure functions, types and weak caches; no I/O. + +Source: `packages/chord/src/next/contracts/` (declarations) and `packages/chord/src/next/data/` (per-call checks, +principals, errors), split into files by concept. Everything else under `src/next/` is the frozen old kernel: never +imported, never edited. Reused: `src/context/` (`Context`, `BACKGROUND_CONTEXT`, `context.telemetry`) and +`src/json.ts` (`JsonValue`), both unchanged. Schemas are TypeBox 1.3 (`typebox` in `node_modules`). + +## 1. Values + +```ts +type Value = null | boolean | number | string | Uint8Array | Value[] | { [key: string]: Value }; +``` + +- JSON plus byte leaves. No `undefined` inside values, no non-finite numbers, no foreign prototypes, no accessors. +- Values from links are already strict JSON plus bytes; values from same-host code are plain data **by convention**, + immutable once handed over. L1 never walks, copies or freezes a value to enforce that. State is never checked. +- `canonicalJson(value: JsonValue, context)`: sorted object keys, `-0` written as `0`. Used only for principal and + revocation keys, which are JSON (no bytes), so it lives with principals. + +## 2. Contracts: declarations + +A contract is the declared type of an object: name, kind, version, params, members. Two kinds: + +```ts +const Notes = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: arr(Note) }, methods: { add: method(obj({ text: str() }), str()), clear: method() } }, + reader: { state: { notes: arr(PublicNote) } }, + }, +}); +const Info = local("app.session-info", { + version: 1, + params: {}, + members: { init: obj({ title: str() }), clear: method() }, // a schema is state, method(...) is a method +}); +``` + +- `remote` has variants (per-principal views, at least one); `local` has one flat `members` map and no variants. +- `method()`, `method(args)`, `method(args, result)`, `method(undefined, result)`. An omitted side is `null` on the + wire: a no-argument method is called with `null`. +- **Variants (D11):** a member name has the same kind (state or method) in every variant that has it; a method has the + same signature (args and result) in every variant; state schemas may differ per variant (redaction). Enforced in the + types of `remote(...)` and at declaration (for untyped callers). +- **Closed argument objects (Q12):** every object schema inside a method's arguments sets + `additionalProperties: false`, nested ones included, raw `Type.Object` included; record schemas (`map`) are allowed. + Checked once per declared or edited method, at declaration. `obj()` always produces closed objects and cannot be + opened. +- Declaration checks (the types cannot express them for JavaScript callers): nonempty name, version a positive safe + integer, no unknown declaration keys, params policy (section 4), D11, closed arguments, the schema vocabulary + (below). Each fails with an `Error` naming the path (`notes.editor.methods.add.args`). +- Declarations are immutable by convention: a contract holds Chord-owned snapshots of its schemas (copied once at + declaration, so later changes to the app's schema objects cannot desync validation from the fingerprint), never the + app's schema objects. Nothing in L1 is frozen (owner, 2026-10-22). Contracts are branded: hand-built or spread copies are refused by `evolve`, `checkAccepts` + and `bindProvider` (a private `WeakMap` from contract to its data). +- **Placement** is not part of L1 contracts: the `placement` field arrives with host kinds (L7) and is never part of + the fingerprint. + +### Schema helpers + +`str({ min, max, pattern })`, `num({ min, max })`, `int({ min, max })`, `bool()`, `nul()`, `lit(v)`, `oneOf(...literals)`, +`union([...])`, `obj(props)`, `arr(items, { min, max })`, `map(value)` (string-keyed record), `opt(schema)`, +`bytes({ min, max })`, `unknown()`, and `type Static`. They return ordinary TypeBox schemas, so `Static` and TypeBox's +compiler work unchanged. `min`/`max` mean string length, numeric bounds or array length. + +`bytes()` is a `Uint8Array` leaf with optional length bounds, checked by `instanceof` and length, never enumerated. +Its enumerable form is the data marker `{ type: "bytes", minLength?, maxLength? }`; the TypeBox check is attached +non-enumerably and rebuilt from the marker whenever a schema is snapshotted, so no callback is ever copied from app +data. + +**Vocabulary (rejected at declaration otherwise):** data primitives, arrays and tuples, objects and records, unions and +intersections, `$defs`/`$ref` (use `$ref` for recursion), constraints (`pattern`, `format`, bounds, `multipleOf`, +`uniqueItems`, property counts), `default`, `readOnly`/`writeOnly`, annotations, bytes. Rejected: conditional and +dependent schemas, `not`, `contains`, `propertyNames`, `unevaluated*`, `contentSchema`, `definitions`, unknown +keywords, TypeBox's executable features (`Unsafe` other than bytes, refinements, transforms), accessors, symbol keys, +cyclic objects, non-JSON data. + +### Fingerprint + +Every version gets a fingerprint at declaration: `hash64` (a portable 64-bit hash, two FNV-style lanes, 16 hex +characters; not a security boundary) of `JSON.stringify({ normalFormVersion, value })` over a **normal form**: + +- object keys sorted; `required` and `enum` sorted; union, intersection and tuple order **kept** (it is meaning); +- annotations dropped on schema nodes only (`title`, `description`, `examples`, `$comment`, `deprecated`); defaults, + formats, patterns, bounds and `additionalProperties` kept; +- params as ordered `[fieldName, schema]` pairs (declaration order is identity order); members as sorted dictionaries; +- `{ kind: "remote", params, variants }` or `{ kind: "local", params, members }`; placement excluded. + +`NORMAL_FORM_VERSION = 2` is exported; changing the normal form changes it and is a link-protocol change. +One schema-aware walk produces both the snapshot and the normal form. + +### Evolution (Q10) + +A history starts with a base written in full (`remote()`/`local()` with an explicit version). `evolve(prev, (e) => +e...)` appends the next version (`prev.version + 1`) with structured edits: + +| Edit | Meaning | +| --- | --- | +| `state(variant, name, schema)` | change one state member (local: `state(name, schema)`) | +| `addState(variant, name, schema)` | add a state member | +| `removeState(variant, name)` | remove a state member | +| `args(name, schema)`, `result(name, schema)` | change one side of a method in every variant that has it | +| `method(name, args, result)` | replace both sides | +| `addMethod(variant, name, args, result)` | add a method (local: no variant); to another variant only with the same signature | +| `removeMethod(name)` | remove a method from every variant | + +- Params and the variant set never change by evolution. A change to either is a **new history base**: a new full + declaration with an explicit version and no accepted older versions. +- A member's kind never changes, also not after a removal and re-add. Checks run after every edit. +- `evolve` must return the builder it was given (no foreign builders). Every version is materialized and + fingerprinted at declaration, so an old plugin's copy of v1 has the same fingerprint as the history's v1. +- The types track the whole history: `Contract` carries the version, the current description and every step with its + edits, so `checkAccepts` can require exactly the right mappings at compile time. + +### Accepted versions and mappings + +A provider may accept older versions of its contract with mappings it owns: + +```ts +const NotesV2 = evolve(NotesV1, (e) => e.state("editor", "notes", arr(NoteV2)).args("add", NoteV2)); +const accepts = { + 1: { + state: { editor: { notes: (current) => current.map(toV1) } }, // current member -> old member + methods: { add: { args: (old) => toV2(old) } }, // old args -> current args + }, +}; +checkAccepts(NotesV2, accepts); // { versions: [{ version: 1, fingerprint }] } +``` + +`checkAccepts(contract, accepts)` is pure: it validates the table without calling any mapping and returns the accepted +versions with their fingerprints. Mappings are **direct** (old to current, never chained). For accepted version `v`, +considering every edit after `v`, a mapping is required exactly for: + +- a changed state member: `(currentMember) => oldMember`; +- a removed state member: `(currentVariantState) => oldMember` (the whole current state of that variant), or + `"refuse"`; +- a method whose args changed: `{ args: (old) => current }`; whose result changed: `{ result: (current) => old }`; + both: both functions; +- a removed method (from any variant that had it in `v`): `{ call: async (oldArgs, context) => oldResult }` or + `"refuse"`; `call` and `"refuse"` are also allowed for changed methods. + +Added members need nothing. No-op edits and changes made and reverted still require (identity) mappings: the check is +by edit, not by schema comparison. Unknown versions, the current version, missing mappings, unnecessary keys and wrong +shapes fail at declaration with the path (`notes accepts[1].methods.add`). An absent `state` or `methods` section means +empty and is allowed only when nothing is required there. The same rules hold at the type level (`AcceptsTable`). + +`Implementation` derives the provider's shape: per variant, state getters `() => value`; methods +`(args, context) => Promise`. + +## 3. Principals + +- The app's principal type `P` is plain JSON with a `kind` field: no schema, no version, never persisted by Chord + (objects persist their own projection, e.g. chat's `Author`). +- `principalKey(principal, context, customKey?)`: default `canonicalJson(principal)`; an app may pass its own key + function, which must be the same on every host. Cached per principal object in `WeakMap`s; a custom key function's + cache is keyed by the function too, so a dropped function is never retained by a live principal. +- `revocationKeys(principal, revokedBy, context)`: the canonical JSON of each JSON value `revokedBy(principal)` + returns (`host.revoke` is L7). + +## 4. Params and identity + +```ts +const { params, key } = checkParams(Notes, { session: "S1" }, context); // throws invalid/params +const identity = identityKey(Notes, key, principalKey(principal)); // "notes|S1|" +``` + +- Declaration policy: params are a flat record of required fields, each `str()`, `int()`, `bool()` or a union of + string literals (`oneOf("chat", "agent")`), constraints allowed; no defaults, optional, nested, bytes, numbers other + than integers. Checked in the types and at declaration. +- `checkParams(contract, raw, context)`: one compiled closed-object check of `raw` (constraints enforced), the compiled + validator cached per contract in a `WeakMap`. Then the key is the field values in declared order, each escaped + (`\` and `|`), joined with `|`. It returns the caller's own object: no pre-check walk, no copy, no freeze. +- `identityKey(contract, key, principalKey)`: `name|key|principalKey`, each part escaped; `local` takes no principal + key (by type). The key is an in-process map key; links carry the params themselves. + +## 5. Arguments + +```ts +const provider = bindProvider(NotesV2, accepts); // once per registration and reload +const checked = checkArgs(provider, "add", raw, boundVersion, ctx); // invalid/args with path +// { kind: "method"; args (current shape); result?: (current) => old } | { kind: "call"; args (old shape); call } +checkArgs(provider, "clear", null, 2, ctx); // no-argument methods take null +``` + +- `bindProvider(contract, accepts)` runs `checkAccepts` once and keeps the app's accepts table as it is (immutable by + convention; no copy, no freeze). It builds a private dispatch table, held in a `WeakMap` keyed by the returned + provider: accepted version -> method -> `{ schema, validator, mapping }`. Validators compile on first use per + (version, method); a weak cache keyed by schema object lets equal schema objects share a compiled validator. +- `checkArgs(provider, method, raw, boundVersion, context)`: three map lookups, one compiled check against the + provider's schema for that version, then at most one direct `args` mapping old -> current. It returns a tagged + value (owner, 2026-10-22): `{ kind: "method"; args; result? }` (args in the current shape; `result` maps the current + result back for an old-version caller) or `{ kind: "call"; args; call }` (a replaced method: the checked old args + and the provider's `call`, which L4 invokes). Current-version calls are never mapped. Unknown method: + `invalid/member`. Version not accepted, or `"refuse"`: `contract_mismatch`. Mapping output is trusted. +- Every caller is checked, `local` included (Q15). Unknown fields fail (closed schemas); nothing is stripped, copied, + frozen or defaulted. Results and state are never checked. +- Memory: a binding lives as long as its registration and the calls in flight. No module-level strong cache holds a + contract, binding, schema or validator. +- TypeBox keeps its last build in module-level accumulators: after every `Compile`, reset them (`ResetFunctions`, + `ResetExternal`, in `finally`), so no build state is retained. + +### A failed check + +A failed compiled check uses TypeBox's public `Errors(schema, value)` and reports the first error. The error is +`invalid/args` or `invalid/params`, message `args/text: ` (fixed text for the two common cases: +`args/extra: unknown field`, `args/text: missing`), `details: { path }` with TypeBox's +instance path (`/text`, `/` for the root), including missing (`required`) and unknown (`additionalProperties`) +fields at their own path. TypeBox does not escape `/` and `~` inside keys, so a key containing them makes the path +ambiguous (owner, 2026-10-22: accepted; it is a diagnostic). + +## 6. Errors + +```ts +type ChordErrorCode = "invalid" | "contract_mismatch" | "forbidden" | "gone" | "not_ready" | "released" + | "too_large" | "unavailable" | "aborted"; +``` + +| Code | Reasons | +| --- | --- | +| `invalid` | required: `args`, `params`, `member`, `placement`, `reentrant`, `read_after_bulk`, `cycle` | +| `contract_mismatch` | optional: `updating` | +| `unavailable` | optional: `locked`, `park_timeout` | +| `too_large` | optional: `part` | +| others | none | + +- One `ChordError` class (an `Error`), with `code`, `reason`, `message`, `details?: JsonValue`. The constructor, + instances and JSON form are tagged unions: a wrong pairing is a type error, and narrowing on `code` narrows + `reason`. `toJSON()` emits `{ code, reason?, message, details? }`; stack and cause never cross the wire. +- `PROTOCOL_VERSION = 1` is exported. Reasons and pairings are link protocol: adding or moving one bumps it. A test + pins the table together with the version and fails with "bump PROTOCOL_VERSION" when they diverge. L6 compares the + version in the handshake before accepting anything. +- `chordErrorFromJson(raw)`: `raw` is fresh JSON from the link decoder (a trust boundary). Checks the envelope only: + known code, a reason allowed for that code (required for `invalid`), string message, no other keys. `details` is + kept by reference: no walk, no copy, no counters. +- Helpers: `isChordError(error, code?)`, `invalid(reason, message, details?)`, `asChordError(error, label)` (wraps a + foreign error as `unavailable`). +- `FailureCause = { kind: "transient"; error } | { kind: "broken"; error }`: the failing stage decides, never the + error code. `broken(cause)` marks an open's failure as broken (bad configuration); `openFailure(cause, label)` turns + an open's thrown value into the cause. Used by L4. +- Plugin errors have their own shape (L4); they never use Chord reasons. + +## Cost per hop + +| Hop | Work | +| --- | --- | +| a `get` | one compiled closed-object check, one joined key; no walk, no copy | +| a call | three map lookups, one compiled check, at most one direct mapping; no copy | +| first `get` of a contract | one compilation | +| first call per accepted version and method | one compilation (or a cache hit by schema object) | +| a failed check | one counted diagnostic walk (`Errors` builds the whole list on this rare path) | +| a first default principal key, a revocation key | one canonical walk | +| declaration | snapshot, closure check per method, fingerprint per version | +| binding | `checkAccepts` once, dispatch table built, nothing compiled | +| a state change | nothing in L1 | + +**Counters:** `chord.data.walks`, `chord.data.validations`, `chord.data.compilations` are telemetry counters +(`context.telemetry.metrics?.counter(name).add(1)`), always on. The function doing the work counts it; call sites never +count for others. Tests assert them per hop with the in-memory telemetry adapter. + +## Footguns + +- A provider must not mutate arguments or params it received (caller-owned, immutable by convention). +- `principalKey` must be the same function on every host; a different one splits identities silently. +- Params keys must escape the separator, so two different params never join to one key. +- Mappings are direct: when version 3 arrives, every still-accepted version's mappings are updated (compose step + functions: `(old) => v2to3(v1to2(old))`). + +## Tests (one group per mechanism, every case) + +- Contracts: every declaration check with its error path; D11 (kind and signature across variants, at runtime and + in the types); closed arguments (nested, raw `Type.Object`, records allowed, edits checked, state not checked); + schema vocabulary (each rejected feature); bytes (marker, bounds, check rebuilt after snapshot, no callback copied); + snapshots are independent of later changes to the app's schema objects; branding (spread copies refused). +- Fingerprints: stable across key order and annotations; sensitive to defaults, bounds, union order, params order, + `additionalProperties`; the same v1 from a separate declaration; local vs remote differ; a fixed known hash for one + fixed contract (protects the normal form). +- Evolution: each edit; kind changes refused (also remove then re-add); params and variant set fixed; new bases; + long chains (versions beyond 9, 99); history types. +- Accepts: every required-mapping rule above, at runtime and at the type level (`@ts-expect-error` for missing, + unnecessary and wrongly typed mappings); `"refuse"`; `call`; direct mapping over several versions; no mapping is + invoked by `checkAccepts`. +- Params: missing, extra and wrong fields with their paths; literal membership; constraints; the same key whatever + the property order; separator escaping (no two params join to one key); declaration refusals. +- Arguments: unknown fields (nested too); no copy (identity of the passed object); `local` and `remote`; bytes by + `instanceof` and length; old version checked with the old schema then mapped; current never mapped; `null` for + no-argument methods; unknown method; unaccepted version; `"refuse"`; lazy compilation (counted once per version and + method). +- Principals: default key, custom key, caching (no second walk), revocation keys. +- Errors: tagged union in types (`@ts-expect-error` for wrong pairings); JSON round trip; envelope checks (each + refusal); details kept by reference; the pinned protocol table; `FailureCause`, `broken`, `openFailure`. +- Cost guards: the counters per hop in the table above. +- GC: dropped contracts, bindings, dispatch tables, mappings, schemas and validators are collected after the last call + settles; params validators per contract; custom principal-key functions dropped while their principals live; nothing + retained by TypeBox's build accumulators. + +## Size + +Target: well below 1,500 source lines for `contracts/` and `data/` together, tests excluded. The type-level history +and mapping types are the heaviest part; keep the rest lean. diff --git a/packages/chord/docs/next/layer-2.md b/packages/chord/docs/next/layer-2.md new file mode 100644 index 000000000..e5dc5dee2 --- /dev/null +++ b/packages/chord/docs/next/layer-2.md @@ -0,0 +1,216 @@ +# L2: watchables and derives (spec, for the owner's approval) + +Status: **superseded (owner, 2026-10-22)**: the L2 code built from this spec was discarded (too much generic +machinery for callers that did not exist yet: nodes, sinks, deliveries, publications) and the build switched to +vertical slices (`slices.md`). Still valid: the section "Decided while specifying" below, and Chord's shared watch +handle (`src/watch.ts`). Everything else here is background only; do not implement from it. + +Original status: approved; open points decided (below). The single source for implementing L2; every rule comes from the decision log +(`packages/chord/docs/next/decisions.md`: "Watchable and one watch API", W1, W2, W3, W4, W12, W14, Q14, R23, R24, +"Changing contributions, `self.writable`", "Cost counters") unless marked **Decision needed**. Code style: the section +"Chord v2 Code Style" of the root `AGENTS.md`. + +## What L2 is + +Everything observable on one host, and how a change travels: a value changes inside a **publication** (one atomic step +that may change many sources), the derives over it are recomputed right after, and every consumer sees each +publication once, completely, in order. No objects, no storage, no links: L3 (store) publishes document commits +through it, L4 (objects) builds members from it, L6 (links) encodes what it flushes, L9 (registries) composes from it. + +Source: `packages/chord/src/next/watch/` (one directory; files by concept). Reused unchanged: Chord's shared +`src/watch.ts` (the watch-handle types **and implementation**, moved there from Durable's `CommittedWatch` by a pi change +first, owner 2026-10-22; Durable uses the same implementation), `src/delta/` (`Op`, `diffRevisions`), `src/context/`, +`src/json.ts`, and L1's `next/data` errors (`invalid/reentrant`). Nothing from `next-frozen/`. + +## 1. The two kinds + +```ts +interface Watchable { // documents, derives, reference members, constant(v), Durable views + readonly value: T; + watch(context: Context): Promise>; // listener (value, ops, context) +} +interface NativeWatchable { // writable(initial), composed registry values + readonly value: T; + watch(context: Context): Promise>; // listener (value, context) +} +``` + +- A `Watchable` carries JSON and changes with ops (Delta's `Op[]`): the ops turn the previous value into the next. + Values are immutable by convention: a new value shares unchanged parts with the old one; nobody mutates a value. +- A `NativeWatchable` carries any in-process value (functions, runtimes); a change is a new value by identity + (`!==`). Mutating a native value in place notifies nobody. It has no ops and can never be a contract state member or + go on the wire (Q14, W4). +- Both carry a runtime kind tag (`"json"` / `"native"`), so L4 can refuse a native one as state with one comparison + (no value walk). + +## 2. Publications (W1, W12) + +```ts +const publication = createPublication(); // one per host +publication.publish(() => { // one atomic step + notes.set(nextNotes, ops); // adopt: pointer swap + mark + index.set(nextIndex, indexOps); +}); // flush runs here, synchronously, before publish returns +``` + +- `createPublication()` returns the host's publication. A **source** (`jsonSource(initial, publication)`, + `writable(initial, publication)`) belongs to one publication. +- `publish(adopt)` runs `adopt` synchronously. Inside it, `source.set(value, ops)` (JSON) or `source.set(value)` + (native) swaps the value and **marks**: dependent derives become dirty, and the ops are appended by reference to each + watch's queue and each sink's batch. O(1) per dependent. Then, still synchronously and without any `await`, the + **flush**: + 1. recompute every dirty derive that has a consumer, in dependency order (each derive once per flush, after all its + sources), diffing its old and new value with `diffRevisions` into ops, and marking its own dependents; + 2. deliver each touched sink its batch for this publication (section 6); + 3. call the publication's flush listeners once (L6 uses this to send one message per link); + 4. schedule the queued watch frames (their listeners run later, asynchronously, in order). +- Every change goes through `publish`: L3's commit step (after the backend commit; S1), in-memory commits (no store + seq, the publication's own counter, W12), `writable.set` (which publishes itself), registry recomposition and + inbound wire batches. Each publication has a number (a counter per publication) for ordering and lazy checks. +- **Reentrancy:** during `adopt` and the flush, `publish`, `watch`, `attach`, a sink's `detach` and `writable.set` + throw `invalid/reentrant` (W1: no new transaction, watch or release inside the step). Stopping a watch handle is + allowed anywhere: it only detaches, and delivery tolerates removal during iteration. +- **Contexts** (context last, required, as in Durable): `createPublication(context)` keeps the host's context (it + counts lazy-read computes, since a getter has none); `publish(adopt, context)` and `writable.set(value, context)` take the context + that counts the flush's work and becomes the frames' context. `publication.number` is public (L3's in-memory commit + number, W12); `publication.onFlush(listener)` returns its removal. +- **Source ends:** a JSON source has `set(value, ops)`, `retire()`, `end()` and `end({ error })`, all only inside + `publish` (events in publication order), delivered in that publication's flush after its batch: watches retire, end + or fail; sinks get a retirement as `update(null, [["r", null]])` then `end()`, an end as `end()`, a failure as + `fail(error)`. A failed source's `value` throws its error, so derives over it fail with the same error; a retirement + is a change to `null` for derives over it; an end changes nothing for them. `watch` or `attach` on an ended source + returns a handle or sink ended the same way. A writable has only `set`. +- A source set outside `publish` throws (a programming error, `Error`, not a ChordError). + +## 3. Derives (W1, W4, W14, R24) + +```ts +const count = derive([notes], ([n]) => n.list.length, publication); +const visible = derive([notes, settings], ([n, s]) => ({ list: n.list.filter(...), theme: s.theme }), publication); +``` + +- `derive(sources, compute, publication)`: sources are `Watchable`s and `NativeWatchable`s (mixed) created by L2 on + the same publication (sources, derives, writables, constants; a foreign watchable such as a Durable view becomes an + L2 source first, L3d); `compute` is a synchronous pure function of their current values; the result is JSON and the + derive is a `Watchable`. +- **Attached only while consumed:** a derive with at least one consumer (a watch, a sink, or a consumed derive over + it) is attached to its sources and recomputed in the flush. A derive without consumers holds no subscription; reading + its `value` recomputes it if any source's publication number moved since its last compute (lazy). So an unused + derive never keeps work or memory alive, and nothing leaks when consumers go away. +- Ops: when a consumed derive recomputes, `diffRevisions(old, new)` gives its ops (identity short-circuit: unchanged + shared subtrees cost one `===`). A native source's change marks the derive dirty without ops. A recompute that + returns the identical value (`===`) publishes nothing. +- **Dependency order:** a derive's rank is one more than its highest-ranked source; the flush recomputes by rank. + Sources exist before the derive, so cycles cannot be built (cross-object cycles are rejected at activation, Q5, L4). +- **A throwing derive** fails only itself: its `value` throws the error, its watches end `{ reason: "failed", error }`, + its sinks get a failure, and derives over it fail with the same error. It stays failed (broken: no automatic + retry; L4 rebuilds it on reload). The publication and every other consumer continue. +- No `previous` (W14). The footgun and its remedy are documentation: a derive that builds new objects for unchanged + data costs O(value) per flush per consumer; `mapShared` (section 7) keeps unchanged elements identical. + +## 4. Constant and writable (R23, W4) + +- `constant(value)`: a `Watchable` that never changes; its watches never get a frame and end only by `stop` or + cancellation. No publication needed. Used for start data. +- `writable(initial, publication)`: a `NativeWatchable` with `set(value)`; `set` publishes by itself (one publication + per call); an identical value (`===`) publishes nothing. Used for registry contributions known only at runtime. + +## 5. Watch handles (W2, W3; Chord's shared implementation) + +L2 does not implement watch handles: it uses the one implementation in Chord's `src/watch.ts` (Durable's semantics, +moved there by the pi change). Sources create a handle per `watch(context)` and feed it: `advance` per publication +(during the flush, by reference), `retire`, `end`, `end({ error })`, cancellation from the context's signal. + +- `watch(context)` returns the handle at once (the Promise resolves immediately for local sources; L5 reference + members may wait). It holds the value current at acquisition and buffers every later publication. +- `start(listener)` installs the one listener; frames are delivered in order, each awaited, never inline; the listener + gets a context without the abort signal. +- **Overflow:** after 100 pending frames the backlog collapses: JSON to one frame `[["r", newest]]`, native to the + newest value. +- **Retirement** is an explicit source event (`source.retire()`, used by L3 for retired documents): the last frame is + `[["r", null]]` with value `null`, then the watch ends `retired`. Never inferred from a `null` value (W2a). +- **Ends:** `stopped`, `cancelled`, `retired`, `ended` (the source ended), `failed { error }` (the source failed, e.g. + a throwing derive), `listener_error { error }`; `stop` drops pending frames and never aborts or awaits a running call. + Termination clears every reference into the source (W3). +- `NativeWatchHandle`: the same lifecycle with listener `(value, context)`. +- **W2b** (routines delivering the acquisition value first) is L8's. + +## 6. Sinks (internal, for L6 and L9) + +A sink is a synchronous consumer inside the flush, used where a watch's asynchronous delivery is too late: the wire +encoder (L6: one part per hold, one message per link per publication) and registry composition (L9). + +```ts +const sink = attach(watchable, { update(value, ops) {...}, fail(error) {...}, end() {...} }); // returns detach() +``` + +- `update` is called once per publication that touched the source, during the flush, with that publication's ops + (several sets of one source within one publication arrive as one batch, in order). Native sources: `update(value)`. +- A sink counts as a consumer (derives over the source stay attached). `detach()` removes it; detaching inside the + flush throws `invalid/reentrant`. +- Not exported to apps; L4-L9 import it from `next/watch`. + +## 7. `mapShared` (W14, design §12) + +`mapShared(f)` returns a function over arrays that calls `f` only for elements it has not seen (by identity) and +returns the previous result for unchanged ones, so a derive over a list of 2,000 notes with one change builds one new +object and `diffRevisions` emits one op. The cache is a `WeakMap` from source element to result. + +## Cost per hop + +| Hop | Work | +| --- | --- | +| a publication | per set source: pointer swap, O(1) per dependent (mark, append ops by reference) | +| flush | each dirty consumed derive: one compute plus one `diffRevisions`; nothing for unconsumed derives | +| a watch frame | the value and ops by reference; no copy, no diff | +| a sink update | the ops by reference | +| an unconsumed derive read | one compute if a source moved, else nothing | +| constant | nothing | + +**Counters** (telemetry, the function doing the work counts it): `chord.watch.computes` (derive computes) and +`chord.watch.diffs` (`diffRevisions` calls). No walks or copies anywhere in L2. + +## Footguns + +- A derive that rebuilds unchanged data costs O(value) per flush per consumer (use `mapShared`; R24: changing one + item of a flat array still copies the array's slots). +- Mutating a value in place breaks every consumer silently (immutable by convention). +- Async work in a derive is impossible by design; it belongs in a routine (L8) writing an in-memory document. + +## Tests (one group per mechanism, every case) + +- Publication: several sources in one publication reach a watch as consecutive frames and a sink as one batch; + publication numbers increase; setting outside `publish` throws. +- Derives: dependency order (diamond: A→B, A→C, B+C→D computes D once, after B and C); identical result publishes + nothing; native source marks without ops; lazy read recomputes only after a source moved; attach on first consumer, + detach after the last (no subscription left; GC); a chain A→B→C where B throws (C fails, A's other consumers + continue); a failed derive stays failed. +- Two sources published alternately with a derive over both: the derive sees every combination exactly once per + publication. +- Reentrancy: `publish`, `watch`, `attach`, `writable.set`, `detach` inside a derive and inside a sink throw + `invalid/reentrant`; `stop` inside a sink or listener is allowed and delivery continues correctly. +- Watch handles (the handle itself is tested in Chord's shared tests): through L2 sources, a watch started after 150 + publications; retire; end and fail from the source; stop during a running call; cancellation by the context. +- Constant and writable: constant never delivers; writable identical set publishes nothing. +- `mapShared`: unchanged elements return the identical result; a changed element is recomputed. +- Sinks: one batch per publication; failure and end; detach. +- Cost guards: computes and diffs per hop (an unconsumed derive computes nothing on publish). +- GC: stopped handles hold no source; unconsumed derives hold no subscription and are collected with their sources; + `mapShared` caches drop with their elements; retained revisions after overflow are bounded (W1's GC count). + +## Size + +Target: about 500 source lines, tests excluded. + +## Decided while specifying (owner, 2026-10-22) + +- **Watchable state is JSON only.** No bytes in documents, derives, reference members or any watchable value; bytes + appear only in method arguments and results (passed whole, carried as raw byte segments by L6) and in chunked + transfers. Delta stays JSON-only. Supersedes W4's "JSON plus bytes" for watchables. +- **`mapShared` lives in `next/watch/`** for now; it moves into Delta when v2 goes upstream. +- **C10 rejected:** `constant` (JSON `Watchable`) and `writable` (`NativeWatchable`) stay two, because Q14 needs the + kind distinction. +- **One publication per host** (`createPublication()`, created by the host and passed to every source, derive and + writable on it; apps never see it), never a module-level singleton. +- **No CBOR:** the wire stays JSON plus raw byte segments (R1, Q4); revisit only if L6 benchmarks show encoding as a + bottleneck. diff --git a/packages/chord/docs/next/slice-1.md b/packages/chord/docs/next/slice-1.md new file mode 100644 index 000000000..e0f17f077 --- /dev/null +++ b/packages/chord/docs/next/slice-1.md @@ -0,0 +1,236 @@ +# Slice 1: one host, local objects (spec, for the owner's approval) + +Status: **approved (owner, 2026-10-22)**, after Astra's and Fable's spec reviews. The single source for building slice 1, +together with the feature map in `slices.md` (entries assigned to slice 1) and `design.md`, read in full (the most +relevant sections: §3 host, §5-§7, §8, §9, §11, §12, §16, §18); rules come from the decision log +(`packages/chord/docs/next/decisions.md`); code style: the section "Chord v2 Code Style" of the root `AGENTS.md`. Builds on +L1 (`src/next/contracts`, `src/next/data`) and Chord's shared watch handle (`src/watch.ts`). + +## What an app can do after this slice + +```ts +const Notes = local("notes.store", { + version: 1, + params: {}, + members: { + notes: obj({ list: arr(Note) }), + titles: arr(str()), + count: int(), + add: method(obj({ text: str() })), + saveDraft: method(obj({ conversation: int(), body: str() })), + }, +}); +const Draft = local("notes.draft", { + version: 1, + params: { conversation: int() }, + members: { summary: str() }, +}); + +const NotesDoc = defineState("notes", { schema: obj({ list: arr(Note) }), version: 1, initial: () => ({ list: [] }) }); +const Drafts = defineState("drafts", { schema: obj({ body: str() }), version: 1, initial: () => ({ body: "" }), keyed: true }); + +const facet = defineFacet((host) => { + host.serve(Notes, (_, self) => { // empty params: lives as long as the host + const notes = self.state(NotesDoc); + const drafts = self.state(Drafts); // keyed: one document per conversation (P3 pattern) + const title = mapShared((note: Note) => note.text.slice(0, 20)); + return { + notes, + titles: self.derive([notes], ([n]) => title(n.list)), + count: self.derive([notes], ([n]) => n.list.length), + add: async ({ text }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(notes)).list.push({ text }); }, ctx); + return null; + }, + saveDraft: async ({ conversation, body }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(drafts, String(conversation))).body = body; }, ctx); + return null; + }, + }; + }); + host.serve(Draft, ({ conversation }, self) => { // params: lives while referenced + const store = self.get(Notes, {}); // declared here, live before this object is + return { summary: self.derive([store.count], ([c]) => `conversation ${conversation}, ${c} notes`) }; + }); +}); + +const host = createHost({ onError: (error) => log(error) }); // no address: local types only +await host.load([{ name: "@app/notes", facet }], ctx); +const notes = await host.get(Notes, {}, ctx); // live and hydrated +await notes.add({ text: "milk" }, ctx); +notes.count.value; // 1 +const w = await notes.count.watch(ctx); +w.start(async (value) => console.log(value)); +notes.release(); // ends w; the object lives on (empty params) +await host.close(ctx); +``` + +## 1. Host and facets + +- `createHost({ telemetry?, onError? })`: a host without an address; it serves only `local` types in this slice. + `telemetry` is the adapter (NOOP by default); `onError` receives background errors (a throwing watch listener). +- `defineFacet

(setup)` returns a facet typed for the app's principal type `P` (base code binds it once; nothing is + checked at runtime). `setup(host, self)` runs synchronously inside `host.load` and only declares: `host.serve(Type, + factory)` and facet documents through `self.state(Def)` (C16, owner: built now). `host.serve` outside a setup + throws. +- **Facet documents (design §9, §16):** the facet is a pure object with the host's lifetime; its documents live as long + as the host. Objects the facet serves may read the facet's handles and draft them in their own `self.tx`, together + with their own documents, atomically (lock order: the object's own line, then the facet's line). The facet never + drafts a served object's documents; objects of other facets cannot use them. +- `host.load(facets, context)`: `facets` is `[{ name, facet }]` with npm-style plugin names; duplicate names or a type + served twice are refused. After every setup ran, every served `local` type with empty params is activated (Q-A, the + start barrier); `load` resolves when all are live. If one fails, `load` ends everything it activated and rejects; + the host is then empty. A `host.get` made before `load` resolves waits for it. +- `host.close(context)`: rejects new calls, ends every object (in reverse activation order) and resolves when all have + ended. + +## 2. Objects + +- **Identity:** one object per `(type, params key)` (L1's `checkParams` and `identityKey`; local has no principal). +- **Lifetime (Q-A):** a local type with empty params is created at `load` and lives until `close`; a type with params + is created by the first `get` and ends when its last reference is released. +- **Factory:** `(params, self) => implementation`, synchronous and declarative, run once per object. The + implementation has exactly the contract's members: a state member is a JSON `Watchable` of its value (a document + handle, a derive, a member of a reference), a method is `(args, ctx) => Promise`. L1's `Implementation` + types state members as `Watchable>` (owner, 2026-10-22). +- **Activation (design §6):** run the factory; acquire the references it declared with `self.get` (activating their + objects if needed) and wait until they are live; check the members (exactly the contract's, state members JSON + watchables, Q14); compute every derive's first value; then the object is live and calls proceed. A throw anywhere + in activation (factory, a dependency's activation, a member check, a first derive compute, `initial()` failing its + schema) rejects the `get` (or `load`) with that error, releases everything acquired and leaves no object. Failure + states, backoff and retries are slice 2. +- **Ending:** new calls on the object fail `released`; the context of each of its running calls is aborted; from then + on its `self.tx` throws `released` (a still-running method can no longer change anything); its derives are detached + from their sources (no source keeps a reference to an ended object's derive); its references are released; its + documents are dropped; every watch on its members ends `ended`. Ending does not wait for running calls (the drain + arrives with natives in slice 2, design §3). + +## 3. References + +- `host.get(Type, params, context)` returns a reference once its object is live; `self.get(Type, params)` declares one + in a factory: it is live by the time the object goes live, and released when the object ends. +- A reference is a plain object built from the contract's member list (no proxy): `ref.` is a `Watchable` + (`value`, `watch(ctx)`), `ref.(args, ctx)` calls the method, `ref.release()` releases it (idempotent). After + release, every use throws `released`. +- **Calls:** arguments and results pass by reference; arguments are checked at the provider with L1's `checkArgs` + against the current version (`bindProvider(contract, {})` at serve): unknown method `invalid/member`, bad arguments + `invalid/args`; results are not checked; no-argument methods take `null`. +- **Per-reference watches** (owner): watches obtained through a reference end (`ended`) when it is released, even + while other references keep the object alive. +- **Contract match** (owner): at `get`, the caller's contract must have the provider's version and fingerprint (the + same declaration, or an identical one), otherwise `contract_mismatch`. Older versions through `accepts` come with + slice 7. +- **Reserved member names** (owner): `then`, `release`, `state` and `retry` are refused as member names at + declaration (an L1 check in `remote`, `local` and `evolve`), because a reference has them (`release` now, `state` and + `retry` in slice 2) and `then` would make an awaited reference thenable. +- **Cycles (Q5, one host; owner):** at `self.get` during an activation, Chord follows the chain of objects waiting for + activation; reaching the object being activated fails that activation with `invalid/cycle`, naming the cycle + (`a → b → a`); nothing is kept. +- `ctx` in this slice is a plain `Context` (cancellation and telemetry); `ctx.get` and `ctx.principal` come with + slice 3. + +## 4. Documents in memory (§9, P2) + +- `defineState(name, { schema, version, initial, keyed? })`: the root is a JSON object. `checkpointWhen` is required + only for persisted documents (slice 6; owner, 2026-10-22). `migrate` comes with slices 6 and 7. +- `self.state(Def)` returns a handle bound to this object; every document is in memory, instance state of the object + (P2), gone when it ends. +- **Singleton handle:** `handle.value` is the current value, `initial()` until written or after a retire (Q11); the + handle is a `Watchable`. +- **Keyed handle:** `await handle.get(key)` gives the value or `undefined`; `handle.keys()` is an `AsyncIterable` of + the present keys (the shape persistence keeps); keys are strings. A keyed handle is not a `Watchable` (a live + collection is a singleton holding a list or map). +- `initial()` is checked against the schema once, the first time a definition's `initial()` is used (R8). + +## 5. Transactions (§9, R12, R13) + +- `self.tx(fn, ctx)`: an object's transactions run one at a time (its line). `fn` gets `tx`: `await tx.draft(handle, + key?)` returns the document's draft (Delta's `track`, `beginChange`; the same document gives the same draft within + one transaction); `tx.retire(handle, key?)` removes a document (a no-op when absent). `self.tx` returns what `fn` + returns. +- **Commit:** when `fn` resolves, if `ctx` is aborted the transaction is discarded; otherwise every draft is prepared + (Delta's `prepare`: ops recorded, only placed values checked), unchanged drafts commit nothing, and all changed + documents are adopted (pointer swap) in **one batch** (section 6). A throw discards everything. +- Drafts are revoked when the transaction settles (touching one throws, R12); a transaction returns plain values and + data is read from the handle after the commit. +- A transaction drafts only its own object's documents and its facet's (any other handle throws). Inside `fn`: no + second `self.tx`, no calls to other objects (Durable's rule; documented, not detected). +- **Shorthands (C14, owner: built now):** `handle.change(fn, ctx)` (keyed: `handle.change(key, fn, ctx)`) runs one + transaction that loads the document and calls the synchronous `fn(draft)`; `handle.retire(ctx)` (keyed: + `handle.retire(key, ctx)`) retires it. Both are thin wrappers over `self.tx`. + +## 6. Derives and the batch (W1, §11, §12) + +- `self.derive(sources, compute)`: sources are JSON `Watchable`s (this object's handles and derives, members of + its `self.get` references); only the declared sources are tracked. `compute` is a synchronous pure function of their + values; its result is JSON. +- **Pull model.** A commit only marks: every derive that depends on a changed document, directly or through other + derives, including derives of other objects on the host, is flagged dirty (O(dependents), flags only). A derive's + `value` recomputes when it is dirty, after reading its sources' values (which recompute first if they are dirty); + so each derive computes at most once per batch and always after its sources, diamonds included. +- **The batch** (one per host): a commit's adoptions happen inside it; after the last adoption, still synchronously, + every watched value that changed is read (watched derives recompute through the pull), its ops computed + (`diffRevisions(old, new)` for a derive, the commit's ops for a document), and its watches get one frame each. A + derive is recomputed in the batch only if something watched depends on it; otherwise it stays dirty until read. + Reading `.value` once does not make it watched. +- A derive that throws fails only itself: its `value` throws the error and its watches end `failed`; a derive over it + throws the same error when read. A derive that fails during activation fails the activation (section 2). A failed + derive stays failed until the object is rebuilt. +- **Reentrancy:** inside the batch (adoptions, derive computes, including lazy computes outside a commit), `self.tx`, + `watch`, `release`, `get`, `load` and `close` throw `invalid/reentrant`. One flag per host, cleared in a `finally`. +- `mapShared(f)`: maps an array calling `f` only for object elements not seen before (by identity), returning the + previous result for unchanged ones; primitive elements are mapped every time; the cache is a `WeakMap`. +- `constant(value)`: a JSON `Watchable` that never changes (start data, §18); usable as a state member and a source. +- State is JSON only (L2 decision): no bytes in documents or derives. + +## 7. Watching + +`watchable.watch(ctx)` returns Chord's shared `Watch` (`src/watch.ts`): the value at acquisition, then one frame per +batch that changed it, `(value, ops, ctx)`, in order, asynchronously, with its overflow, `stop` and end semantics. +Each frame is consistent with its commit; two different watches are not ordered relative to each other. A retired +singleton gives the frame `[["r", initial]]` (it changes back to `initial()`; a singleton always has a value). `ctx`'s +abort cancels the watch. A listener that throws ends its watch `listener_error` and is reported to `onError` (taken +from the carry-over). + +## Cost per hop + +| Hop | Work | +| --- | --- | +| a call | L1's three lookups and one compiled check; arguments by reference | +| a commit | Delta prepare per changed document, pointer swaps, dirty flags per dependent, then per watched changed value one read (derives recompute once) and one frame per watch, by reference | +| an unwatched derive | nothing until read | + +Counters (telemetry, where the work happens): L1's, plus `chord.derive.computes` and `chord.derive.diffs`. + +## Carry-over taken by this slice + +From `carry-over.md` (L4 section): watch listener errors reported to `onError`; native watchables refused as state +(by the member check: only JSON watchables exist in this slice); `defineFacet` typed by `P`. Left for +later: plugin error shape, old-version calls (7), `broken()` (2), cuts C5, C6, C7, C13 (2). + +## Tests + +- Host: load runs setups, activates empty-params types, rejects duplicates and double serves; a failing activation + makes `load` end what it activated; `get` during load waits; `serve` outside setup throws; close ends everything. +- Objects: one object per params key; params type created on first get and ended on last release (documents gone, + watches end `ended`, its derives detached from other objects' members); empty-params type lives across releases; + activation failures (factory, dependency, member check, first derive, `initial()`) reject the get and leave nothing; + a method running while its object ends cannot commit (`self.tx` throws `released`) and sees its `ctx` aborted. +- References: hydrated on get; members watchable; calls checked; released reference throws; `self.get` dependency + kept alive and released on end; per-reference watches; contract mismatch; reserved member names refused at + declaration; a two-object and a three-object cycle fail with `invalid/cycle` and leave nothing. +- Documents: singleton initial, retire back to initial; keyed get, keys, retire; `initial()` checked once. +- Transactions: serialized per object; same draft twice; unchanged commits nothing; throw and abort discard; drafts + revoked after settle; another object's handle refused; a served object's transaction over its own and its facet's + documents commits both in one batch; shorthands. +- Batch and derives: two documents in one transaction reach a derive over both in one compute and a watch in one + frame; a derive over another object's member updates in that object's commit's batch; diamond computes once; + unwatched derive stays lazy and a one-off read does not make it eager; a throwing derive fails only itself and its + dependents; reentrancy throws in commits and lazy computes and the flag clears after a throw; `mapShared` identity. +- Watching: frames in order, retire frame, cancellation, listener error reported. +- Cost guards per the table. GC: an ended object with its documents, derives and watches is collected; a released + reference and a stopped watch hold nothing; a detached derive is not held by its former sources. + +## Size + +About 1,100 source lines. diff --git a/packages/chord/docs/next/slice-2.md b/packages/chord/docs/next/slice-2.md new file mode 100644 index 000000000..003532ace --- /dev/null +++ b/packages/chord/docs/next/slice-2.md @@ -0,0 +1,174 @@ +# Slice 2: failures and natives (spec, for the owner's approval) + +Status: **approved (owner, 2026-10-22)**, after Astra's and Fable's spec reviews. The single source for building slice 2, together with the feature map in `slices.md` +(entries assigned to slice 2), `design.md` read in full (most relevant: §3 timeouts, §6 lifecycle and failures, §7 +references, §13 natives, §21 start barrier), the decision log, the carry-over (L4 section) and the root `AGENTS.md` +style section. Builds on slice 1 (`src/next/state`, `documents`, `host`, committed `d8f5e67e6`) and L1. + +## What an app can do after this slice + +```ts +host.serve(Inbox, (_, self) => { + const settings = self.get(Settings, {}); + const watcher = self.open( + async (context) => watchDir(settings.dir.value.path, context.abortSignal), // opened during activation + async (w, reason) => w.close(), // closed in reverse when the object ends + ); + return { count: …, rescan: async (ctx) => watcher.value.rescan(ctx) }; +}); + +host.serve(Remote, (_, self) => { + self.open(async (context) => { + const config = readConfig(); + if (!config) throw broken(new Error("no remote configured")); // retrying is pointless: broken + return connect(config, context.abortSignal); // a throw here is transient: retried with backoff + }, (c) => c.close()); + return { … }; +}); + +const host = createHost({ onError, closeTimeout: 30_000, readyTimeout: 60_000 }); +await host.load(facets, ctx); // resolves once every host-lifetime object is live or failed; failures reported +const inbox = await host.get(Inbox, {}, ctx); // waits through transient retries up to readyTimeout +await host.close(ctx); // closes natives (reason "stop"); rejects with every close error +``` + +## 1. Natives (§13) + +- `self.open(open, close, options?)` in a factory returns a handle (`Native`). Its `value` is readable from the + moment its own `open` resolved until its `close` starts (so a later opener and handlers during the drain can use + it); before that it throws `not_ready`, after that `released`. +- `open(context)` receives a context whose abort signal fires when the attempt is abandoned (the object ends, the + attempt times out). `close(value, reason, context)` with `reason: { kind: "end" } | { kind: "stop" } | { kind: + "reload" }` (`end`: last reference gone or a failed attempt; `stop`: `host.close`; `reload` is used from slice 7); + the close context's signal fires at the native's `closeTimeout`, so a slow close can stop its own work. `options.closeTimeout` + overrides the host default for this native. +- **Activation order:** factory; singleton documents hydrated; references live; natives opened **one after another in + declaration order**; members checked; derives computed; live. An opener may read documents, references and earlier + natives, not derives (they are computed after the opens). +- **Closing:** in reverse order. Each close is awaited up to its `closeTimeout`; one that takes longer is reported to + `onError` as a possible leak and the others continue. A throwing close does not stop the others: `host.close` collects + every close error and rejects with them (`AggregateError`); an object ending in the background reports each to + `onError`. +- **Abandoned opens:** a failed or abandoned attempt closes the natives already open (reason `end`). An `open` that + resolves after its attempt was abandoned has its value closed at once, exactly once (reason `end`); an abandoned + attempt never comes back to life. A later attempt opens every native anew. +- Facets cannot open natives (`FacetSelf` has no `open`). The two restart patterns of R16 (drain in the background, + swap through a registry) are documentation in the host README; there is nothing to build. + +## 2. Failures (§6) + +- Every activation failure becomes a `FailureCause` (L1: `{ kind: "transient" | "broken"; error: ChordError }`, + foreign errors wrapped by `asChordError`). The kind comes from where it failed, never from the error: + + | Where | Kind | + | --- | --- | + | factory throws; a singleton's `initial()` fails its schema; member check (incl. a native watchable as state, Q14); first derive compute; a dependency cycle (Q5) | broken | + | an `open` throws | transient, unless it threw `broken(error)` (L1) | + | the attempt did not become live within `readyTimeout` | transient: `unavailable` "did not become live within …" | + | a dependency failed (R18) | transient: `unavailable` "A: dependency B failed", with B's error as its `cause` | + + A keyed document's first `initial()` runs inside a later transaction; a failure there fails that transaction, not the + object. +- **Every attempt is bounded** by the host's `readyTimeout`: when it passes, the attempt is abandoned (openers' + contexts aborted, open natives closed `end`, a late open closed at once) and fails transient. A hanging `open` can + therefore never block an object or the start barrier. +- **What a failed object keeps:** only an R18 failure keeps the object's references (so the dependency's recovery can + retry it); every other failure releases them at once (a cycle, a factory, an open, a derive). A failed object stays + while anything references it (host-lifetime objects always); it ends with its last reference. +- **Retries.** One attempt per object at a time; a trigger arriving during an attempt is ignored. Transient: retried + with a **backoff per identity**, 1 s doubling to 60 s. The backoff lives in a host map `identity → { delay, timer }` + holding no object (the window is its timer); it is deleted on success, on a broken failure, or when a window has + passed with no object left to retry, and cleared by `host.close`; while its object stays failed the entry keeps the + delay for the next doubling. A get inside the + window waits until `until` before the first attempt; a failed object without references is not retried + (host-lifetime objects always are). Broken: no automatic retry and no entry; a new object after its end tries at + once. In this slice a broken host-lifetime object stays failed until the host is recreated (reload, slice 7, and + `retryOn`, slice 8, add triggers). +- **Dependency recovery (R18):** when a failed dependency goes live, its R18 dependents are retried at once (their + backoff timers cancelled). On every retry the new attempt binds its `self.get` references **before** the previous + attempt's are released, so a dependency held only by this object never ends between attempts. +- **Reporting:** every failed attempt is reported to `onError` (with the object's name and cause); gets waiting for it + are rejected in addition (section 3). +- **The start barrier (§21):** `load` resolves when every host-lifetime object's **first** attempt has an outcome (live + or failed); later retries never hold `load`. A facet setup that throws or a facet document's `initial()` failing its + schema still rejects `load` (code errors). + +## 3. Getting a failing object (§7) + +- `host.get(Type, params, context, { readyTimeout? })` resolves with a live reference. Each getter has its own deadline + (`readyTimeout`, host default, overridable per get) for its whole wait, including a wait for `load`; it is separate + from the attempt's own bound. A broken cause rejects the getter at once with its error; a transient one keeps it + waiting through retries until the object is live or its deadline passes, then rejects with the last cause's error + (or `unavailable` if no attempt has failed yet). A getter whose deadline passes or whose context is cancelled + rejects and releases only its own reference; it never cancels the attempt for other getters. + +## 4. Ending with natives (§3, §6) + +- When an object ends: its transactions are sealed at once (`self.tx` throws `released`); callers of its running calls + are rejected `released` at once and the handlers' contexts aborted (slice 1); Chord keeps each running **handler's** + own promise and waits for those to settle (the drain), bounded by `closeTimeout`; then it closes its natives in + reverse order (a handler still running after `closeTimeout` sees its natives close; that is reported); its documents + stay readable until its closes are done, then are dropped; then the rest as in slice 1. +- A new attempt or a new object for the same identity starts only after the previous instance's ending has finished; + gets arriving meanwhile wait. +- `host.close(context)`: marks every object ended at once (every call on the host fails `released`, every `self.tx` + throws), aborts pending opens (their late values are closed), joins endings already running in the background, then + drains and closes object by object in reverse live order (a dependent before its dependencies), with reason `stop`; + resolves when all are done, or rejects with the collected close errors. + +## 5. Timeouts (§3, cut C13) + +Host options with per-thing overrides: `closeTimeout` (default 30 s; per native), `readyTimeout` (default 60 s; per +get; also bounds each attempt). The call drain uses `closeTimeout` (no separate setting; the design's per-type drain +override is not built). Park deadline (slice 3), routine handover and `finish` retry (slice 5) and rollout install +(slice 7) come with their slices. + +## Cost per hop + +Commits and derives are unchanged. A call adds one set entry for its handler's promise (the drain needs it). +Activation adds one awaited open per native and one attempt timer; a failure adds one backoff entry; ending adds the +drain and the closes. + +## Carry-over taken by this slice + +From `carry-over.md` (L4): `broken()`/`openFailure()` used for opens; a native watchable as state fails `broken` +naming the member; the drain on ending; cuts C5, C6, C7, C13 decided here. + +## Tests + +- Natives: open order and access to earlier handles; `not_ready` before live; close in reverse with reasons `end` and + `stop`; a slow close reported after `closeTimeout` and the others still closed; throwing closes collected by + `host.close` (AggregateError) and reported to `onError` for a background end; a partly failed attempt closes what it + opened; the opener's context aborted when the object ends while opening; facets have no `open`. +- Failures: each row of the table gives the right kind; the cause's error is a ChordError; transient retries follow + the backoff (fake timers) and reset on success; concurrent gets share one attempt; a broken failure is not retried; + a failed object ends with its last reference and its backoff entry expires; a get right after a failure waits for + the scheduled retry. +- R18: a dependent of a failing dependency fails transient with the dependency's cause, keeps the reference and is + retried at once when the dependency goes live; releasing the dependent releases the dependency. +- Start barrier: load resolves with a failed host-lifetime object, reports it to `onError`, and the object goes live + on a later retry; a throwing facet setup still rejects load. +- Gets: broken rejects at once; transient waits and succeeds on retry; transient past `readyTimeout` rejects with the + cause; per-get `readyTimeout`; a cancelled caller releases only its own reference. +- Ending: the drain waits for a running handler (its own promise, not the rejected caller's) before its native + closes, bounded by `closeTimeout`; documents readable during closes; transactions sealed at once. +- Races: an open resolving after its attempt was abandoned is closed once; a get arriving while the previous instance + closes waits and gets a fresh attempt; simultaneous retry triggers give one attempt; two getters with different + deadlines; a dependency held only by a retrying dependent is not ended between attempts; `host.close` during a + pending open, during a background ending, and with several objects in dependency order; a cycle releases its + references. +- GC: a failed object (documents, dependencies, natives, callbacks) is collected after its last reference while its + backoff entry still exists; the entry expires on its own. +- Examples: `06-natives.ts`, `07-failures.ts` (numbering continues from slice 1). + +## Decided with the owner (2026-10-22) + +- `ref.state`, `ref.retry()` and `host.hold` move to slice 3 (no app-held reference can be failed before then). +- `host.get` on a failing object and the start barrier as in sections 2 and 3; C13 as in section 5; C5: no natives in + facets; openers take a `Context`; `close` gets a context aborting at `closeTimeout`. +- C7: `linger` is built in slice 3, with links. The plugin error shape is decided in slice 3, where errors first cross + a link. + +## Size + +About 700 source lines (both reviewers: the attempt states, backoff map, drain and timers). diff --git a/packages/chord/docs/next/slice-3a.md b/packages/chord/docs/next/slice-3a.md new file mode 100644 index 000000000..4605bb95c --- /dev/null +++ b/packages/chord/docs/next/slice-3a.md @@ -0,0 +1,296 @@ +# Slice 3a: remote objects over one link (spec, for the owner's approval) + +Status: **approved (owner, 2026-10-22)**, after Astra's and Fable's spec reviews. Guiding rule (owner): **simple, fast, +correct**; where a choice was open, this spec takes the simplest option that is correct and cheap per hop. + +Slice 3 of the plan is split: **3a** (this) gets two hosts talking over one link; **3b** makes the link survive drops +(re-holds, parking, `ref.state`, `host.hold`, `ref.retry()`, `linger`, `host.root`, `state: false`). Flow control, size +limits, compression, relays, routing by address and IPC/WebSocket stay in slice 4; trace carriers (R5) in slice 9. The +single source for building 3a, together with the feature map in `slices.md`, `design.md` read in full (most relevant: +§3 links and values, §4, §5, §7, §8, §22), the decision log (R1, R4, R6, R20, Q1, Q4, W1, D11), the carry-over (L5, +L6) and the root `AGENTS.md` style section. Builds on L1 and slices 1-2 (`da3a16e9f`). + +## What an app can do after this slice + +```ts +type P = { kind: "owner" } | { kind: "guest"; session: string } | { kind: "server" }; + +const Notes = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(obj({ text: str() })) } }, + reader: { state: { notes: arr(PublicNote), count: int() } }, + }, +}); + +const facet = defineFacet

((host) => { + host.serve(Notes, ({ session }, principal, self) => { + const store = self.get(NoteStore, { session }); // a local shared store (slice 1) + const count = self.derive([store.notes], ([n]) => n.length); + if (principal.kind === "guest") { + return { variant: "reader", members: { notes: self.derive([store.notes], redact), count } }; + } + return { variant: "editor", members: { notes: store.notes, count, add: async (args, ctx) => store.add(args, ctx) } }; + }); +}); + +const server = createHost

({ principal: { kind: "server" }, onError }); +await server.load([{ name: "@app/notes", facet }], ctx); +server.accept(link.a, { principal: { kind: "guest", session: "S1" } }); // the entry point authenticated it + +const client = createHost

({ onError }); // a tab: no principal of its own +await client.load([], ctx); +client.connect(link.b, { principal: { kind: "server" } }); +const notes = await client.get(Notes, { session: "S1" }, ctx); +if (notes.variant === "reader") console.log(notes.notes.value); // narrowed by variant +``` + +## 1. Hosts and links + +- `createHost({ principal?, principalKey?, onError, … })`: a host with a `principal` acts as it for its own requests; + a host without one (a client) has no principal. `principalKey` defaults to L1's canonical JSON. +- **A link** is an ordered, bidirectional channel of `Uint8Array` messages: `send(message)`, `onMessage(listener)`, + `close()`, `onClose(listener)`. This slice ships `memoryLinkPair()`, which delivers each message one task later (as + every link kind will). `send` after close is a no-op. +- `host.accept(link, { principal })`: a link from a child. Every hold and call arriving on it acts as `principal`, + stamped by this host. `host.connect(link, { principal })`: this host's parent link (at most one; a second `connect` + throws). In 3a a host serves over `accept` links and requests over its `connect` link; the `connect` principal is + stored for 3b and slice 4 (nothing arrives for it yet). +- **Handshake:** each side sends `hello { protocol, incarnation }` first and nothing else until it received the + peer's. A message before `hello`, a second `hello`, or another `PROTOCOL_VERSION` (L1) closes the link. `incarnation` + is a uuidv7 minted per host; calls arriving on an `accept` link get the peer's as `ctx.incarnation` (R20). +- **Where a reference goes:** + + | Type | Served on this host | Not served here | + | --- | --- | --- | + | `local` | local (slices 1-2) | fails `unavailable` (local types never cross a link) | + | `remote` | local, as the requester's principal (table below) | over the `connect` link; without one, `unavailable` | + + Any object may reference remote types, local objects included (a client's local view may `self.get` a server type). + Remote objects live while referenced, also with empty params; `load` never activates them (Q-A). +- **Which principal a same-host remote reference uses:** + + | Requester | Principal | + | --- | --- | + | `host.get` | the host's `principal` (a host without one fails `forbidden`) | + | `ctx.get` in a handler | `ctx.principal` (a guest's call gets the guest's view); a local handler: the host's | + | `self.get` in a remote object's factory | that object's principal | + | `self.get` in a local object's factory or a facet | the host's | + +- **Closing:** a link closing (either side) ends what was bound over it: on the consumer, references through it end + (`released`, watches `ended`) and pending calls and gets reject `unavailable`; on the provider, its holds are + released. `host.close` closes its links first, then ends its objects (no `gone` to closed links). Surviving a drop + is slice 3b. + +## 2. Messages (R1, R4, Q1, Q4, §22) + +``` +| u32 header length, big-endian | header JSON (UTF-8) | part 0 | part 1 | … | +``` + +- **One format on every link.** The header is a JSON object, a tagged union over `kind`, defined as a TypeBox schema; + every received header is checked with the compiled schema (the wire is a trust boundary; skipping it on trusted IPC + links comes with IPC in slice 4). The header lists its parts: `parts: [{ json, bytes }]`, the JSON text's byte length and + the byte lengths of the segments that follow it (`bytes: []` for state parts and arguments without bytes). +- **Kinds,** consumer → provider: `hello`, `hold { hold, type, version, fingerprint, params }`, `release { hold }`, + `call { call, hold, method }` (1 part: the arguments), `cancel { call }`. Provider → consumer: `hello`, `bound { + hold, variant, failed? }` (1 part: the snapshot), `state { holds: [{ hold, failed? }] }` (1 part per listed hold), + `failed { hold, cause }`, `result { call }` (1 part), `error { call, error }`, `gone { hold, error }`. +- **Ids** are numbers chosen by the sender, unique per link and direction, never reused while in use (a counter). + Ids a receiver does not know are ignored (Q1: a `result` crossing a `cancel`, a `state` part crossing a `release`). + A `hold` or `call` reusing a live id is a protocol error. +- **One Delta encoder per state member** (owner, 2026-10-22, after the code review): the snapshot part is `{ [member]: + value }` (each member's encoder restarted); a `state` part is `{ [member]: WireOp[] }` with only the members that + changed (a derive's change is its `diffRevisions` ops; a retired singleton is `[["r", initial]]`). +- **A failed member** is listed in the header (`failed: [{ member, error }]`) of the `bound` or `state` that carries the + failure, never in the part; the consumer's replica of that member then fails (reads throw the error, watches end + `failed`), the others keep updating (W1). +- **Argument and result parts** are JSON text followed by raw byte segments: a `Uint8Array` leaf is written as + `{"$bytes": i}`, segment `i`'s length in the part's header entry. Decoding replaces each reference with a `subarray` + view of the received buffer (no copy). Example: `upload({ name: "chart.png", data: <1 MiB>, thumbs: [<2 KiB>, <512 B>] + })` travels as header `…"parts":[{"json":74,"bytes":[1048576,2048,512]}]`, then the JSON + `{"name":"chart.png","data":{"$bytes":0},"thumbs":[{"$bytes":1},{"$bytes":2}]}`, then the three raw segments. Encoding uses the + `JSON.stringify` replacer only for methods whose schema contains a `bytes()` leaf (decided once per method); others + are plain `JSON.stringify`. The key `$bytes` is reserved in those methods only (see below). State parts are plain JSON (state is JSON + only). +- **Protocol errors close the link:** a header failing the schema, a part that is not valid JSON or whose wire ops fail + Delta's checks, an id reused while live, traffic before or a second `hello`. These are structural checks of the wire, + never checks of application state (state is never validated). +- **Errors on the wire** are a tagged union: `{ kind: "chord", error }` (a `ChordError`'s JSON, L1) or `{ kind: + "plugin", code, message, details? }` (open point 3). Never a stack or a cause. + +## 2b. Codecs, the `Message` type and checks (owner, 2026-10-22, after the code review) + +Replaces the JSON-header framing of section 2 (`| u32 header length | header JSON | parts |`); the message kinds, ids, +races and rules of section 2 stay. + +- **`Message`** is a tagged union defined once as a TypeBox schema (`MESSAGE`; `type Message = Static`): `hello { protocol, incarnation }`, `hold { hold, type, version, fingerprint, params }`, `release { hold + }`, `call { call, hold, method, args? }`, `cancel { call }`, `bound { hold, variant, members: [{ member, value }], + failed: [{ member, error }] }`, `state { holds: [{ hold, members: [{ member, ops }], failed: [...] }] }`, `failed { + hold, cause }`, `result { call, value? }`, `error { call, error }`, `gone { hold, error }`. Values (`params`, `value`, + `ops`, arguments) are `unknown()` in the schema. `args` absent: a method without arguments; `value` absent: `null`. + A payload that may carry bytes is marked so by the protocol layer (decided once per method from its schema). +- **Names are indexes:** `member`, `method` and `variant` are indexes into the **hold's bound version's** names, + sorted per variant. Both sides hold that version's description (version and fingerprint matched at `hold`). An index + is valid only while its hold is bound to that version; a provider that stops accepting a version ends that version's + holds with `gone` (contract_mismatch) before its new code serves anything (slice 7 builds the reload side; carry-over). +- **`Codec { name, encode(message): Uint8Array, decode(bytes): Message }`**, chosen per link + (`host.accept(link, { principal, codec? }, context)`, `host.connect(...)`, default `binaryCodec`); both ends use the + same; a mismatch fails the first `hello` (a protocol error naming the expected codec). No negotiation. + - **`binaryCodec`:** a kind byte, then that kind's fields as unsigned varints (ids, indexes, counts, lengths), + length-prefixed UTF-8 strings, the incarnation as 16 raw bytes and the fingerprint as 8 raw bytes; values as varint + length plus JSON text; byte segments (payloads marked as carrying bytes) as a count, lengths, raw bytes, referenced + from the JSON by `{"$bytes": i}`. No outer length (links deliver whole messages). + - **`textCodec`:** one JSON tuple per message, Delta style (a short kind tag, positional fields, no field names): + `["h", protocol, incarnation]`, `["H", hold, type, version, fingerprint, params]`, `["R", hold]`, `["c", call, hold, + method, args?]`, `["x", call]`, `["b", hold, variant, [[member, value]…], [[member, error]…]?]`, `["s", [[hold, + [[member, ops]…], [[member, error]…]?]…]]`, `["f", hold, "t"|"b", error]`, `["r", call, value?]`, `["e", call, + error]`, `["g", hold, error]`; byte segments follow the tuple (their lengths in it). + - `describeMessage(message)` prints any message readably (the text tuple form) for logs and tests. +- **Who checks what** (layers 1-3 failing is a protocol error that closes the link): + + | Layer | Checks | + | --- | --- | + | 1. codec `decode` | lengths and varints fit, the buffer consumed exactly, a known kind, strict UTF-8, each JSON value parses, segment counts and references valid and only in payloads with bytes | + | 2. `MESSAGE` schema (protocol layer, after any codec) | the envelope: kinds, ids and indexes are integers, lists and error shapes; not the values | + | 3. the hold's bound version (protocol layer) | ids not reused while live (unknown ids ignored), indexes within the bound version's lists, order (`hello` first, `bound` before `state`) | + | 4. owners | `params`: `checkParams` (a broken `failed`, link stays); arguments: `checkArgs` (`error { call, invalid/args }`, link stays); ops: Delta's decoder and `applyImmutable` before anything is adopted (protocol error); errors: `chordErrorFromJson` (protocol error); snapshots and results: never validated (R9) | + +- **Memory link:** delivers with `queueMicrotask` (portable, asynchronous, no reentrancy); no Node-specific APIs. + +## 3. Remote objects (§4, §5, §6) + +- `host.serve(remoteContract, (params, principal, self) => ({ variant, members }))`: the factory returns one variant of + the contract with exactly that variant's members. L1's `Implementation` for a remote contract becomes the tagged + union `{ [V in Variants]: { variant: V; members: VariantImplementation } }[Variants]` (an L1 change); a local + contract's stays its members. An unknown variant, or missing or extra members of the chosen variant, fails the + activation broken (the member check, slice 1). +- Identity: `(type, params key, principal key)` (L1 `identityKey`). +- **Calls are checked against the binding's variant:** a call whose method is not in that variant fails + `invalid/member` before anything else (a reader cannot call `add` by forging a message); then L1's `checkArgs` checks + the arguments with the current version. +- **Contract match (R6):** a `hold` binds only when its version and fingerprint equal the provider's current ones + (`accepts` is slice 7); otherwise `failed { hold, cause: broken contract_mismatch }`. +- **A hold's activation** follows slice 2 on the provider: a broken failure is sent as `failed { hold, cause }` and is + final (the provider releases the hold); a transient one is sent as `failed { hold, cause }` with the hold kept + pending while the object retries; going live sends `bound`. The provider has no deadline of its own for a hold: the + consumer's getter deadline decides (it releases the hold when it gives up). A hold arriving before `load` resolved + waits for it; one arriving while the previous instance of its identity tears down waits for that (slice 2). + +## 4. References over a link (§7) + +- `host.get(Remote, params, ctx, { readyTimeout? })` sends `hold` and resolves with a live, hydrated reference on + `bound`. A `failed` with a broken cause rejects at once; a transient one is remembered while the get keeps waiting; + at its deadline (or cancellation) the get sends `release` and rejects with the last cause's error (or `unavailable` + if none arrived). `self.get(Remote)` follows slice 2 (R18): its object waits for `bound`, and a broken or transient + `failed` fails it as a failed dependency. +- `ref.variant` is the variant's name and the discriminant TypeScript narrows on; a local reference has no `variant`. +- **Replicas:** each state member of a remote reference is a watchable on the consumer, built from the snapshot and + then by Delta's `applyImmutable` of its ops (no validation, no copy). One `state` message is applied as **one batch** + of the consumer host, for all its holds, before any watch frame, so no consumer sees half a provider commit (W1). +- `ref.method(args, ctx)` (or `ref.method(ctx)` without arguments) sends `call` and resolves with its `result`, or + rejects with its `error`. A call before `bound` cannot be made (the reference does not exist yet); a forged one is + answered `error { call, not_ready }`. The caller's cancellation sends `cancel` and rejects the caller at once; the + provider aborts that call's context. +- **Release** (open point 1): `ref.release()` rejects the pending calls made through that reference with `released`, + sends `release`, and the provider aborts the handler contexts of that hold's calls and unholds (an object with params + ends with its last reference, local or remote). The same rule applies to local references (a change from slices + 1-2, where a release left its pending calls running). +- `gone { hold, error }` (the provider object ended): the reference ends (`released`, watches `ended`) and its pending + calls reject with the error; no per-call errors are sent. +- Two references to one object from one host are two holds, each with its own encoder (two tabs are two clients). + +## 5. The provider side: one message per link per commit (W1) + +- A hold attaches to its object's state members as a **synchronous subscriber**: `Source` gets a second set of + subscribers besides its watches, fed `(value, ops)` inside the batch's delivery; a hold counts as watching, so exposed + derives recompute in the batch. Taking the snapshot and attaching happen in one synchronous step, so the first + `state` part continues exactly from the snapshot. +- During a batch each touched hold collects its members' ops; at the **end of the batch** (a hook `Batch.commit` runs + after its delivery loop) each touched hold encodes its collected ops **once** into its part, and each link with + touched holds sends **one** `state` message. A link that closed meanwhile drops it. +- A hold whose object ends sends `gone`; a failed member is announced once in the next `state` (or the `bound`). + +## 6. `ctx` (§8) + +`ctx.principal` (the stamped caller; on a local handler the host's), `ctx.incarnation` (the calling host's over a link, +the own host's otherwise, R20), `ctx.get(Type, params)` (a reference released when the **handler** settles; a +`ctx.get` after that throws `released`), the abort signal and telemetry. + +## Races + +1. A `result` crossing a `cancel`; a `call` crossing `gone`; a `state` crossing a `release`; a `bound` crossing a + `release` (the consumer ignores a `bound` for a hold it released): ignored by id (Q1). +2. A commit between binding a hold and sending `bound`: the snapshot and the subscription are taken in one synchronous + step. +3. A forged call before `bound`: `error { call, not_ready }` (parking is slice 3b). +4. The provider object ends with calls pending: `gone`; the consumer rejects those calls itself. +5. The consumer releases with calls pending: rejected locally `released`; the provider aborts the hold's calls. +6. `release` while the provider is still activating for that hold: unhold (may end the object, slice 2); `bound` is + never sent. +7. A `hold` while the previous instance of its identity tears down: waits (slice 2's endings). +8. One object held over two links, one link closes: only that link's holds are released; the other keeps updating. +9. A link closing during a batch: its `state` message is dropped; holds and calls on it end (section 1). +10. A handshake that never completes: nothing else is accepted; a get over it waits up to its deadline. +11. A `ctx.get` finishing after its handler settled: the late reference is released at once. +12. A failed member and healthy members of several holds in one batch: each hold's part carries its healthy ops, its + header the failure. + +## Benchmarks (owner: light benchmarks for memory, timing and wire size) + +A script `packages/chord/test/next/bench/links.ts` (run by hand, not in CI) prints one table **per codec, side by +side** (owner: both codecs, memory and compute); results are recorded in `packages/chord/docs/next/benchmarks.md` for +later slices to compare against. Per case and codec: wire bytes per message; compute (encode and decode time per +message over many messages, and commit-to-frame time end to end); memory (heap allocated per message while encoding and +decoding 10,000 messages, measured with `--expose-gc`, and memory retained per hold). Payloads modelled on p2: + +| Case | Measured | +| --- | --- | +| hold of a notes list of 1,000 items; of a 500 KB document | snapshot encode/decode time, wire bytes | +| one appended item, one changed field, one removed item | commit-to-frame time on the consumer, wire bytes per message, encodes per hold | +| a streamed reply: 2,000 small text appends to one entry | time and bytes per append, total bytes | +| 1 and 10 holds of one object on one link | memory per hold (encoder and replica), messages per commit | +| a call with a small argument; with a 1 MiB `bytes()` argument | round trip, wire bytes (about 1 MiB, not 1.33) | + +## Carry-over taken by this slice + +L6: `PROTOCOL_VERSION` in the handshake; one message per link per commit; bytes as raw segments; a malformed error +envelope closes the link; the plugin error shape. L5: `ctx.get`, `ctx.principal`. Moved to 3b in the feature map: +`host.root`, C8 (`state: false`). + +## Tests + +- Messages: every kind round-trips; parts with byte segments; `$bytes` refused in methods with bytes and plain data elsewhere; each protocol error closes + the link; unknown ids ignored; id reuse refused. +- Handshake: traffic before `hello`, a second `hello`, protocol mismatch; incarnations stamped. +- Remote objects: variant per principal; identity per principal key; a call outside the binding's variant fails + `invalid/member` (forged at the wire, not just in types); arguments checked; contract mismatch; invalid variant or + members fail activation broken; remote objects with empty params are not activated by `load`. +- Principals: the four same-host cases of section 1. +- References: hydrated on get; broken `failed` rejects at once; transient `failed` keeps waiting and the deadline + rejects with the last cause and releases the hold; replicas apply one message as one batch (a consumer derive over two + members never sees half a commit); calls, results, cancel, errors of both kinds; release rejects pending calls (local + and remote); `gone`. +- One message per commit: three holds on one link, one message; a derive with its source; two links, two messages. +- Every race above. +- Cost guards: one encode per hold part; no validation or copy of state on either side. +- GC: a closed link and a released reference, **retained** by the test, hold no holds, encoders, replicas or objects. +- Examples: `08-remote.ts` (two hosts, variants per principal), `09-calls.ts` (calls, results, errors, cancellation, + release). + +## Decided with the owner (2026-10-22) + +Slice 3 is split into 3a and 3b; a remote factory returns `{ variant, members }`; `ref.variant` is the discriminant +(C15: no `inVariant`/`expectVariant`; `variant` reserved as a member name); headers are checked on every link for now. +Release rejects the calls made through the released reference, locally and over links (their handlers' contexts are +aborted). `$bytes` is reserved in the arguments and results of methods whose schema has a `bytes()` leaf (encoding +refuses it with `invalid/args`); elsewhere it is plain data, since only parts with byte segments resolve references +(owner, 2026-10-22, narrowing the first decision). Plugin errors: `new PluginError(code, message, details?)`; any other thrown value crosses +as `{ kind: "plugin", code: "internal", message }`; only code, message and details cross; same-host callers get the +thrown value; the consumer rethrows a `PluginError`. + +## Size + +About 1,500-2,000 source lines (both reviewers), including the L1 change and the batch's subscriber hook. diff --git a/packages/chord/docs/next/slice-3b.md b/packages/chord/docs/next/slice-3b.md new file mode 100644 index 000000000..f4f79d7c9 --- /dev/null +++ b/packages/chord/docs/next/slice-3b.md @@ -0,0 +1,197 @@ +# Slice 3b: surviving drops (spec, for the owner's approval) + +Status: **approved (owner, 2026-10-22)**, after Astra's and Fable's spec reviews. Guiding rule (owner): simple, fast, +correct. The single source for building 3b, together with the feature map in `slices.md` (entries assigned to 3b), +`design.md` read in full (most relevant: §3 links, recovery and timeouts, §4, §6, §7 references, §8, §19, §22), the +decision log, the carry-over (L4, L5, L6) and the root `AGENTS.md` style section. Builds on L1 and slices 1, 2 and 3a +(`76b32ed2d`). + +## What an app can do after this slice + +```ts +const notes = host.hold(Notes, { session: "S1" }, ctx); // a reference at once, not awaited +notes.state.value; // { kind: "loading" } +const w = await notes.state.watch(ctx); // the current state, then every change +w.start(async (state) => render(state)); // loading → live → loading (link dropped) → live … + +await notes.add({ text: "milk" }, ctx); // parks while loading, sent once live +if (notes.state.value.kind === "failed") notes.retry(); // a Retry button + +client.connect(newLink, { principal: { kind: "server" } }, ctx); // after a drop: every reference re-holds + +host.serve(Notes, factory, { linger: 5 * 60_000 }); // a just-released object stays warm for 5 minutes +``` + +## 1. Reference state (§7) + +- Every reference has `ref.state`, a `NativeWatchable` (in-process: it holds errors; open point 1): + + ```ts + type RefState = + | { kind: "loading" } + | { kind: "live" } + | { kind: "failed"; cause: FailureCause; retryAt: number | undefined } // undefined: broken, no automatic retry + | { kind: "released" }; + ``` + + `ref.state.watch(ctx)` delivers the current state, then every change. On release the state becomes `released` and + the watch ends `ended` (the end is the notification; a native watch drops undelivered frames when it ends). +- **One rule for both sides:** a reference is `loading` while its object is activating or retrying, while it waits for + `load` or a parent link, and while its link is down; `failed` while its object is failed and no attempt runs; `live` + when bound; `released` once released. `variant` is `undefined` before the first binding and keeps the last bound + variant while `loading` or `failed`. + + | State | Calls | State members | + | --- | --- | --- | + | `loading` | park (section 3) | readable: the last value; before the first binding a read throws `not_ready` and a `watch` waits for the first value | + | `live` | run | readable | + | `failed` | reject at once with the cause's error | reads throw the cause's error; existing watches stay open | + | `released` | throw `released` | reads throw `released`; watches end `ended` | + +- Member watches survive `loading` and `failed` (reference states): they get the next frame when the reference is live + again. They end only when the reference is released, the member itself fails (W1, `failed`), or the member is not in a + newly bound variant (`ended`). +- **Retry** (open point 2): `ref.retry()` asks the object for an attempt now, skipping the backoff window and renewing a + broken object's readiness (carry-over L5); one attempt runs at a time, so a retry during one is ignored. Local: the + object's `attempt()`. Remote: a new message `retry { hold }`; the provider does the same for the held object, for all + its holders. On `live` and `released` references `retry()` does nothing; without an open parent link the next + `connect` re-holds anyway. +- **Attempt notice** (open point 2): when a failed object starts a new attempt (backoff, retry, or a dependency going + live), its holds get `loading { hold }`, so remote references move `failed → loading` and park calls exactly like local + ones. `failed { hold, cause, retryIn? }` carries the remaining delay of the provider's window (`until - now`, from a + new `until` on slice 2's backoff entry, carry-over L4) as advice for `retryAt`; the attempt itself is announced by + `loading`. + +## 2. Acquiring: `hold` and `get` (§7) + +| Situation | `host.hold(Type, params, ctx, options?)` | `host.get(…)` | +| --- | --- | --- | +| invalid params, a local type not served here, no principal for a same-host remote type | throws at once (made before `load`: `failed` broken once `load` resolved, since the types are known only then) | rejects at once | +| before `load` resolved, or no parent link yet (remote) | `loading` | waits | +| the object activates | `loading` | waits | +| a transient failure | `failed` (with `retryAt`), then `loading` on the next attempt | waits; at its deadline it releases the reference and rejects with the last cause | +| a broken failure | `failed` | releases the reference and rejects at once with the cause | +| the link drops before the first binding | `loading`, re-held on the next `connect` | keeps waiting within its one deadline | +| live | `live` | resolves | +| the caller's context is aborted | (no effect on a `hold`) | releases the reference and rejects | + +A `hold` follows its object until released and never rejects. A `get` has one deadline (`readyTimeout`, per get) from +the moment it is called, across reconnects. + +## 3. Parking (§3, §7) + +- A call made on a `loading` reference is queued with its own deadline, measured from the moment it was called (a + reconnect never renews it). When the reference goes `live`, queued calls are **sent** in the order they were made, + before any later call (a new call cannot overtake queued ones); their completions may arrive in any order. When the + reference goes `failed` they reject with the cause; at its deadline a call rejects `unavailable/park_timeout`; the + caller's cancellation rejects it at once; release rejects them `released`. +- A queued call is resolved against the binding it is sent on: after a reconnect that bound another variant, a method + the new variant lacks rejects `invalid/member`; indexes are taken from the new binding. +- Calls already **sent** when the link drops fail `unavailable` (never resent, §7). +- **The park deadline is `readyTimeout`** (owner, 2026-10-22; C13's two host timeouts stay): a queued call waits at most + the reference's `readyTimeout` (host default 60 s, per `get`/`hold` override), then rejects `unavailable/park_timeout`. + +## 4. Link drops and re-holds (§3 recovery, §4, §22) + +- **One owner of remote objects across links:** the host keeps its remote references (their objects) independent of any + one link; each `connect` creates a link with fresh id counters (ids are link-local) and re-holds every non-released + remote reference on it; a closed link's tables are dropped, and nothing of an old link (a late callback) touches the + new one. +- **On a drop:** every non-released remote reference goes `loading` (also a `failed` one), sent calls fail + `unavailable`, new calls queue. `host.connect(link, …)` may be called again once the previous parent link has closed + (open point 6); while one is open it throws. +- **The re-hold:** a new `hold` (new id); on its `bound`, in **one batch** of this host and before any queued call is + sent: each member of the bound variant gets the snapshot value with one frame `[["r", value]]` and a fresh Delta + decoder (open point 4); members not in the bound variant end (`ended`, derives over them fail); new members get their + first value; failed members listed in `bound` fail (W1); `variant` is updated. A different variant is the normal result + of a changed grant (design §4: reconnect, authenticate anew, re-hold). Then queued calls are sent. +- **The provider side** is unchanged: a closed link releases its holds (objects with params end with their last + reference unless they linger, section 5); a reconnecting child is a new `accept` with new holds. +- **References a host's objects hold** (`self.get` of a remote type) go `loading` the same way; a **live** dependent + stays live (R18: a dependency leaving `live` does not change it) and its calls through them queue. An **activating** + dependent keeps waiting for the dependency; its own `readyTimeout` bounds it (slice 2), and R18 retries it on `bound`. +- Who reconnects is the app (its link adapter calls `connect` again); Chord only re-holds. + +## 5. `linger` (§3, §6, §19, D8, C7) + +`host.serve(Type, factory, { linger })` (milliseconds, default 0): a **live** object with params whose last reference +is released stays live for `linger`, keeping its own dependencies; a `get`, `hold` or arriving `hold` message within +that time reuses it; otherwise it ends. An object that is waiting, activating or failed when its last reference goes +ends at once (slice 2). A failed state member is not a failed object and does not end a lingering one. `host.close` +ends lingering objects at once. Host-lifetime objects (`local`, empty params) never end by references; a remote type +with empty params lingers like any other. + +## Races (listed before sizing) + +1. A sent call when the link drops: fails `unavailable`, never resent. +2. A call made while `loading`: queued with its deadline; sent once live, in order; `park_timeout`; cancellation; + release. +3. A reference released while `loading` (no link): nothing sent; not re-held; queued calls reject `released`. +4. `connect` while the previous parent link has not finished closing: throws. +5. A drop during a re-hold's handshake or before its `bound`: stays `loading`; the next `connect` re-holds again. +6. The re-hold finds the object failed: `failed` with `retryIn`; the provider's next attempt sends `loading`, then + `bound` when live. +7. A reconnect creates a fresh object (the old one ended without linger): new snapshot; the old in-memory documents are + gone (documented). +8. A get or an arriving `hold` within `linger` versus the linger timer: a reuse cancels the timer; the holder count + decides, so an expiry never ends an object that has a holder again; no grace period. +9. A failing object without references ends at once, never lingers; the last release during activation ends it. +10. `host.close` with queued calls and loading references: queued calls reject `released`, references go `released`. +11. `ref.retry()` during an attempt or a pending re-hold: ignored. +12. Shared remote retry: two consumers hold one failed object; one retries; both get `loading`, then `bound` or + `failed`. +13. A broken object held by another consumer: a `retry` renews it for everyone. +14. A reconnect binds another variant (a changed grant): replicas rebuilt in one batch; vanished members' watches end; + queued calls checked against the new variant. +15. A release or a call's deadline while a snapshot is being adopted or queued calls are being sent: the reference's + state decides at each step; nothing is sent for a released reference. +16. A member `watch` before the first binding: waits; rejects on `failed` (cause) or release. +17. A re-hold arrives while the previous instance (released by the drop, no linger) still tears down: waits for that + ending (slice 2), then a fresh object. +18. An activating dependent whose remote dependency's link drops: it fails transient at the drop (the dependency's + pending outcome rejects `unavailable`); R18 retries it when the dependency is bound again. +19. A drop while `failed`: `loading`, re-held on the next `connect`. + +## Cost per hop + +Nothing changes for live references (calls, commits, frames as 3a). A drop costs one `hold` and one snapshot per +reference on reconnect; a queued call holds its arguments until sent; `ref.state` is created lazily on first access. + +## Benchmarks + +Extend `test/next/bench/links.ts` (both codecs): reconnect with 1 and 100 references (time to all live, bytes), 1,000 +queued calls flushed on reconnect (time, order), and the live paths of 3a unchanged (regression rows). + +## Carry-over taken by this slice + +L4: `retryAt` (backoff `until`). L5: `ref.retry()` renewing a broken object's readiness; calls on a failed reference +reject with its cause; C8. L6: `linger`, `ref.state`, `ref.retry()`, `host.hold`. Moved out: a second `bound` as a reset +(slice 4 for `reset` after overflow, slice 7 for restarted objects); the park deadline's per-type override is not +built. + +## Tests + +- `ref.state`: every transition for local and remote references, watched (first value, each change, the end on + release); `variant` before, during and after bindings. +- `hold` and `get`: every row of the table in section 2. +- Parking: queue order, one deadline across reconnects, cancellation, release, failure, a variant change. +- Drops and re-holds: references go loading; sent calls fail; reconnect re-holds with fresh ids; one batch per bound + with `[["r", value]]` frames; another variant on reconnect; old-link callbacks never touch the new link. +- Retry: local and remote; shared across consumers; broken renewed; ignored during an attempt; `loading` notices. +- `linger`: reuse within the window, end after it, only live objects linger, failed members do not end it, + `host.close`, reconnect within linger keeps in-memory documents. +- Every race above, both codecs where a link is involved. +- GC: a released `hold` reference, a reference across a drop, a closed link's tables, a lingering object after expiry. +- Examples: `10-drops.ts` (drop, park, reconnect, re-hold, `ref.state`), `11-linger-and-retry.ts`. + +## Decided with the owner (2026-10-22) + +`ref.state` is a `NativeWatchable` (the `NativeWatchable` interface joins `Watchable` in `src/watch.ts`, +upstream first); remote retries use two new messages, `retry { hold }` and `loading { hold }`; parked calls wait at most +`readyTimeout` (no third host timeout); a re-hold replaces each member with `[["r", value]]`; C8 (`state: false`) is +cut; `connect` may be called again once the previous parent link has closed; `host.root` moves to slice 8. + +## Size + +About 1,100-1,400 source lines (reference state, acquisition, parking with its queue, re-holds with variant changes, +retry and loading notices, `linger`). diff --git a/packages/chord/docs/next/slice-4.md b/packages/chord/docs/next/slice-4.md new file mode 100644 index 000000000..f82525ff2 --- /dev/null +++ b/packages/chord/docs/next/slice-4.md @@ -0,0 +1,146 @@ +# Slice 4: the topology (draft: scope, structural changes, races) + +Status: **overview** (owner, 2026-10-23: slice 4 is split into 4a and 4b). The 4a spec proper is `slice-4a.md`, which +wins where the two differ; 4b is sketched at the end of this file. Reads with `design.md` (§3 hosts, links, routes, flow control; §4 principals and +revocation; §6 cycles; §7; §19 presence; §22 wire), the decision log (R1, R2, Q1-Q5, Q13, R20, "Flow control: an +ack-clocked window", "One pending queue for slow consumers of state", "Compression at the origin", "Wire protocol"), +the carry-over (L6, L7) and slices 3a/3b. Builds on `9066b343c`. + +## The split + +| Slice | Contents | +| --- | --- | +| **4a: links and relays** | IPC and WebSocket links; trusted links; parts as opaque bytes; flow control (acks, pending queue, call window); host kinds, addresses, `placement`; `route` (`authorize`, `resolve`, re-resolve with backoff); relayed holds and calls (ids mapped, principal and origin incarnation stamped, per-hop acks, `reset` and a second `bound`) | +| **4b: authority and housekeeping** | revocation (`revokedBy`, `host.revoke`, propagation to children); cycle checks across hosts (activation chain in hold headers); `host.references` (origin ids); `host.activity`; size limits per part (the admitting link's decoded limit stamped on relayed work); compression at the origin | + +4a is what the playground needs (server, workers over IPC, tabs over WebSocket). 4b adds header fields on top of 4a's. + +## 4a: structural changes the current code needs + +1. **A link carries both directions.** Slice 3 has `ProviderLink` (`accept`: serve only) and `ConsumerLink` (`connect`: + hold only). With routes, the server holds worker objects over a resolved link *and* serves the worker's holds of + server types over the same link; the worker serves the server's holds over its `connect` link *and* holds server + types over it. One per-link object with both sides (two id spaces: holds this side sent, holds it received; message + kinds are directional, so dispatch stays by kind). +2. **Parts are opaque bytes.** Today the codecs parse every value (`params`, `args`, `value`, `ops`, snapshot members) + at decode. A relay must forward them unparsed (R1), so values become byte parts that only the end that owns them + decodes (the consumer's replica, the provider's `checkArgs`). Exception: a `hold`'s params, which a relay parses to + compute placement and authorize (small; checked by `checkParams` anyway). Sizes then come for free (`byteLength`). +3. **Stamped header fields.** `hold` and `call` gain the origin's principal and incarnation (Q13), set by the relay + from the link the work came in on; honored only on a link from a parent; overwritten on every other link. 4b adds + the activation chain, the origin id and the decoded limit the same way. +4. **`ack`** (consumer to provider, after applying each `bound`/`state` message) and the call window in `hello` + (`calls: N`). +5. **Resets as `r` ops** (decided below): a `state` entry flagged `reset: true` carries `[["r", value]]` per member; + the consumer applies it like any ops (the decoder restarts at `r`). A second `bound` on a live hold stays refused + until slice 7. +6. **`loading` on a live hold**: a relay whose upstream link dropped tells its downstream holds; 3b's consumer moves + only `failed -> loading` on it today. +7. **The pending queue** (fold, leaf index, cap 10,000) per hold member on the provider side, and in Chord's shared + watch handle (pi first; carry-over L6). + +## 4a: races + +Routes and resolve: + +1. Two holds (or a hold and a call's re-hold) for one address while its `resolve` runs: one `resolve` per address, + shared; both holds go out on the link it returns. +2. `resolve` rejects or its signal aborts: the holds stay `loading`; re-resolve with the per-address backoff (1 s + doubling to 60 s) while holds for the address exist. +3. Every hold for an address is released while `resolve` runs: the resolve finishes (a worker start that produced a + link finishes even if its first requester left, §3); the link is kept for the next hold. Who closes an idle worker + link is the app's supervisor (`host.activity`, 4b). +4. `authorize` refuses: the hold fails broken `forbidden` before anything starts; a later hold with another principal + for the same address still resolves. +5. Placement: the caller computes `placement(params)` and routes by it; the provider computes it again and refuses an + object that is not its own (`invalid/placement`, broken). A placement rule changed by a reload between the two is + that refusal (rollout handling is slice 7). +6. A worker restarts (crash, the supervisor stopped it): its link drops; the new link's `hello` carries a new + incarnation; re-holds get fresh objects (in-memory documents gone, as 3b race 7). + +Relays (server between a tab and a worker): + +7. The worker link drops with relayed holds: the server sends `loading { hold }` downstream (tabs go `loading`, + values readable, new calls park on the tab), fails relayed calls in flight with `unavailable` (never resent), + re-resolves, re-holds upstream, and forwards the new `bound` downstream (accepted while `loading`, as 3b's + re-hold; another variant is allowed). Superseded by `slice-4a.md` section 4. +8. A tab call that crossed the `loading` notice reaches the server while the upstream link is down: answered + `unavailable` (as a call sent before a drop, §7), not parked at the relay. +9. The tab releases a hold while its upstream re-hold or `resolve` is pending: the relay releases upstream once the + upstream hold exists; nothing is sent downstream after the release (Q1 covers a `bound` crossing it). +10. The tab link closes: the relay releases every upstream hold it made for that tab and cancels its relayed calls. +11. Two tabs of one principal hold the same object: two upstream holds, each with its own encoder (R1, as today). +12. A worker sends `gone` for a relayed hold: forwarded downstream; the relay forgets the mapping. +13. A worker holds a server type (or another worker's): it goes over its `connect` link; the server serves it or + routes it on (worker to worker through the server, §3). + +Flow control: + +14. An `ack` crossing a `reset`, a `release` or a link close: acks count messages per link, not holds; an ack for a + message whose hold was released still frees its slot. +15. A `reset { hold }` reaching the provider while the hold is pending, failed or released: ignored unless bound + (Q1). +16. A relay with a full window towards a tab: it drops new parts for that tab's holds and marks them; when the tab + acks, it sends `reset { hold }` upstream for each marked hold and drops further parts for a hold until its new + `bound` arrives. +17. **Torn resets:** one commit touched holds A and B of a tab; both were dropped and reset; the two new `bound`s + arrive as two messages, so a derive over A and B could see A new and B old in between. Proposal: `reset` carries + a list of holds and the provider answers with one message holding all their snapshots. +18. The pending queue reaches its cap while the window is full: only "replace with the current value" stays; when a + slot frees, the provider sends a snapshot of the member (a second `bound` part or an `r` op; see open point 3). +19. Calls: more than N calls from one link are parked by the caller (one FIFO per link, across references); a + cancel frees the slot at the caller when sent and at the provider when received (per-direction order keeps the + counts equal). +20. **Call fairness at the relay:** each tab has N calls towards the server, but the server has one window of N + towards a worker for all tabs and its own calls. Proposal: the relay parks relayed calls per origin and sends + round-robin, so one flooding tab cannot occupy the worker's window; the relay holds at most tabs x N parked calls. + +Links and transports: + +21. IPC: `send` while the child is exiting, `disconnect` racing a message in flight: the link closes once; messages + after close are dropped; `process.send` errors close the link. +22. WebSocket: a frame above the transport's `maxPayload` closes the connection (code 1009); a client that stops + reading is bounded by the window (2 messages) plus the pending queue cap, not by the socket buffer. +23. Trusted links skip the message schema check (IPC, carry-over L6) but never grant stamping: stamped fields are + honored only from the link to the parent. + +## Decided (owner, 2026-10-23) + +1. **Trusted links are an explicit option:** `accept(link, { principal, trusted: true })`, and `resolve` may return + `{ link, principal, trusted }`; default untrusted. Trusted skips the message schema check; stamping is still + honored only from the parent link. +2. **Resets and overflow are `r` ops in an ordinary `state` message** (races 16-18): a relay's `reset { holds }` + (a list) is answered with one `state` message whose entries carry `[["r", value]]` for every member of each reset + hold that is still bound, all snapshots taken in one synchronous step, applied by the consumer in one batch (no + torn commit across the reset holds). An overflowed pending queue sends `[["r", value]]` the same way. Delta's + encoder and decoder restart at an `r`. A state entry gets the header flag `reset: true`, so a relay (which never + reads parts) knows when to stop dropping a reset hold's parts. A second `bound` stays only for slice 7 (a restarted + object may choose another variant). +3. **Call fairness: round-robin per origin at the relay** (race 20): each origin's calls keep their order; while the + worker's window is full, origins take turns, so one flooding tab cannot starve the others. +4. **The Durable conversation view:** known cost until Durable's windowed view (pi #11327, waiting for its reporter) + lands: a re-hold sends the whole active transcript. + +## 4b: scope and races (sketch) + +- **Revocation:** `createHost({ revokedBy })`; `host.revoke(r)` ends every object and closes every link whose + principal has `r` in `revokedBy` (L1's `revocationKeys`), then propagates `r` to children (a `revoke` message on + trusted child links). Races: a hold stamped with a revoked principal already on its way to a worker (it binds, then + its link closes: R10's window); revoke during an activation; revoke while a relayed call runs (its handler is + aborted; R11 jobs later). +- **Cycles across hosts (Q5):** a hold made during an activation carries the chain of activating instance ids + (bounded); the provider refuses a hold whose chain contains the instance being activated (`invalid/cycle`); chains + from untrusted links are ignored. Races: the chain crossing a relay (copied, never extended by the relay); a cycle + through a host that restarts. +- **`host.references(Type)`** (presence, §19): who holds objects of a type, with an origin id per client (stamped by + the relay, so two tabs are two clients); a link drop is a departure. +- **`host.activity`**: `{ references, calls, jobs, lingering }` as one readable for the app's keep-alive. +- **Size limits per part (R2, Q2)**: checked from the header before decoding; an oversized part fails only its hold or + call (`too_large`); the admitting link's decoded limit is stamped on relayed work and enforced where it is decoded. +- **Compression at the origin**: a part above a threshold is compressed once by its origin (codec field in the + header); relays forward it compressed; only the end that decodes inflates it, with the decoded limit. + +## Size + +Reviewers' estimate for slice 4 as a whole: 2,500-3,000 source lines; 4a carries most of it (the two-sided link, the +relay, transports, flow control). diff --git a/packages/chord/docs/next/slice-4a.md b/packages/chord/docs/next/slice-4a.md new file mode 100644 index 000000000..8bcf15562 --- /dev/null +++ b/packages/chord/docs/next/slice-4a.md @@ -0,0 +1,418 @@ +# Slice 4a: links and relays (spec, for the owner's approval) + +Status: **approved (owner, 2026-10-24)**, revised after Astra's and Fable's spec reviews (`/tmp/s4a-review-*.md`); +sections 4-6 and their races revised for the decision "Relay flow control per origin" (owner, 2026-10-24), which +replaces per-hop acks, `reset { holds }`, relay recovery and id renumbering. +Guiding rule (owner): simple, fast, correct. The single source for building 4a, together with the feature map in +`slices.md` (entries assigned to 4a), `design.md` read in full (most relevant: §3 hosts, links, routes, flow control, +recovery; §4; §7; §18; §22), the decision log (R1, Q1, Q4 and "Wire protocol", Q13, R20, "Flow control: an +ack-clocked window", "One pending queue for slow consumers of state", "Slice 4 split and first decisions", "Slice 4a +spec decisions", "Slice 4a review decisions", "Relay flow control per origin"), the carry-over (L6, L7, 4b), `slice-4.md` (the split and 4b) and the +root `AGENTS.md` style section. Builds on L1 and slices 1-3b (`9066b343c`). + +## What an app can do after this slice + +```ts +export const Server = hostKind("server"); +export const Session = hostKind("session", { id: str() }); // params like a contract's + +export const Notes = remote("notes", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), // never part of the fingerprint + variants: { … }, +}); + +// server.ts +const server = createHost

({ address: Server(), principal: { kind: "server" }, onError }); +server.route(Session, { + authorize: (principal, address) => principal.kind === "owner" || (principal.kind === "guest" && principal.session === address.params.id), + resolve: async (address, context) => { + const child = fork("worker.ts", [], { serialization: "advanced" }); // no `signal`: see below + child.send({ id: address.params.id, init: initFor(address.params.id) }); // the app's own start message + return { link: ipcLink(child), principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, +}); +await server.load(serverFacets, ctx); +wss.on("connection", (ws, req) => server.accept(webSocketLink(ws), { principal: authenticate(req) }, ctx)); + +// worker.ts: connect first (W6), then load; start data through the core facet (§18) +const { id, init } = await startMessage(process); +const worker = createHost

({ address: Session({ id }), principal: { kind: "worker", session: id }, onError }); +worker.connect(ipcLink(process), { principal: { kind: "server" }, trusted: true }, ctx); +await worker.load([{ name: "@app/core", facet: coreWorkerFacet(init) }, ...workerFacets], ctx); // serves SessionInfo + +// a tab +const tab = createHost

({ onError }); +tab.connect(webSocketLink(new WebSocket("/chord")), { principal: { kind: "server" } }, ctx); +const notes = await tab.get(Notes, { session: "S1" }, ctx); // tab → server → route(Session) → worker +``` + +Do not pass `context.abortSignal` to `fork`: `host.close` aborts it, which kills the worker and makes Node emit an +`error` event on the child process that crashes an app without a handler. Chord closes the link at `host.close`; the +worker exits when its parent link closes. `resolve` returns the link before the worker has loaded (the worker connects first, W6): a `resolve` that waited for +the worker's `load` while the worker's facets hold server types would deadlock. + +## 1. Host kinds, addresses, placement (§3) + +- `hostKind(name, params?)` declares a kind of addressable host; `params` follow L1's params policy (a flat record of + required `str()`, `int()`, `bool()`, `oneOf(...)`), checked with L1's compiled params check. `Session({ id: "S1" })` is + an **address**: `{ kind, params, key }`, its key built like an identity key (`session|S1`). Two addresses are equal + when their keys are. No host-name strings in app code. +- `createHost({ address })`: an addressable host. Without an address a host is a client (§3 roles). A host without a + principal serves only local types: `serve` of a remote type makes `load` throw (owner, 2026-10-24; a remote object + is owned by the serving host's principal). +- `remote(name, { …, placement })` (an L1 change): `placement(params) => Address`, a typed function; not part of the + fingerprint (L1 normal form unchanged). `local` contracts have no placement. +- **The caller computes the address** and sends it in the `hold` (section 5). A host that relays never needs the + contract: it routes by the address alone. +- **Where a reference or an arriving hold goes** (first match): + + | Case | Goes | + | --- | --- | + | the type is served here and its placement (computed here from the params) is this host's address, or it has none | here | + | a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) to a served type whose address is another host's | on, as if not served here (next rows): worker S1 reaches `Notes { session: "S2" }` through the parent | + | an arriving hold for a served type whose params place it on another host | broken `invalid/placement` (misrouted; never forwarded); the address the hold names only routes it | + | the hold names an address of a kind this host has a route for | over that route's link (the address's params checked against the kind; `authorize` first) | + | otherwise (a remote type) | over the parent link; a hold that arrived *on* the parent link is never sent back to it: `unavailable` | + | no parent link | `unavailable` (as 3a) | + +- `placement` or `authorize` throwing fails the hold broken with that error; an address whose params fail the kind's + check is broken `invalid/placement`. + +## 2. Routes (§3) + +```ts +host.route(Session, { + authorize(principal: P, address: Address): boolean; + resolve(address: Address, context: Context): Promise<{ link: Link; principal: P; trusted?: boolean }>; +}); +``` + +- `route` is called by base code before `load`; one route per kind (a second throws). +- **`authorize`** runs for every hold routed over the route, with the hold's principal (section 4), before anything + starts. `false` fails the hold broken `forbidden`. +- **`resolve`** runs at most once at a time per address: the first hold starts it, later holds wait for the same + promise. The returned link becomes the address's link (section 3, role `route`). `context` is aborted only when the + host closes. A resolve whose holds were all released meanwhile finishes and its link is kept (§3). +- **Failure and re-resolve:** a rejected `resolve`, a returned link that is already closed, or the address's link + closing while holds over it exist, starts the per-address backoff (1 s doubling to 60 s; a backoff map keyed by + address, separate from objects' identity backoff). While waiting, the holds are `loading` (3b). Only a link that + completed its handshake resets the backoff. Without holds for the address, nothing is resolved again. +- A `resolve` that never settles is the app's (it bounds its own `fork`); getters still time out by their deadlines. +- Chord never closes a route link itself except at `host.close`; stopping idle workers is the app's (`host.activity`, + 4b). + +## 3. Links carry both directions (§3, §22) + +- **One link object per link** replaces 3a's `ProviderLink`/`ConsumerLink` pair: one channel (handshake, codec, + checks, heartbeat), a **provider side** (holds and calls it received; ids chosen by the peer), a **consumer side** + (holds and calls this host's own references sent; ids from its own counters). The provider side is one per + **origin** (section 4): the peer's own work, and each downstream link of a peer that relays. Message kinds are + directional, so each goes to its side. Locally served holds and relayed holds are two kinds of entries in a provider + side's table; a relayed hold is never hydrated into a `RemoteObject`. +- Every link has the peer's principal, `trusted`, and a role: + + | Role | Made by | A hold arriving on it acts as | This host holds over it | + | --- | --- | --- | --- | + | parent | `connect` | the hold's `principal`/`incarnation` fields when present, else the link's principal and peer incarnation | types routed to the parent (a worker's holds of server types, everything a client gets) | + | child | `accept` | the link's principal and peer incarnation (fields ignored) | nothing | + | route | `resolve` | the link's principal and peer incarnation (fields ignored) | the address's objects | + +- **Trusted links** (`trusted: true` on `accept`, `connect` and `resolve`'s result; default untrusted) skip only the + `MESSAGE` schema check (carry-over L6): framing, `checkParams`, `checkArgs` and Delta decoding still run. Trust never + grants the principal fields: they are read only on the parent link. +- `host.close` closes every link (children, routes, parent), then ends objects (3a order). +- **Remote objects** (3b's `RemoteObject`) belong to a **path**: the parent, or a route address. Each new link for a + path holds every remote object of that path again (3b's re-hold); a closed link drops its tables and the objects go + `loading`. + +## 4. Principals and relays (§4, §22, R1, Q13, R20) + +**Principals on holds.** A `hold` carries `principal?` and `incarnation?`. The receiver uses them only on its parent +link; everywhere else they are ignored and the link's principal and peer incarnation apply (a tab cannot claim to be +the owner). Calls carry no principal: a call runs as its hold's principal and with its hold's incarnation +(`ctx.principal`, `ctx.incarnation`), so stamping calls separately is not needed (refines R1/Q13 "hold and call"). + +**What a host sends** in those fields when it holds over a route link (towards a child that honours them): + +| The reference was made by | `principal` | `incarnation` | +| --- | --- | --- | +| `host.get` / `host.hold` | the host's own | this host's | +| `ctx.get` in a handler | the handler's `ctx.principal` (the caller's) | the handler's `ctx.incarnation` | +| `self.get` in a remote object | that object's principal | this host's | +| `self.get` in a local object or a facet | the host's own | this host's | +| a relayed hold (below) | the arriving hold's principal | the arriving hold's incarnation | + +So a guest's call on the server whose handler `ctx.get`s a worker type reaches the worker as the guest, and gets the +guest's variant. `RemoteObject` keeps its binding's caller (3b drops it today) and two references with different +principals are two remote objects. + +**Upward is a boundary.** A worker's hold of a server type goes over the worker's parent link; the server ignores +principal fields from a child, so it acts as the worker, whoever the worker's handler serves. A worker that acts for a +user towards the server passes the user as data and the server object decides. Documented. + +**Relays.** A host that receives a hold it cannot serve (section 1's table: a route or the parent) relays it: it holds +it again on that link (the **upstream** link) with the arriving hold's principal and incarnation, and forwards between +the two without parsing payloads (decision "Relay flow control per origin"). + +- **Origins.** The relay gives each downstream origin (a downstream link, and the origin its messages carry when that + link is itself a relay's) a number, one counter per host, never reused; 0 is a host's own work. Every message + upstream for that origin carries `origin` (section 5); the provider keys holds, calls, windows and acks by (link, + origin, id), and its answers carry the origin back. `origin` is honoured on every link (a worker may relay a + child's hold to its parent); principals and incarnations stay parent-only. +- **Ids are not renumbered:** an upstream hold has the downstream hold's id, under the origin's number; calls the + same. Two tabs (two holds) of one principal are two upstream holds with their own encoders (R1). +- **Nothing is parsed:** the relay routes by the hold's address and authorizes it; params, arguments, results, + snapshots and ops travel as opaque parts (section 5). +- **What the relay keeps:** per relayed hold, its upstream link and its original `hold` message (params copied once), + which routes `call`, `cancel`, `retry` and `release` upstream at once and holds again on the next link; per + downstream origin, the FIFO of the sender of each `bound`/`state` message in flight (section 6), and the set of + upstream links it relayed to. +- **Forwarding up:** `hold`, `call`, `cancel`, `retry`, `release` with the origin's number; a `hold` waiting for its + route is sent when the link exists; a `release` before that drops it. A call is forwarded at once; with no + upstream link it is answered `unavailable`. +- **Forwarding down:** a message from upstream with an origin goes to that origin's provider side, which forwards it + with its own origin (none for a tab), ids and parts unchanged: `bound`, `state`, `failed`, `loading`, `result`, + `error`, `gone`. One upstream message concerns one origin, so it is never split. **A link answers only for what was + relayed over it** (owner, 2026-10-24): every entry of a `bound` or `state`, every `failed`, `loading` and `gone` by + its hold, every `result` and `error` by its call is checked against the relay's table: the hold (the call's hold) + must be relayed over the link the answer came from; otherwise it is dropped (Q1: also an answer crossing a release). + A `bound` or `state` left with some entries is forwarded with those; left with none, it is acknowledged to that link + at once (its window would otherwise lose the slot). A message for an origin from a link the origin relayed nothing + to is ignored whole. Cost: one table lookup per entry, a copy of the entry list only when one is dropped. +- **Upstream drop:** the relay sends `loading { hold }` downstream for every relayed hold on the dropped link, fails + their relayed calls in flight with `unavailable` (never resent), re-resolves (section 2) and holds again upstream + from the kept message (same id, same origin); the new `bound` is forwarded. A `bound` on a hold the consumer has + bound already replaces its replicas in one batch, another variant allowed (3b's re-hold path, now also for live + holds); `loading` on a live hold moves it to `loading`, and on a hold still loading fails a dependent's wait at once + (R18, as a dropped link does). +- **Downstream drop:** the origin's provider side ends: its served holds are released, its relayed holds forgotten, + and every upstream link it relayed to gets one `closed { origin }`, which releases the origin's holds and aborts its + calls there. The origin's number is never used again. +- **`gone`** from upstream is forwarded; the relayed hold is forgotten. + +## 5. Messages (§22, 2b) + +Changes to slice 3a's `MESSAGE` and both codecs: + +- **Parts are opaque bytes:** `params`, `args`, `value` (results), `bound` member values and `state` member ops are the + UTF-8 JSON text as bytes (plus byte segments for payloads marked `bytes`). The binary codec already writes them + length-prefixed; decoding returns views of the received buffer. A part is parsed only where it is owned, right when + its message is handled (so a malformed part is still a protocol error that closes the link): the provider parses + `params` (`checkParams`) and `args` (`checkArgs`), the consumer parses results and snapshots and decodes ops. A + message naming an unknown id is ignored before its parts are parsed (Q1). The text codec, a debugging codec, + re-serializes parts at decode (accepted cost: one parse per part per relay hop). +- `hello { protocol, incarnation, calls }`: `calls` is the number of concurrent calls this side accepts (section 6). +- **`origin?`** on `hold`, `release`, `call`, `cancel`, `retry`, `ack` (towards the provider) and `bound`, `state`, + `failed`, `loading`, `result`, `error`, `gone` (back): the relay's number of the downstream origin (an integer from + 1; absent: the sender's own work). Binary codec: the kind byte's high bit, then the origin as a varint; text codec: + `["@", origin, tuple]`. The schema refuses an origin on `hello`, `ping` and `pong`. +- `hold { origin?, hold, type, version, fingerprint, params, address?, principal?, incarnation? }`: `address` is the + address the caller computed (absent for a type without placement); it routes the hold, the provider places it by + its params. +- **`bound { holds: [{ hold, variant, members, failed }] }`:** one message may bind several holds; the consumer applies + all of them in one batch. Used for first bindings and re-holds. +- `ack { origin? }`: consumer to provider, one per applied `bound` or `state` message of that origin. +- `closed { origin }`: relay to provider, the downstream origin ended: its holds are released and its calls aborted. +- `ping {}` and `pong {}`: the heartbeat (section 7). +- `PROTOCOL_VERSION` stays 1 (nothing is deployed; carry-over L6). + +## 6. Flow control (§3, decision log) + +**The window.** Per origin of a link, at most **K = 2** of the provider's own `bound` and `state` messages +unacknowledged; only these two kinds count (every other message is control and is never queued behind them). The +consumer sends `ack` (with the message's origin) after applying each (after its batch); an `ack` after the link or +the origin closed, or crossing nothing in flight, is ignored. The provider keeps per origin the sender of every +message in flight, in order: itself, or (at a relay) the upstream link it forwarded the message from. Acks arrive in +that order: one for its own message frees a slot, one for a relayed message is sent to that upstream link with the +origin's number there. So a tab served through a relay by W workers may have 2W messages in flight (plus the relay's +own 2), each worker's window bounding its own. + +**The provider while the window is full:** + +- each touched hold member keeps its ops in its **pending queue** (raw ops, never mutated: the same ops feed other + holds and watches; `r` drops everything before it; the leaf index merges a new append or primitive set into the last + pending op on the same path, any other op clears it; past 10,000 pending ops only "replace with the current value" + stays). The cap bounds entries, not payload bytes. When an `ack` frees a slot, one `state` message carries every + waiting hold's entry: its members' pending ops encoded once (or `[["r", value]]` after an overflow), all reflecting + the latest commit. +- a hold that binds while the window is full waits for a slot; its snapshot and subscription are taken in one + synchronous step when its `bound` is sent (3a race 2). +- a hold whose object leaves `live` while its `bound` waits goes back to pending (no `bound` is built for it; slice + 7's reload). + +**The pending queue** is shared Chord code (`src/delta/`); the shared watch handle uses it too, in the same pi change +(carry-over; pi first). + +**The relay** folds, drops and resets nothing: it forwards each upstream `bound`/`state` at once and returns its +`ack` when the downstream acknowledges it. A slow tab stalls only its origin at each worker, whose pending queues fold +its commits as they do for a direct consumer; other tabs and the relay's own holds on the same worker are other +origins with their own windows. A relay keeps no state per message beyond its senders FIFO. + +**Calls.** `hello` announces how many concurrent calls a side accepts per origin (N, default 256, the same on every +host); a side sends no call before the peer's `hello`. A host sends at most N of its own calls per link in flight; +further calls wait in the link's **call queue**, one FIFO, sent in order as replies free slots (it does not preempt +calls already running). A relay forwards calls at once: each downstream origin's consumer is bounded by its own N, +so the upstream sees at most N per origin. Rules: + +- a slot is freed once per call: by its `result` or `error`, by its `cancel` (when sent, at the caller; when received, + at the provider), or by its hold's `release`, `gone`, its origin's `closed` or the link closing; +- a call still queued when its hold rebinds (`loading`, then `bound`) is rejected `unavailable` (it may name a method + index of the old variant); +- a provider seeing more than N running calls of one origin closes the link (protocol error); a call naming a hold it + does not know is ignored before it is counted (Q1). + +## 7. Transports and heartbeat (§3) + +- **`ipcLink(port)`** over a Node IPC channel (`fork` with `serialization: "advanced"`, or `process` in the child), + typed structurally (`send`, `on("message")`, `on("disconnect")`, `disconnect()`), no Node import in Chord. A + `Uint8Array` crosses as one message. `send` returning `false` is backpressure, not failure; a thrown error, an error + callback or `disconnect` closes the link once; messages after close are dropped; listeners are removed on close. +- **`webSocketLink(socket)`** over a WebSocket (`ws` in Node, the browser's), typed structurally: binary frames, + `binaryType = "arraybuffer"`; messages sent while the socket is still connecting are buffered and sent on `open`; + a socket already closed gives a closed link; close on `close` and `error`. The transport's `maxPayload` stays its + hard cap (code 1009; size limits per part are 4b). +- Both deliver whole messages in order and close both ends, like `memoryLinkPair`. They live in `src/next/link/`. +- **Heartbeat on every link kind**, in the channel: each link keeps a `lastReceived` timestamp (updated on every + received message; sends do not count) and the host runs one interval timer for all its links (no timer per + message). A link silent for 15 s sends `ping` (once its handshake is done; `hello` comes first); the peer answers + `pong` at once, never queued behind the window. A link silent for 30 s, the handshake included, is closed (its + references go `loading`, 3b). Both numbers are host options; tests use fake timers. The heartbeat detects a silent + transport, not a peer that answers pings but withholds acks (bounded instead by the window and the pending cap). + +## Races + +Routes and resolve: + +1. Two holds for one address while its `resolve` runs: one `resolve`, shared. +2. `resolve` rejects or returns a closed link: holds stay `loading`; re-resolve after the address's backoff while holds + exist; only a link that completed its handshake resets the backoff. +3. Every hold for an address is released while `resolve` runs: it finishes; the link is kept. +4. `authorize` refuses or throws: the hold fails broken; other principals for the address still resolve. +5. Placement mismatch at the provider: broken `invalid/placement`. +6. A worker restarts: its link closes; the next link's `hello` has a new incarnation; re-holds get fresh objects. +7. `host.close` during `resolve`: its context is aborted; a link it still returns is closed at once. +8. A hold from the parent for a type this host does not serve and has no route for: `unavailable` (never back to the + parent). + +Principals and relays: + +9. A tab sends `principal: { kind: "owner" }` on its hold: ignored (child link); the hold acts as the tab's principal. +10. A guest's handler on the server `ctx.get`s a worker type: the worker sees the guest (principal and incarnation). +11. A worker's handler serving a guest gets a server type: the server sees the worker (upward boundary). +12. The upstream link drops with relayed holds: `loading` downstream, relayed calls in flight `unavailable`, + re-resolve, re-hold with the same id and origin, the new `bound` forwarded; acks for messages of the dropped link + still in flight downstream go nowhere. +13. A tab call that crossed the `loading` notice reaches the relay while upstream is down: `unavailable`. +14. A tab releases a hold whose upstream hold is not sent yet: dropped; already sent: released upstream; messages + crossing it are forwarded and ignored by the tab (Q1). +15. The tab link closes: one `closed { origin }` to each upstream link it relayed to (no `release` or `cancel`); the + worker releases the origin's holds and aborts its calls; its messages crossing the `closed` are dropped at the + relay (the origin is gone). +16. `gone` from upstream: forwarded, the relayed hold forgotten. +17. Holds of two tabs on one worker: two origins, so never one message; each tab's messages are atomic for it. + +Flow control: + +18. An `ack` crossing a `release` or a close, or crossing nothing in flight: acks count messages per origin; one + with nothing in flight is ignored. +19. Two workers' messages for one tab interleave at the relay: the senders FIFO returns each `ack` to its worker. +20. A first `bound` from a worker while the tab is slow: forwarded at once (the worker's window for the tab origin + has room for it, or the binding waits there). +21. A commit touches A and B of one tab on one worker while the tab is slow: one entry each in the worker's next + message for the tab origin, never split. +22. A worker answers for a hold (or a call) of the same origin relayed over another worker's link, or for one the tab + released: the entry, notice or reply is dropped; a `bound`/`state` left empty is acknowledged to it at once. A + link the origin relayed nothing to: its message is ignored whole. +23. The pending queue overflows while the window is full: the next message carries `[["r", value]]` for that member. +24. A failed member, a constant and an empty variant in a first binding: carried in the `bound` (values, failed list). +25. More than N calls: the excess waits in the caller's call queue; a relay forwards at once, so a worker counts at + most N per origin; one more is a protocol error. +26. A call queued across a rebind with another variant: rejected `unavailable`. + +Links and heartbeat: + +27. IPC: the child exits during `send`, or `disconnect` races a message in flight: the link closes once. +28. WebSocket: a frame above `maxPayload` closes the connection (1009): references go `loading`. +29. A WebSocket still connecting: `hello` and early messages are buffered until `open`. +30. A peer that never sends `hello`: closed after 30 s. A busy link sends no pings. Two sides pinging at once: both + answer. +31. A peer stalled by a synchronous handler for over 30 s is closed (documented: handlers must not block the event + loop that long). + +## Cost per hop + +| Hop | Work | +| --- | --- | +| relay, per message | decode the envelope, swap the origin (one shallow copy of the message), encode the envelope; parts copied once into the outgoing buffer (O(bytes)), never parsed; per `bound`/`state` one FIFO entry | +| relay, per hold | the address's params checked once (route admission, `authorize`) | +| provider, window free | as slice 3a: one encode per commit per touched hold member | +| provider, window full | per op: one path key and one map lookup (leaf index); one encode per waiting member per freed slot | +| provider, per origin | a window and pending state per origin with holds; none for a relay | +| consumer | as slice 3a, plus one `ack` per `bound`/`state` message | +| origin | two bytes per message on a relay's upstream links | +| heartbeat | one timestamp write per received message; one host-wide timer | +| trusted link | no `MESSAGE` schema check | + +Producer-side string diffs are not 4a's: a provider whose documents grow by plain string assignment pays Delta's +O(length) append detection per commit until it writes through `appendText` (pi #9265). + +## Benchmarks + +Extend `test/next/bench/` (by hand, recorded in `benchmarks.md`): commit-to-frame and bytes through a relay (tab ↔ +server ↔ worker over memory links, and over real IPC and a loopback WebSocket); hold-to-live through a relay including +`resolve` of an in-process worker; a consumer throttled to 64 KiB/s with stalls of 1, 10 and 60 s receiving a streamed +reply into a 500 KB view (bytes, catching up, sender heap; the producer's commit time recorded separately from the +encode); 10,000 calls from one tab with another tab's latency (p95) through the relay. + +## Carry-over taken by this slice + +L6: trusted links skip the message check; a second `bound` on a bound hold (now in 4a: upstream re-holds); +`PROTOCOL_VERSION` unchanged. L7: `placement` (`invalid/placement`, not in the fingerprint). Decided: C12 (no call +credits; the call window). The pending queue and the shared watch handle change (pi first). §18's core facet pattern +(start data from the start message), shown in example 13. Left for 4b: revocation, cross-host cycles, +`host.references`, `host.activity`, size limits, compression (carry-over section 4b). Left as it is: L6's +`map(bytes())` declaration bug. From the 4a reviews (owner, 2026-10-24): an activation's `readyTimeout` covers its wait for the +previous instance's teardown too (an attempt abandoned there has opened nothing). + +## Tests + +- Host kinds and placement: addresses and keys; `placement` excluded from the fingerprint; every row of section 1's + table; `invalid/placement`; throwing `placement` and `authorize`; never back to the parent. +- Routes: one `resolve` per address, rejection and backoff (fake timers), a closed returned link, re-resolve after a + drop while held and not without holds, `authorize` per principal, `host.close` aborting `resolve`. +- Two-sided links: the server holds a worker type and serves the worker's hold of a server type over one route link; + worker to worker through the server; a host that applies its own messages and forwards a tab origin's, each + acknowledged by its consumer. +- Principals: every row of section 4's table (guest `ctx.get`, remote `self.get`, local `self.get`, `host.get` + through a route), a forged `principal` from a tab ignored, the upward boundary, two origins with equal principal + keys but different incarnations. +- Relays: ids kept under the origin both ways, an origin per downstream link never reused, a worker's entries, + notices and replies for another worker's hold of the same origin dropped (a mixed message forwarded with its own + entries only, an emptied one acknowledged at once); no part parsed at a binary relay (a counter); upstream drop (`loading`, calls + `unavailable`, re-hold with the same id and origin, `bound` with another variant); downstream drop (`closed` per + worker); `gone`; mixed codecs (binary tab, text worker link). +- Flow control: K = 2 with a slow consumer; folding through the leaf index and the cap; a `bound` waiting for a slot, + and going back to pending when its object leaves `live`; a window per origin (a stalled origin folds, another + keeps receiving); `closed` per origin; through a relay: every message forwarded at once, each `ack` back to its + worker in order across two workers, a derive over two holds of a slow tab never torn. +- Calls: the window of N, the FIFO call queue, release and `gone` freeing slots once, a queued call across a + variant-changing rebind, N per origin at the provider (one more closes the link; an unknown hold ignored first), + calls in both directions with equal ids. +- Transports and heartbeat: IPC with a real `fork` (one small test) and a fake port (`send` returning false is not a + close); WebSocket over `ws` loopback and a fake socket (connecting buffer, already closed); silence before `hello`, + a busy link without pings, a half-open link closed at 30 s, simultaneous pings; timers and listeners released. +- Cost guards: no part parsed at a binary relay; one encode per waiting member per freed slot; no `MESSAGE` check on + trusted links; no timer per message. +- GC: a released relayed hold, a closed route link, a dropped downstream link, pending `resolve` waiters, the call + queue, the pending queue after a `closed`, received buffers not kept by parts after handling. +- Examples: `12-routes.ts` (a server routing to an in-process worker, a guest's `ctx.get` through it), `13-relay.ts` + (tab, server, worker over IPC and WebSocket; the worker's `SessionInfo` from its start message), `14-flow-control.ts` + (a slow tab: folding at the provider, and through a relay folding at the worker per origin while a fast tab keeps + receiving). + +## Size + +About 2,500-3,000 changed or new source lines (two-sided links, routes, principals, the relay with origins, opaque parts in both codecs, flow control with the pending queue and the call queue, two transports, the +heartbeat), plus the pi change for the pending queue and the watch handle. Tests and examples about as much again. diff --git a/packages/chord/docs/next/slice-4b.md b/packages/chord/docs/next/slice-4b.md new file mode 100644 index 000000000..51e267828 --- /dev/null +++ b/packages/chord/docs/next/slice-4b.md @@ -0,0 +1,208 @@ +# Slice 4b: authority and housekeeping (spec) + +Status: **revised (2026-10-24) for flow control per origin; the owner sent it to building directly**, with reviews of +the build by fresh reviewers. Guiding rule (owner): simple, fast, correct. The single source for building 4b, together +with the feature map in `slices.md` (entries assigned to 4b) and its "Carry-over" section for slice 4b, `design.md` read +in full (most relevant: §3 links, relays and keep-alive, §4 principals and revocation, §6 cycles, §19 presence, §22 +size limits and compression, §26 open points), the decision log (R2, R10, R11, Q2, Q5, R26, "Compression at the +origin", "Relay flow control per origin", "Slice 4b: `host.activity` is a watchable", "Simplification pass after 4a") +and the root `AGENTS.md` section "Chord v2 Code Style". Builds on slice 4a as simplified (`79a0206dc`). + +Points marked **proposal** are the builder's default; the owner decides them after the build (they are collected in +the last section). + +## What an app can do after this slice + +```ts +type Revocation = { grant: string } | { device: string } | { session: string }; +const server = createHost({ + address: Server(), principal: { kind: "server" }, onError, + revokedBy: (p) => p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] + : p.kind === "owner" && p.access === "device" ? [{ device: p.device }] : [], + compression: nodeCompression, // from `@earendil-works/chord/next/node` +}); +server.revoke({ grant: "g1" }, ctx); // ends g1's objects here and on every worker, closes g1's links + +host.serve(Notes, ({ session }, principal, self) => { + const viewers = self.derive([host.references(Notes)], ([refs]) => + refs.filter((r) => r.params.session === session).length); + … +}); + +const activity = await worker.activity.watch(ctx); // the app's supervisor decides + +server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: 64 * MiB } }, ctx); +``` + +## 1. Revocation (§4, R10, R11) + +- `createHost({ revokedBy })`: `(principal: P) => R[]`, `R` JSON; the same app code on every host. A principal's + **revocation keys** are the canonical JSON of each value `revokedBy` returns, computed once per principal object and + cached weakly (a `WeakMap` keyed by the principal object, with a GC test). L1's unused `revocationKeys` was cut + (owner, 2026-10-24): write the helper here, next to its user. +- `host.revoke(r, context)` on a host, synchronously: + 1. ends at once every object whose identity principal `p` has `r` in its keys (remote objects; local objects have no + principal), whatever references remain (R11): each holder gets `gone` with `forbidden` ("revoked"), running + calls' contexts are aborted (their callers are rejected), an activation in progress is abandoned. Teardown then + runs as for any ending (drain, closes with reason `end`). + 2. closes every **child** link (made by `accept`) whose principal has `r`; never the parent link, never the host's + own principal. Closing a tab's link at a relay sends `closed { origin }` upstream (4a), so that tab's holds on the + workers are released without a revocation message. + 3. sends `revoke { revocation: r }` on every **route** link (its children), which apply it with their own + `revokedBy`, and so on down. Nothing is replayed: a child connected later gets no past revocation (its objects + are new, and the revoked links are closed already). +- A `revoke` message is honoured only on the **parent** link (like the principal fields on holds); arriving on any + other link it is ignored. +- What `gone` means for consumers is 3a's rule (calls on the hold reject; a relay forwards it). Holds acting for a + revoked principal arrive at a worker as their own objects (identity principal = the hold's principal, 4a), so the + worker's own `revokedBy` ends them. +- Reconnecting never restores revoked authority: authentication is the app's; a changed grant is a new principal (its + key includes the grant revision, design §4). +- Jobs and revocation (R11) arrive with slice 5. + +## 2. Cycles across hosts (§6, Q5) + +- Every object instance gets an **instance id** (uuidv7) at its first attempt; a restarted object (a new instance) + gets a new one, so an old chain never matches it (owner, 2026-10-24, D4). +- A hold made during an activation (a `self.get` of a type that is not served here, routed over a link) carries + `chain: string[]`: the chain of the activating object (the chain of the hold that created it, if any) plus its own + instance id; at most 32 entries (a longer chain keeps its last 32: a longer cycle is not detected; documented). +- A provider refuses a hold whose chain contains the instance id of the object it would bind while that object is + activating: the hold fails broken `invalid/cycle` naming the cycle by labels where known. +- **Proposal:** a dependency failing with `invalid/cycle` fails the dependent **broken** with the same error (an + exception to R18's "dependency failures are transient"). Otherwise, A (server) → B (worker) → A would fail B + transient, B would retry with backoff, and every retry would hit the same cycle. +- `chain` is honoured on trusted links (any role: a cycle runs server → worker → server over route and parent links) + and dropped from holds arriving on untrusted links. Relays copy it upstream unchanged (they activate nothing). +- One-host cycles stay as built (slice 1: following waiting objects). + +## 3. Presence: `host.references(Type)` (§19) + +- `host.references(Type)` (on the host, and on the facet's host): a JSON `Watchable` of the references to objects of + that type on this host: `{ params, principal, origin }[]` (`principal` absent for a local type). `origin` is the + incarnation of the host where the reference was made: for a hold over a link, the hold's caller incarnation (the + tab's own, carried by a relay on route links, 4a); for a local reference, this host's incarnation. Two tabs are two + origins. +- It changes in the batch where a reference binds or ends; derives over it recompute there (W1). A link drop is a + departure (its holds end); smoothing is the UI's. Lingering objects have no references. +- Cost: one entry added or removed per reference, kept per type (a type nobody asks about costs a counter at most: + **proposal** build the list per type only while it is watched or read). + +## 4. Keep-alive counts: `host.activity` (§3, R26) + +- `host.activity`: a JSON `Watchable` of `{ references, calls, jobs, lingering }` (owner, 2026-10-24): holds arriving + over links and served here, method calls running on this host (local and remote), running jobs (0 until slice 5), + lingering objects. +- Counters are kept where the work happens; a change marks the watchable dirty, and one publication per microtask turn + carries every change of that turn (not one per call start and end); its value is built from the counters when + published or read. A slow watcher gets the newest counts (pending queue). + +## 5. Size limits per part (R2, Q2) + +- `accept(link, { …, limits })`, `connect(link, { …, limits })` and `resolve`'s result take + `limits: { part?: number; decoded?: number }` (bytes): `part` bounds a part's size on the wire, `decoded` its size + after inflating (section 6). Defaults (Q2): none on trusted links (IPC); on untrusted links 16 MiB and 64 MiB. +- Checked from the part's length before it is parsed or inflated (parts are opaque); the decoded bound is enforced + while inflating (section 6). +- An oversized part fails only its hold or call (`too_large/part`); the other parts of the message apply: an argument + part fails its call (`error` to the caller), a result fails its call, a snapshot fails its hold (no automatic + re-hold), a state part fails its hold. The link stays up. +- **The admitting link's limits travel with relayed work** (Q2): a relay checks `part` on the admitting link itself + (the bytes are there) and stamps `limit` (the admitting link's `decoded`) on the upstream `hold`; honoured only on + the parent link (like the principal fields); calls on the hold inherit it. The host that inflates enforces the + smaller of its own link's and the stamped one, so a worker never inflates a tab's argument beyond the tab link's + limit. +- The WebSocket's `maxPayload` stays the transport's hard cap (4a). + +## 6. Compression at the origin (#27: owner leaning (b), built as (b)) + +Problem: encoding happens in the synchronous flush (one message per commit, W1), so compression must be synchronous; +browsers have only the asynchronous `CompressionStream`/`DecompressionStream`. + +- **Who compresses:** a host with a synchronous compressor (`createHost({ compression })`) compresses each JSON part + above 16 KiB (deflate raw, level 1) when it encodes it; the part's envelope says so (a codec flag per part in both + codecs). Byte segments stay raw. Hosts without one (browsers) send every part uncompressed; their parts are + arguments and params, which are small. +- **Who inflates:** the end that owns the part, when its message is handled. Relays never inflate (parts are opaque). +- **The adapter**, portable: `compression?: { deflate(bytes): Uint8Array; inflate(bytes, limit): Uint8Array }` + (synchronous). Chord ships `nodeCompression` (Node's `zlib.deflateRawSync`/`inflateRawSync` with + `maxOutputLength`) in a Node-only subpath; `src/next` itself imports no Node module. Without the adapter, a host + inflates asynchronously with the web `DecompressionStream("deflate-raw")`, counting output bytes and aborting past + the limit. +- **Order with asynchronous inflate:** a received message whose parts need an asynchronous inflate is held, and every + later message of that link waits behind it, in order, until it is inflated and handled. The queue is bounded: a + provider has at most two `bound`/`state` messages in flight per origin (2W + 2 for a tab behind a relay with W + workers), at most N = 256 call results per origin, and the `ack` goes out only after the message was applied, so a + slow inflate is backpressure like a slow consumer (its commits fold at the provider). Control messages (`ping`, + `pong`) are answered when they are reached in order; the 30 s heartbeat leaves ample room. +- Threshold (16 KiB) and level (1) are protocol constants, not per link. +- `PROTOCOL_VERSION` stays 1 (nothing deployed). + +## Messages + +- `hold` gains `chain?: string[]` (trusted links only) and `limit?: number` (parent link only). +- `revoke { revocation }` (parent to child; honoured on the parent link only; no origin). +- A part's codec flag: in the binary codec a flag in the part's length prefix (builder's choice, documented); in the + text codec, which is for debugging but may sit on a relayed link, a compressed part travels as a tagged element + holding its base64 (**proposal**; the text codec itself never compresses). + +## Races + +1. `revoke` while an object of the principal activates: the attempt is abandoned, holders get `gone`. +2. `revoke` while a call of the principal runs: its context is aborted; the caller is rejected (`gone` reaches it). +3. A hold stamped with a revoked principal is already on its way to a worker: it binds, then the worker's `revoke` + (sent right after on the same link, per-direction order) ends it: R10's window, accepted. +4. Two tabs of one grant, one revoked: both principals have the key, so the shared objects end for both and both links + close. +5. `revoke` crossing a `resolve`: the new worker link gets no past `revoke`. +6. A `revoke` from a child link: ignored. +7. A cycle through a relay: the chain is copied upstream; detection at the provider. +8. A cycle through a restart: new instance ids, no false cycle. +9. `host.references` and a link drop in one batch with a commit: the derive sees the departure in that batch. +10. An oversized part inside a `state` message with other holds' entries: only its hold fails. +11. A compressed part whose inflated size exceeds the limit: its hold or call fails `too_large/part`; the link stays. +12. A stamped `limit` or `chain` from a child or untrusted link: ignored. +13. An asynchronous inflate in flight when the link closes: its message and everything queued behind it are dropped. +14. A slow asynchronous inflate at a tab: the worker's window for the tab's origin fills; its commits fold. + +## Cost per hop + +| Hop | Work | +| --- | --- | +| revoke | keys cached per principal; one lookup per object and link; O(objects + links) once | +| hold during activation | one chain array (at most 32) | +| presence | one entry added or removed per reference, in the batch | +| activity | counters; at most one publication per microtask turn with changes | +| size limits | one comparison per part before parsing | +| compression | one deflate per part above 16 KiB at its origin, one inflate at its owner; relays nothing | + +## Carry-over taken (from `slices.md`, section "Slice 4b") + +All of it: revocation, cross-host cycles with instance ids, `host.references` and `host.activity`, size limits and +compression, the Durable conversation view's size on re-hold (known cost until pi #11327; nothing to build), route +links' context (give `route` a `context` option, defaulting to the host's background context, used for the link's +work and telemetry), one N on every host (unchanged; note stays), `gone` becomes reachable (test it through +revocation; drop the hand-ended test only if the new one covers the same paths). + +## Tests + +Every race; revocation on one host, across a relay to a worker, two tabs of one grant; a child's `revoke` ignored; +cycles server → worker → server and through a relay, no false cycle after a restart, the broken dependent (proposal); +presence per origin with derives and link drops; activity counts per kind and one publication per turn; limits per +part in every message kind, the stamped decoded limit, a compressed bomb (both inflate paths); compression round +trips in both codecs, relays not inflating (a counter), the asynchronous inflate's ordering and its close race; cost +guards; GC (revoked objects, presence entries, revocation key cache, chains, the inflate queue). Examples: +`15-revocation.ts`, `16-presence-and-activity.ts`, `17-limits-and-compression.ts`. Benchmarks in `benchmarks.md`: a +500 KB snapshot and a 1 MiB result compressed and not (bytes, encode and inflate time, synchronous and asynchronous). + +## Size + +About 800-1,000 source lines. + +## Decisions for the owner after the build + +Collected here by the builder and reviewers; known now: + +1. A cycle error makes the dependent broken (section 2), an exception to R18. +2. Compression (b) as built: adapter shape, the asynchronous fallback, the text codec's form of a compressed part. +3. `host.references` built lazily per type, or always. diff --git a/packages/chord/docs/next/slice-5.md b/packages/chord/docs/next/slice-5.md new file mode 100644 index 000000000..97cdaa264 --- /dev/null +++ b/packages/chord/docs/next/slice-5.md @@ -0,0 +1,281 @@ +# Slice 5: routines and jobs (spec) + +Status: **reviewed, ready to build** (2026-10-24; reviews by Astra, Fable, Opus, Sol folded in). Guiding rule (owner): simple, fast, correct. The single source for building +slice 5, together with `design.md` §14 (routines) and §15 (jobs), read in full with the rest of `design.md`, the +feature map in `slices.md` (entries assigned to slice 5) and its "Carry-over" for slice 5, the decision log entries +"Routines for slice 5", "Routines and jobs open points" (with "Jobs keep accepts" and "Job crash handling"), D4, R11, +Q9, S8, W2b, P2, and the root `AGENTS.md` section "Chord v2 Code Style". Builds on slice 4b (`1c88b45d7`). + +Slice 5 builds routines completely and jobs **in memory**. Persistence of jobs is slice 6 (it needs the store); the +reload rules of both (handover, `accepts` in use, `blocked { version }`, native restarts) are slice 7 (they need +reloads). Their APIs are declared here so that later slices add behaviour, not shape. + +## What an app can do after this slice + +```ts +// routines +host.serve(ForwardStore, (_, self) => { + const forwards = self.state(Forwards); + const catalog = self.get(Catalog, {}); + self.on(catalog.sessions, async (sessions, ops, ctx) => { + for (const f of forwards.value.list) if (sessions[f.session]?.archived) await closeForward(f.id, ctx); + }); + self.onLive(async (ctx) => { + for (;;) { await sleep(HOUR, ctx); await dropExpired(ctx); } + }); + return { … }; +}); + +// jobs +const ShellJob = defineJob("chat.shell", { + version: 1, + args: obj({ command: str(), author: Author }), + result: obj({ exit: int(), output: str() }), + state: ShellState, + concurrency: 1, +}); +const shell = self.job(ShellJob, { run, finish }); +await shell.start({ command: "npm test", author }, ctx, { key: requestId }); +``` + +## 1. Routines (§14) + +- `self.on(source, fn)` and `self.onLive(fn)`, declared only while the factory runs (as `state`, `derive`, `get`, + `open`), in objects of both kinds and in facets (design §16: a facet holds documents, references, routines and + contributions). +- `on` over a JSON `Watchable` calls `fn(value, ops, ctx)`; over a `NativeWatchable` (a registry value, `ref.state`) + `fn(value, ctx)`. Overloads by the source's type. At runtime the source is resolved at declaration with + `resolveWatchable` (Chord's own sources, constants, reference members); anything else that is a `NativeWatchable` + (its `watch` yields a `NativeWatch`) is followed as native. There is no separate kind tag. +- Start: right after the object went live, in the same turn (owner); a method call admitted in that turn may run + before the first `on` call. Stop: when the object ends, their `ctx` is aborted with the object's end error; the + drain (§6) waits for routine calls like for method calls, bounded by `closeTimeout`. +- `on` is implemented over the shared `Watch` with a listener that **never throws** (a throwing listener would end the + watch): it runs the handler, catches its error, reports it (`routine_failed`) and schedules its own retry. Calls + never overlap; changes during a call fold into the next (the watch's pending queue: newest value, folded ops); a + running call is never aborted by a change. + - The first call gets the current value with `[["r", value]]` (W2b). + - After a call that threw, the next call (the retry after the backoff, 1 s doubling to 60 s, or a change during the + backoff, which calls at once) again gets `[["r", value]]`, never only the changes since: a handler that applies + ops incrementally would otherwise drift. A call that succeeds resets the backoff. + - The source ending (`ended`, `retired`: a document retired, a reference released) ends the routine without a + report; the source failing (`failed`: a derive's compute threw, a provider member failed) ends it with one + `routine_failed` report. (A failed provider member that recovers after a provider reload: slice 7 carry-over.) +- `onLive`: called once when the object is live. Returning ends it (until the next reload or activation, slice 7); + throwing is reported and restarts it after the same backoff (reset when a run lasted at least a minute); an abort + (the object ends) is no error and not reported. +- Each call's `ctx`: the object's principal (remote) or the host's (local), the host's incarnation, telemetry, and + `ctx.get` references released when the call settles (`CallContext`). +- Not counted in `host.activity`. +- Helpers, exported for apps: `sleep(ms, ctx)` (rejects with the abort reason when `ctx` is aborted; clears its timer + and listener; delays above the timer limit of 2^31-1 ms are chained) and `untilAborted(ctx)` (resolves when `ctx` + is aborted). Portable (timers and `AbortSignal` only). +- `HostError` gains `routine_failed { object, routine, error }`, `routine` being the kind and declaration position + (`"on #2"`, `"onLive #1"`). +- The reload handover per object is slice 7 (carry-over); in this slice a routine only starts with the object and + stops with its end. + +## 2. Jobs (§15), in memory + +### Declaration + +- `defineJob(name, { version, args, result, state?, concurrency?, accepts?, persist? })`: `args` closed (like method + arguments: checked at `start` with a compiled validator, `invalid/args`); `result` not validated (the provider's); + `state` a singleton document definition; `concurrency` a positive integer; `accepts` checked at declaration (shape, + older versions only) and used in slice 7. `persist` has its final shape now, `persist: { resume: "rerun" | + "abandon" }` (resume required, Durable's wording), and is refused in this slice (`not supported before slice 6`) + rather than silently kept in memory. +- `self.job(Job, { run, finish? })` in an object's factory: once per job name per object; returns `JobHandle`. +- The per-run state documents are Chord's: an internal keyed handle per job (named after the job, keyed by run id), + separate from the author's own `self.state` handles, so an author may also use the same definition for a document + of their own without "declared twice". + +### Run records and `running` + +- The job keeps its records on the owner (a map by run id, with the key index); records and state documents change + only inside transactions on the owner's line. Two hooks on `Transaction`: staged record changes (create, outcome, + delete, block, drop) adopted inside the commit step with the documents, and actions after the commit (start a + `run`, settle waiters). +- `running` is a `Source` of its own (not a `Derive`: keyed documents publish nothing), marked changed in the commit + step whenever a record or a run's state document changed, delivering the new list in that batch. Entries: + `{ id, version, key?, args, state, phase: "running" | "finishing" } | { id, version, phase: "blocked", reason }`. + An entry is a new object exactly when its record or its state document's value changed (cached per run by record + and state value identity, never by record alone); the list is copied on change (O(unsettled runs)). + +### Start + +- `start(args, ctx, { key? })`: one transaction on the owner's line; resolves with a `Run` after the commit. + `tx.start(job, args, { key? })` inside `self.tx` joins the transaction's other writes and returns a `Run` handle at + once (a handle, not data from a draft); its `outcome(ctx)` waits for the run and rejects `released` if the + transaction is discarded. So a caller can never miss a short run: the handle exists before the run can settle. + `job.outcome(id, ctx)` waits by id for an unsettled run only (an id of a settled or unknown run rejects `gone`; + documented). +- Order inside a transaction: check args; dedupe by key (an unsettled run with the key, including one started earlier + in the same transaction, returns its `Run`; the args are not compared: a key names one run); concurrency: running + plus finishing runs plus runs started earlier in the same transaction, **excluding the run whose `finish` this + transaction is** (so `finish` can chain the same job at `concurrency: 1`); at the limit `unavailable/busy`; then the + record and the state document (`initial()`, checked once per definition, R8/D3). +- `run` starts after the commit, outside the batch, after `self.tx` settled (an action after the commit). An `abort` + between the commit and the start pre-marks the run's signal, so `run` starts aborted. +- A `start` reaches only a live object (methods and routines run only then). New error reason `unavailable/busy` (the + error table; `PROTOCOL_VERSION` stays 1). Run ids: uuidv7. + +### Run + +- `run(args, ctx)` runs outside transactions. `JobCtx` = `Ctx` plus `jobId` and `state: { value; change(fn) }` (only + when the job has a `state`; typed so). +- `ctx.state.change(fn)` takes **no context**: it is a transaction on the owner's line drafting only the run's + document, under the owner's context without the run's abort signal, so changes made while `run` stops after an + abort still commit. It is accepted while the run's invocation is live (until `run` settles) and refused afterwards + with `released` (an invocation fence: it checks the run's invocation, not only the owner's liveness); it never + recreates a retired document. Changes admitted before `run` settled commit before the outcome is recorded (line + order), so `outcome.state` includes them. +- `run`'s abort signal fires on `abort(id)`, on the object's end and on `host.close`. A run that throws after its + signal fired ends `failed { aborted, reason: "abort" }` (not `threw`); a run that returns after an abort keeps its + result. +- `run` throwing (not aborted): `failed { threw, error }` and a `HostError` `job_failed { object, job, id, error }`. + +### Finish + +- When `run` settles: the outcome is recorded on the run (`phase: "finishing"`), and `finish(outcome, tx)` runs in + one transaction; the same commit deletes the record and the key and retires the state document. Without `finish`, + the commit only cleans up. +- `finish` throwing: nothing changes; retried with backoff (1 s doubling to 60 s), each failure reported + (`job_finish_failed { object, job, id, error }`); after five consecutive failures the run becomes + `blocked { reason: "finish_failed" }`: outcome, state and key kept, not counted for concurrency, retried on the + next reload (slice 7); a later `start` with its key returns the blocked run (dedupe). +- `Run.outcome(ctx)` resolves with the outcome when `finish` committed (also without `finish`); a blocked run's + waiters keep waiting (their own `ctx` ends the wait); a dropped run's waiters reject `aborted`; the object's end + rejects them with its error. + +### Abort, keep-alive, object end + +- `abort(id)`: fires the signal of a running run; drops a blocked run (owner: no `finish`; record, key and state + document go in one transaction; reported `job_dropped { object, job, id }`; waiters reject `aborted`); ignored for + finishing runs and unknown ids. +- Keep-alive: running and finishing runs are a **count** on the object, not `Holder`s (they are not references, so + they never appear in `host.references`). An object whose last reference goes while the count is above zero stays; + when the count reaches zero with no holders, it ends as on its last release (after its type's `linger`). Blocked runs + do not count: at that end they are dropped with a `job_dropped` report each. `host.activity.jobs` counts running + plus finishing runs. +- Object end (revocation R11, `host.close`, the last release): runs' signals fire; the drain waits for `run` + invocations like for calls (bounded by `closeTimeout`); no `finish`; outcomes, states and waiters end with the + object (waiters reject with the object's end error). + +## Messages + +None: jobs and routines are host-local; `running` is ordinary state. + +## Races + +1. Two `start`s with the same key in one turn: the line serializes them; the second sees the first's record and + returns its run. +2. A `start` at the concurrency limit while a run is finishing: finishing counts; `busy`. +3. `tx.start` in a transaction that is then discarded: no record, no run; the `Run` handle it returned rejects + `released` from `outcome(ctx)`. +4. `ctx.state.change` after `run` returned but before `finish` committed: refused (`released`); the outcome's state is + the state at `run`'s settle. A change made after `abort` while `run` is still stopping: commits (no run abort + signal on it). +5. `ctx.state.change` in flight (waiting for the line) when `run` settles: it was admitted while live, so it commits + before the finish transaction (line order); the outcome's state includes it. +6. `abort` while `finish` runs: ignored. +7. The object ends while `finish` waits for the line: the finish transaction is refused (`released`); nothing is + reported as a finish failure (it is the object's end). +8. `finish` throws five times: blocked; a new `start` with the same key returns the blocked run; a different key + starts a new run (blocked runs do not count for concurrency). +9. `on`'s source changes during the backoff after a throw: the call happens at once with the newest value. +10. A routine call throws while the object ends: the end wins; no restart, no report of the abort. +11. `sleep` with a huge delay and an abort: the chained timers are cleared. +12. A job's `run` calls `start` on the same job (recursion): allowed; a new run (it is a different key or none). +13. `finish` chains the same job at `concurrency: 1`: the finishing run is excluded from its own transaction's check. +14. Two `tx.start`s with the same key in one transaction: the second returns the first's `Run`. +15. A short run started with `tx.start` settles before the caller awaits `outcome`: the handle resolves anyway. +16. `abort` between the start's commit and `run`'s start: `run` starts with its signal aborted. +17. The last run settles on an object without holders: the object ends after its `linger`; a blocked run is dropped. + +## Cost + +| Work | Cost | +| --- | --- | +| `on` per change | the shared watch's frame; folded under a busy call | +| `start` | one transaction: an args check (compiled), a record and a document | +| `state.change` | one in-memory transaction on the owner's line (prepare, adopt, flush) | +| `running` per change | one entry replaced; the array copied (O(unsettled runs)) | +| `finish` | one transaction (finish's writes plus cleanup) | + +## Carry-over taken + +From `slices.md` "Slice 5": routine errors reported to `onError` with their own kind; W2b (`on` gets the acquisition +value first); C3 and C4 (decided); jobs and revocation (R11) and `host.activity.jobs`. + +## Carry-over left (to `slices.md` in the same change) + +- Slice 6: `persist: { resume }` built (record and state document in the owner's store; persisted only in `local` + objects with empty params, P3, enforced at declaration); outcomes and abort requests persisted (a recorded outcome + resumes only `finish`); the restart limit (five crash interruptions, persisted, clean closes not counted; whether a + state change resets the count; what a clean close records); deferred persistence for state documents (flushed before + the outcome is recorded); the `finish` failure count persisted; the job fingerprint stored with every record from + the start; the run's start trace carrier stored (§29). +- Slice 7: the routine handover per object on reload (bounded by `closeTimeout`, calls not waiting for it); job + `accepts` in use (args and result mapped forwards once per run; `blocked { version }` without an adapter or for a + newer version); when an older run's state document may migrate while its old code still writes it, and an older run + keeping its old handles valid until it settles; a native restart aborts running runs (`aborted`, reason `reload`) + and `finish` runs on the new code; a `finish` in flight across a reload; run records and state documents belong to + the object across attempts (documents belong to an attempt today); blocked runs retried on reload; a job declaration + removed while runs exist; a job fingerprint over args, result and state (a schema change without a version bump); an + `on` routine whose source failed and recovers after a provider reload. +- Slice 9: spans for routine calls and job runs. + +## Tests + +Every rule and race above, one `describe` per mechanism: + +- `on`: first call with `[["r", value]]`; folding; no overlap; a throw reported, retried with `[["r", value]]` after + the backoff, a change during the backoff calling at once, success resetting it; the watch survives a throwing + handler; source `ended`/`retired` (no report) and `failed` (one report); native sources (`ref.state`); in a facet. +- `onLive`: return; throw and backoff, reset after a long run; abort without report. +- Helpers: `sleep` abort, chaining above 2^31-1 ms (fake timers), no timer left; `untilAborted`. +- Routines with the drain and `host.close`; a routine call admitted in the first turn next to a method call. +- Jobs, start: args check; dedupe (across transactions and twice in one transaction; different args, same key); + concurrency and `busy` (running plus finishing; earlier starts in the same transaction; chaining the same job from + `finish` at `concurrency: 1`); `tx.start` atomic with other writes; a discarded `tx.start`'s handle rejecting; the + handle of a short run resolving; `outcome(id)` for unsettled, settled and unknown ids; `run` starting after the + commit outside the batch; an abort before `run` starts. +- Jobs, run: with and without state; `state.change` fence (after settle refused; in flight at settle committed before + the outcome; after abort while stopping committed; never recreating the retired document); throw after abort is + `aborted`; throw is `threw` with `job_failed`; result after abort kept. +- Jobs, finish: outcome kinds; record, key and state document gone in the finish commit; retries with backoff, + `job_finish_failed`, `blocked` after five; dedupe returns the blocked run; abort drops it (`job_dropped`, waiters + `aborted`). +- `running`: entries per phase; blocked entries without args or state; an entry replaced on a state-only change; no + walk of other runs; per-principal views (example 19). +- Keep-alive: an object without holders stays while runs run, ends after the last one (with `linger`), dropping + blocked runs; `host.activity.jobs`; runs not in `host.references`. +- Object end: revocation and `host.close` mid-run (signals, drain, no `finish`, waiters rejected). +- Cost guards: args compiled once per definition; `running` touches only changed runs. +- GC: an ended object releases runs, documents and waiters; routines release their watches; `sleep` leaves no timer; + dropped and finished runs leave nothing. + +Examples: `18-routines.ts` (`on` with a retry, `onLive` with `sleep`), `19-jobs.ts` (the shell job with a throttled +state document, flushed in a `finally`, changes awaited or caught, `OutputBuffer` with its required bounds, +per-principal views, abort and a chained job). + +## Size + +About 900-1,200 source lines. + +## Decided on the draft (owner, 2026-10-24) + +1. Routines start right after the object went live, in the same turn; a method call admitted in that turn may run + before the first `on` call. +2. `abort(id)` on a blocked run drops it: no `finish`, its record, key and state document go, reported + (`job_dropped { object, job, id }`), waiters reject with `aborted`. +3. `running` carries every run's `args`; views filter (no opt-in). + +Folded in from the spec reviews (`/tmp/s5-review-{astra,fable,opus,sol}.md`): routines in facets (design §16); `on` +over a never-throwing watch listener with its own backoff and a full value after a failure; no kind tag (resolution +at declaration); internal per-job state handles; records with transaction hooks and `running` as its own source; +`tx.start` returning a `Run` handle; dedupe and concurrency within one transaction and excluding the finishing run; +`run` started after the commit outside the batch, an early abort pre-marking it; `state.change` without a context; +throw after abort is `aborted`; `job_failed`; runs counted, not held, and the object ending when the last one settles; +`persist`'s final shape. diff --git a/packages/chord/docs/next/slice-6a.md b/packages/chord/docs/next/slice-6a.md new file mode 100644 index 000000000..417297c4a --- /dev/null +++ b/packages/chord/docs/next/slice-6a.md @@ -0,0 +1,676 @@ +# Slice 6a: the store (spec) + +Status: **built** (2026-10-27: four build-review rounds and six fix rounds; later rules are in the decision log's +2026-10-27 entries and `design.md`, which win over this spec). Was: **ready to build** (2026-10-27; the second spec review decided by its majority, section 15), written by the redesign session after +the first spec review (`/tmp/s6a-review-{astra,fable,opus,sol}.md`) and the owner's decisions of 2026-10-26 and +2026-10-27, including the general-model discussion (facets, attempts, the seal fence, keyed members). Guiding rules +(owner): simple, fast, correct; Chord and Durable aligned as much as possible. + +The single source for building 6a, together with: `design.md` read in full; the store book `store/book.md`; the +decision log `packages/chord/docs/next/decisions.md` (later entries win; superseded parts are marked), in particular "The +store", S1-S17, W1-W14, P2, P3, "Flat columns", "Query runner", "Index definitions are recorded", R8, R12-R15, D9, D10, +"Slice 6 first decisions", "One retirement mechanism", "Text is plain UTF-8, in code point order", and every entry of +2026-10-26 and 2026-10-27 from "The host's store and how it is opened" to "Slice 6a small items"; the feature map and +the carry-over of `slices.md` (slice 6a); the root `AGENTS.md` section "Chord v2 Code Style". Builds on slice 5 +(`4a5b45c45`), the adaptation to the shared `watch.ts` (`cc2d61f53`) and pi `8025c414d` (`runCounted`). + +6a builds the store and puts Chord's persisted data on it (tables, persisted documents, keyed state members), and +reworks three parts of the built slices that the store needs: facets only declare, objects own their documents, line +and job runs across attempts, and the seal fence. Durable on the store (`StoreStorage`), persisted jobs and +`durableView` are 6b. Sync is not a slice: the store keeps an optional outbox the sync layer will read. + +**Words.** An **object** is one identity (type, params, principal key); an **attempt** is one try at activating it +(factory, references, natives, members, derives, routines). A **seq** numbers a commit, one counter per store; +**`lastSeq`** is the seq of the store's last commit. A **write** (`StoreWrite`) is one change of the store (`put`, +`update`, `updateWhere`, `table`, `index`, `document.create`, …); a **commit** in the outbox is the writes of one seq; a +**batch** is the commits `from..to` that sync ships; an **archive** is a manifest, the writes from 0 to `lastSeq` and an +`end`. A **line** serializes one object's transactions; the **commit queue** serializes only the store's commit step. A +**pin** keeps one key of a keyed document loaded. + +## What an app can do after this slice + +```ts +// the start code of a session's worker +const db = await openNodeSqliteDatabase(join(dir, "session.sqlite")); // chord/sqlite/node +const store = await openStore(sqliteBackend(db), ctx, { outbox: false }); // omitted: keep the persisted setting +const worker = createHost

({ address: Session({ id }), principal, store, onError }); +… +await worker.close(ctx); +await store.close(); // the app owns the store + +// contracts +const NoteSchema = obj({ title: str(), body: str(), author: str() }); +export const NoteStore = local("notes.store", { + version: 1, params: {}, + members: { + notes: keyed(NoteSchema), // a keyed state member + create: method(obj({ title: str() }), str()), + rename: method(obj({ id: str(), title: str() })), + remove: method(obj({ id: str() })), + }, +}); +export const NoteView = remote("notes.view", { + version: 1, params: { session: str(), id: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { + editor: { state: { note: NoteSchema }, methods: { rename: method(obj({ title: str() })) } }, + reader: { state: { note: obj({ title: str(), body: str() }) } }, + }, +}); + +// a plugin: the facet only declares +const Note = defineState("notes.note", { + schema: NoteSchema, version: 1, initial: () => ({ title: "", body: "", author: "" }), + keyed: true, persist: true, checkpointWhen: (_v, _ops, info) => info.deltasSinceBase >= 64, +}); +const Edits = defineTable("edits", { + version: 1, + columns: { id: str(), path: str(), at: int(), note: opt(str()), meta: json(EditMeta, { version: 1 }) }, + key: ["id"], + indexes: { by_path: ["path", "at"] }, +}); + +export default defineFacet((host) => { + host.serve(NoteStore, (_, self) => { // local, empty params (P3): host lifetime + const notes = self.state(Note); // persisted, keyed, evictable + const edits = self.table(Edits); + self.on(notes.at("pinned"), async ({ value, retired }, ctx) => { … }); // routines live in objects + return { + notes, + create: async ({ title }, ctx) => { + const id = uuidv7(); + await self.tx(async (tx) => { + tx.put(edits, { id: uuidv7(), path: `notes/${id}`, at: Date.now(), meta }); + (await tx.draft(notes, id)).title = title; + }, ctx); + return id; + }, + rename: ({ id, title }, ctx) => notes.change(id, (d) => { d.title = title; }, ctx), + remove: ({ id }, ctx) => notes.retire(id, ctx), + }; + }); + host.serve(NoteView, ({ id }, principal, self) => { // a view with params: only in-memory state of its own + const store = self.get(NoteStore, {}); + const note = store.notes.at(id); // pins key `id` while this view lives + if (principal.kind === "owner") { + return { variant: "editor", members: { note, rename: ({ title }, ctx) => store.rename({ id, title }, ctx) } }; + } + return { variant: "reader", members: { note: self.derive([note], ([n]) => ({ title: n.title, body: n.body })) } }; + }); +}); + +// a browser tab: an ordinary hold of the view +const view = await tab.get(NoteView, { session: "S1", id: "n-42" }, ctx); +const watch = await view.note.watch(ctx); +watch.start(async ({ value, retired }) => (retired ? showDeleted() : show(value))); + +// tables +await self.tx(async (tx) => { + const old = await tx.get(edits, { id }); + tx.update(edits, { id }, { note: "fix" }); // plain values + tx.updateWhere(edits, (e) => prefix(e.path, "a/"), (e) => ({ path: replacePrefix(e.path, "a/", "b/") })); +}, ctx); +const page = await edits.page(query(Edits).where((e) => eq(e.path, p)).orderBy((e) => e.at, "desc"), 20, ctx); + +// export and import +await pipeline(Readable.from(encodeJsonl(store.export(ctx))), createWriteStream(archivePath)); +const copy = await importStore(sqliteBackend(otherDb), decodeJsonl(createReadStream(archivePath)), ctx); +``` + +## 1. Changes to the built slices + +### 1.1 Facets only declare (decisions "Facet documents are removed", "Facets only declare") + +A facet's setup declares only `host.serve` and `host.references` (`self.contribute` comes with slice 8). Removed: +`FacetSelf` (the setup takes `host` alone until slice 8), facet documents (`FacetSelf.state`), facet routines +(`FacetSelf.on`/`onLive`), the facet's `Documents` and line, the lock order across two lines, `Documents.facet`, +`Transaction.#facetLine`/`close()`, the facet hydration and routine start in `host.ts#setUp`, the facet drain in +`#close`, `ChordHost.#facets` beyond the names, and their tests (`facet.test.ts`, the facet cases of +`routines.test.ts` and `documents`/`transaction` tests, `support.ts` helpers). A facet cannot fail after `load` (a +throwing setup still rejects `load`, slice 2). A transaction covers only its own object. + +### 1.2 Objects own documents, line and job runs (decisions "Documents and the line belong to the object", "Job runs belong to the object") + +- **Slots and handles.** The object (`ChordObject`) holds one slot per declared document name (trackers, the stored + version and fingerprint, pins) and the line. Each attempt's `self.state(def)` returns a thin handle bound to the + attempt's seal token and the definition's version; old handles keep their token and version (R15: old code touching + a slot migrated past its version throws, one integer comparison per read and draft). The same name with a higher + version migrates on access (in memory too); the same version with another fingerprint is `broken`; a flipped + `keyed` or `persist` is `broken`. "Declared twice" is checked per attempt. +- **Lifetime.** Slots live from the object's first activation to its end, across retries, failures and native + restarts. `initial()` is taken once per slot (again when the declaring definition changes). In-memory slots the new + factory no longer declares are dropped when the factory returns (never on a throwing factory); persisted ones drop + their trackers only. Persisted singletons load once per object, keyed data once per residency, again after a failed + load or a drop (section 15). In-memory state + survives a failure and retry; ending the object clears it. +- **Job runs.** Per job name the object holds the records, keys, active count, `running`, run state documents and + pending actions; the attempt holds the implementation (`run`, `finish`) and the running invocations (its drain, its + seal). A run's state changes and its outcome are fenced by the run's own invocation; `finish` runs on the current live + attempt (a finishing run waits for one); a run of a job the new code no longer declares stays `blocked`; a run + waiting for a live attempt keeps its object alive only while an attempt is live or activating. `Job.seal` fires + signals only; waiters reject at the object's end. +- **Derives and watches.** Derives detach with their attempt and the next attempt's derives attach to the same slots. + Routine watches end by their attempt's ctx cancellation, never by a slot drop. Local references re-bind every state + member by name when the object's new attempt goes live, as a remote reference re-holds and gets a fresh snapshot: + their member watches go on across attempts (documents and derives alike; one frame from the last value), and a + consumer's derive over a member recomputes from the new attempt's member. While the object is failed or activating + again, the members keep their last values and get no frames; the builder states this in one rule in `members.ts`. + Keys a reference holds on a keyed member (`at(key)`, `watch(key)`) follow the object's attempts the same way (fix + round 3). A failed derive heals on every later change of its sources and on a re-bind; failures reach watches as + `failed` calls (decision "Failures are watch calls", fix round 4). + +### 1.3 The seal fence (decision "The seal fence") + +- **Tokens.** Every attempt has a seal token; every job run invocation its own (for `ctx.state.change` and its + outcome). +- **At use:** after the seal, new work through the attempt's handles is refused with `released`: `self.tx`, `ctx.get` + (it checks the call's abort signal first, before anything is created), a job start, a routine start. The attempt's + references stay usable for its own natives' closes, then are released. +- **At commit:** a commit is checked when it enters the store's commit step (sealed: rejected `released`, nothing + written, not a storage failure) and at adopt (admitted while current, sealed during the SQL: durable, adopted into + the object's slots, its staged job changes adopted into the object, their invocations deferred to the next live + attempt; the handler's `self.tx` resolves with `fn`'s result, Durable's rule). In-memory commits are checked inside + the batch. The store gets the check through its interface: `store.commit(writes, { admit, adopt }, ctx)` (section 15). +- **The line:** at the seal, a transaction of the attempt that has not entered the step yet is aborted and the line + released; a step already started finishes before the attempt's teardown completes. +- **Cleanup still runs** after the seal: `ctx.get` references of stuck calls released at teardown, presence entries, + `host.activity` counts. The existing start guards stay (`ChordObject.#run`'s attempt identity, `Job.#invoke`, a + `ctx.get` resolving after its call settled). +- **Not a replacement:** a pure swap (slice 7) keeps the attempt. Natives' effects outside Chord are the plugin's; + their `close` is honoured. + +### 1.4 A live object can fail (decision "A live object can fail") + +A runtime failure of the owner (a `migrate` failing on access, a stored version newer than the code) seals the live +attempt and makes the object `failed` with a `broken` cause: calls reject at once with the cause (the triggering call +too), state members keep their last values (same-host reads return them) and get no frames after the commits already +admitted to the step (those adopt and publish once), natives close, slots stay; `onError` gets `object_failed { object, cause }` +once; recovery by reload or `ref.retry()` (a new attempt; only the failed load is redone). Live dependents stay live; +their calls into it fail. + +### 1.5 Smaller changes + +- `ctx.get` checks the call's abort signal first (an ended handler creates nothing). +- **Names** (decision "Restricted names instead of assigned table names"): object type names and host kind names use + `[a-z0-9@/._~-]+`, lowercase, checked at declaration (`contracts/declare.ts`, `host-kind.ts`); `hostKind("client")` + is refused. +- **Errors:** a new `HostError` kinds `storage_failed { error }` and `object_failed { object, cause }`; a new reason + `unavailable/storage_failed` in the protocol's reason list (the test pinning the list changes; `PROTOCOL_VERSION` + stays 1 until p3 ships). +- **Keyed handles** take a context: `get(key, ctx)`, `keys(ctx)` (code point order, in memory too), `watch(key, ctx)` + (decision "Slice 6a's three code questions"). +- **`retired` on the wire and in sources** (decision "One retirement mechanism"): `Source.advance` takes `retired`; + `Subscriber.next` gets it; `Singleton.remove` sends `{ value: initial(), ops: [["r", initial()]], retired: true }`; + a `state` entry lists retired members (`retired: [index]`), kept by the hold while its window is full (ORed until a + message carries it, as `#failed`), validated as member indexes; a derive over a retired source carries the flag on; + a `bound` snapshot carries no flag. + +## 2. The store + +- **Opening:** `openStore(backend, ctx, { outbox? })` with `sqliteBackend(db)` over any `SqliteDatabase` (Node through + `chord/sqlite/node`, Durable Objects through `chord/sqlite/cloudflare`, sqlite-wasm later). There is no hand-written + memory backend: tests use SQLite `:memory:` through `chord/sqlite/node`. The backend interface is internal; `src/next` + imports no Node module. No `openNodeStore` shortcut. +- `openStore` runs the store's own schema migrations (`applySqliteMigrations`), checks that the file uses the UTF-8 + encoding, and reads the metadata once: `lastSeq`, the outbox setting, the table definitions (version, state, columns + with `json(...)` versions and fingerprints, index definitions) and the document id counter (S10); in memory from then + on (one writer per store), persisted with every commit that changes them (S9). In-memory metadata changes only after + `COMMIT` returned. +- **Lifetime:** the app owns the store; the host never closes it (6b's Durable Session shares it). `store.close()` (no + context, as Durable's `Storage.close()`) waits for a running step, rejects queued commits, running exports and reads + with `released`, closes the backend; idempotent, also after `storage_failed`. A host without a store fails an object + declaring persisted data at activation (`broken`, "persist needs a store"). +- **`storage_failed` reaches the host** through an internal listener the store calls once; the host reports + `{ kind: "storage_failed", error }` once and closes itself. +- **The outbox** (decisions "The outbox is optional, internal and versioned", small items 3-4): opened with + `{ outbox: true }`, off with `{ outbox: false }`, an omitted option keeps the persisted setting (new stores: off). + Turning it on records "outbox from seq N+1"; turning it off drops its commits. A table `outbox(seq, format, writes)` + in the backend, written in the step's transaction; `store.outbox` is `undefined` without an outbox (typed). The store + persists a floor (N when turned on at N, advanced by `trim`): `store.outbox.read(afterSeq, limit, ctx)` returns + `{ seq, writes }[]` in the current `format` (older ones stepped forward, a newer one refused with the data intact) for + `floor <= afterSeq <= lastSeq` and rejects any other `afterSeq` (`invalid/args`); `store.outbox.trim(throughSeq, + ctx)` is idempotent, rejects beyond `lastSeq`, takes no seq and runs outside the commit queue (serialized by the + backend). No pluggable encoding. + +## 3. The commit step + +- **One commit queue per store** serializes the step only; each object keeps its own line for the whole transaction. +- **A step:** the seal-fence check; `BEGIN`; the writes (rows, revisions, the outbox commit when there is an outbox, + the new `lastSeq`); `COMMIT`; then the owner's `adopt(seq, ids)` in one `host.batch.commit(...)` (W1), so every source + the commit touched is marked in the step and the flush right after sends one message per link. The next step starts + only after `adopt` returned (S1). Document ids are assigned in the step (monotonic with seq) and passed to `adopt`. +- **Interface** (internal, Durable's shape): `store.commit(writes, adopt, ctx): Promise`. +- **seq:** one counter per store, strictly increasing; an empty commit takes none. +- **Errors** (decision "Store errors: Durable's rule"): any backend error in a step is `storage_failed`: the store + stops, every later commit and read rejects `unavailable/storage_failed`, the host closes. Plugin mistakes are refused + when a write is staged (`invalid/args`) and never reach the step. A seal-fenced commit is rejected `released` before + `BEGIN`. A backend error on a read is `storage_failed` unless the read's `ctx` was aborted (`aborted`); invalid + arguments (a foreign cursor, a bad key) are checked before the backend (`invalid/args`). An `adopt` that throws is + `storage_failed`. Accepted: integer overflow past 2^63 in a bulk `add` is fatal. +- **Counts** through `chord/sqlite`'s `runCounted` (pi `8025c414d`), only where a count matters: bulk writes, the + engine's conditional `UPDATE`, single `update`/`delete`. +- **Statements per commit:** a delta commit of one document is about three statements (revision, the record's + `UPDATE … WHERE id = ? AND version = ?` counted, `lastSeq`), plus one `INSERT` with an outbox, plus one `DELETE` on a + checkpoint (pruning). +- **In-memory commits** never enter the queue and take no seq: a transaction with only in-memory writes commits + synchronously in a batch of its own (no extra await, no promise; a flag set at the first persisted write, no scan). + +## 4. Transactions (decision "A Chord transaction enters the commit step as Durable's does") + +``` +take the object's line + -> run fn (only await tx.*) + -> abort or error: discard everything, reject + -> prepare every draft; checkpointWhen picks base or delta per persisted document + -> nothing persisted written: batch.commit(adopt) synchronously (W12) + -> otherwise: await store.commit(writes, adopt, ctx without its abort signal) + queue; seal check; BEGIN; revisions, rows, outbox, lastSeq; COMMIT + adopt(seq, ids) = host.batch.commit(() => { persisted slots; in-memory slots; staged changes }) +-> release the line; settle the staged changes; drafts revoked (R12) +``` + +- The line is held until the step returned; once admitted, a commit runs to its end (the caller's abort no longer + counts); the in-memory part is adopted inside the same step (P2). +- **Cold keyed loads** (decision "Cold keyed loads on the line"): a cold `get`, `watch` or pin of a keyed document is a + job on the object's line, as Durable's `snapshot`; cached reads take no line; `tx.draft` loads inside its + transaction; one load in flight per key, shared, each caller's `ctx` ending only its own wait; a load installs its + tracker only while the key is still absent. `tx.retire(handle, key)` of an unloaded keyed persisted document finds it + inside the transaction (one read; absent: a no-op, R13). +- **A migrated base** is written back by the next commit that changes the document; until then a load migrates again + (once per object). + +## 5. Tables + +- `defineTable(name, { version, columns, key, indexes? })` (later versions through `evolveTable`, section 9). `columns` is a flat record of L1's helpers as + a restricted subset: `str()`, `int()`, `bool()`, `num()`, `opt(...)` of those, and `json(schema, { version, + migrate? })`; defaults as `int({ default: 0 })`; no nesting outside `json(...)` (types and declaration). Keys, index + columns and `orderBy` columns are non-`opt` `str` or `int` columns. Every index ends with the primary key. +- `self.table(Def)` only in the factory of a `local` type with empty params (P3; type error and declaration error), + once per table name per object. +- **Names** (decision "Restricted names"): table names `[a-z0-9@/._~-]+`, lowercase; the SQL table name is + `"::"` (a host without an address uses `client`); `:` cannot occur in the parts, so + the rendering is injective. Index names `[a-z0-9_]+`, rendered `
:`. Column names + `[A-Za-z_][A-Za-z0-9_]*`, no two equal ignoring ASCII case, no `$`. +- **Layout** (S16): a real SQLite table per logical table (`STRICT`, `WITHOUT ROWID`, the declared key as primary + key), one column per declared column; a `json(...)` column is `TEXT` plus a hidden integer `$v`; a hidden + integer `$v` per row (eager steps applied). The store renders the DDL and records each definition as a `table` write + (columns with types, `opt`, defaults, `json(...)` versions and fingerprints, key, version, state). +- **Text** ("Text is plain UTF-8, in code point order"): `str()` is plain UTF-8 `TEXT` compared by code point; written + values and every operand go through `toWellFormed()` (check `isWellFormed()` first; copy only when needed); NUL fails + `invalid/args`; `compareText(a, b)` exported; a prefix's upper bound is the next code point after its last, carrying + past U+10FFFF (an all-U+10FFFF prefix has no upper bound; the empty prefix matches every value); `substr` offsets + count code points. +- **Values at write:** `int` a safe integer, `num` finite, `bool` stored 0/1 and read back as `boolean`, `str` as + above; a row has every non-`opt` column and no other field; one compiled closed check per table (a plugin's row is a + trust boundary); `json(...)` values are not walked (R9). +- **Nullable columns** (decision "Nullable columns and the `set` form"): an `opt(...)` column is an optional field, + absent when `NULL`, never an explicit `null`; SQL's `NULL` rules apply to every comparison. + +## 6. Queries and reads + +- `query(Table)` is immutable data: `.where((t) => …)` with `eq ne lt lte gt gte inArray isNull prefix and or not` over + scalar columns, `.orderBy((t) => t.col, "asc" | "desc")` (several, one direction per query in 6a); no joins, + grouping or projections (S12). The default order is the key ascending; the key is appended to every `orderBy` as + the tie-break. +- **SQL** is generated from the query data; values always bound, identifiers quoted; `limit` and cursor values bound, + `inArray` one JSON parameter through `json_each` (no 32,766-parameter limit; `inArray([])` matches nothing). Query + shapes are unbounded (any `and`/`or` tree), so the facade's prepared-statement cache is bounded (pi change, a fixed + LRU in `chord/sqlite/node`, section 15). +- **Outside a transaction** the handle reads committed state: `page(q, limit, ctx, { after? })` → `{ items, next? }`, + `rows(q, ctx)` (all pages), `get(key, ctx)`, `count(q, ctx)`. A key is an object with exactly the key columns + (`{ id }`, `{ path, entry }`); other or missing fields fail `invalid/args`. A query for another table is refused. +- **Cursors** are opaque strings bound to table, order columns and direction; decoding one is a trust boundary + (`invalid/args`). +- **Inside `self.tx`**: the table handle first (another object's is refused); reads async: `tx.page(table, q, limit, + { after? })` and `tx.rows(table, q)` read committed rows, stable for the transaction; `tx.get(table, key)` overlays + the transaction's own `put`/`update`/`delete` (one evaluator for `where` and `set`, also used for the overlay); + after an `updateWhere`/`deleteWhere` on a table, `tx.get` on it throws `invalid/read_after_bulk` (W10). + +## 7. Writes + +- Five writes (S17), synchronous and buffered like a draft's ops (checked and staged, no I/O, no line; a bad row + throws at its own call; the buffer is written in the step or dropped): `tx.put(table, row)` (an upsert: the whole row + replaced), `tx.update(table, key, set)`, `tx.delete(table, key)`, `tx.updateWhere(table, where, set)`, + `tx.deleteWhere(table, where)`. No write shorthands on the handle. An `update`/`delete` of an absent key changes + nothing. +- **`set`** is a plain object of values or a callback with typed column references: `{ note: "fix" }`, + `(e) => ({ hits: add(e.hits, 1), path: replacePrefix(e.path, from, to) })`; `undefined` clears an `opt` column. + Expressions over real columns only: `add sub max min concat replacePrefix coalesce when` (`when(cond, a, b)` with + `cond` a `where` expression over the row); integer arithmetic only; every right side is evaluated against the row + before the write (`{ a: e.b, b: e.a }` swaps); `replacePrefix` on a value without the prefix leaves it unchanged. +- **Writes in the outbox commit** carry what a replica needs, explicitly (decision "Writes carry their versions"): + `put` with the row, `v` and each `json(...)` column's version; `update` with its `set` and `count`; `delete` with the + key and `count`; `updateWhere`/`deleteWhere` with `where`, `set` and `count` (a replica affecting another count stops + applying, S17). +- `$v` is written only by eager migration steps (an internal write form) and by `put` (the table's version); `update`, + `updateWhere`, `deleteWhere` never touch it; a write of a `json(...)` column stores its current version (W11). + +## 8. The documents engine + +- One engine (S13). Address `(namespace, owner, kind, key)`; Chord's: namespace `chord`, owner + `:` (W7; `client` without an address), kind the definition's name, key the keyed document's + key or `""` for a singleton (so the partial unique index on the live address also refuses two live singletons). Owner, + kind, key and `meta` strings follow the `str()` rules. +- A record per incarnation (`id`, address, `history: "latest" | "rewindable"`, `createdAt` (a seq), `retiredAt?`, `meta`, + `revision`); revisions: a base (version, fingerprint, value) and deltas (ops). Chord uses `history: "latest"` (a new + base and a retirement prune older revisions). +- Reads: `find(address)`, `load(id)` (base plus deltas applied once: value, version, fingerprint, `deltasSinceBase`), + `list(query, limit, cursor)` by key. Reads at an older seq (`at`) and `history: "rewindable"` are 6b. +- Writes inside the step: `document.create`, `document.change`, `document.retire`, checked by the writes (one live per + address by a partial unique index; a delta only on its record's current version through the counted conditional + `UPDATE`; nothing after retirement); a failed check is `storage_failed` (memory and disk disagree). + +## 9. Migrations + +- **`json(schema, { version, migrate? })` columns** ("The `json(...)` column helper"): a read of an older value runs + `migrate` and checks the result once (R8); a read never writes back; without `migrate`, a throwing one or an invalid + result the read is refused and the owner fails (section 1.4); a newer stored version is refused the same way; at + activation, the same version with another fingerprint fails the object `broken` (D9). +- **Eager migrations: `evolveTable`** (decision "Second spec review: the majority"; all four reviewers accept it; the + derived adds and drops are withdrawn: a store at v1 cannot reconstruct v2's columns from v3's declaration): + + ```ts + const EditsV1 = defineTable("edits", { version: 1, columns: { id: str(), file: str(), at: int() }, key: ["id"] }); + const EditsV2 = evolveTable(EditsV1, (t) => + t.rename("file", "path").add("dir", str({ default: "" })).rows((row) => ({ dir: dirname(row.path) }))); + const EditsV3 = evolveTable(EditsV2, (t) => t.drop("dir").index("by_path", ["path", "at"])); + host.serve(FileHistoryStore, (_, self) => { const edits = self.table(EditsV3); … }); + ``` + + - **History.** `defineTable(name, { version, columns, key, indexes? })` is a base written in full (no `migrations` + option); `evolveTable(prev, (t) => t…)` appends `prev.version + 1` (contracts' `Increment`) and its edit returns + the editor it was given (contracts' rule, checked). Every version is materialized and fingerprinted at + declaration (`TABLE_FORM_VERSION`, its own form version: table fingerprints are persisted, not sent). The object + serves the newest. A new base (`defineTable(name, { version: 7, … })`) starts a new history; stores below it fail + `broken`. + - **Edits:** `rename(from, to)` (key, index and hidden `$v` references follow), `add(name, column)` (`opt(...)` + or with a `default`), `drop(name)` (its indexes are dropped first in the same commit; a key column cannot be + dropped), `index(name, columns)`, `dropIndex(name)`, `json(name, json(schema, { version, migrate? }))` (a + `json(...)` column's version change: no DDL, no data step, cells still migrate on access; every change of a table + definition is a table version), and at most one data step: `rows(fn)` or `update({ where?, set })`. No key or + column-type changes (`rebuild`, carry-over). The editor's types enforce the order: `rename`/`add`/`json` before the + data step, `drop`/`index`/`dropIndex` after it; the data step's row type is the columns at that point of the chain, + so a step may read a column its version drops and fill one its version adds. + - **Commits per version N:** (A) the edits before the data step as DDL plus `table { version: N, state: "migrating" }`, + one commit; the data step's chunks; (Z) the last chunk (or the `update`) with the edits after the step, their index + changes and `state: "ready"`, one commit. No data step: (A) and (Z) are one commit. Recorded writes: `alter + { table, version, edit }` per column edit (`rename | add | drop | json`), `index`/`dropIndex`, `table { version, + state, columns, key, fingerprint }`; the applier renders DDL from them. Resume at `version N, migrating`: (A) is + done; the step continues; then (Z); N+1 starts only after N is `ready`. + - **Data steps:** `rows` is synchronous and pure; it gets that version's typed row (`json(...)` cells migrated on + access to that version's column versions) and returns a patch of non-key columns (`{}` stamps `$v` only; a key or + unknown column fails `broken`); chunks of 100 in key order, paging forward by the last processed key (a crash + restarts the scan, skipping rows with `$v >= N`); the patch plus `$v = N` written through an internal write form. + `update` is one `updateWhere` with `and(where, lt($v, N))` stamping `$v = N`, `ready` in the same commit, so a + non-idempotent `set` never applies twice. A throwing `rows` fails the owning object `broken`, the table stays + `migrating`, committed chunks stay. Data functions are immutable once deployed (the guide). + - **Activation:** a recorded version newer than the code's newest, older than the code's base, or the same version + with another fingerprint (a history edited in place) fails `broken`; `self.table` of an older version than the + recorded one fails `broken`. A fresh table is created directly at the newest version (`ready`, `$v` = newest), with + its indexes, no replay. + - **Shared with contracts:** `hash64`, L1's `snapshotSchema` normal form per column, `Increment` and the "returns + the editor it was given" check move into one shared module; the table history is its own concept + (`store/table-history.ts`, `store/table-evolve.ts`), not contracts' `Draft`, `Editor` or `checkAccepts`. +- **Where and when:** at the owning object's activation, after its persisted documents loaded and before its + references are bound, on the store that owns the data (a replica never migrates). The attempt's `readyTimeout` + deadline is paused while eager migration work runs and resumed with its remaining budget (one pause scope over the + table checks, (A) and (Z) commits, data chunks and their waits in the commit queue; resumed in a `finally`); the attempt's abort is checked between chunks (a chunk admitted to the step completes); the migration + yields to the event loop between chunks; the start barrier waits for host-lifetime objects' migrations. Getters and + parked calls keep their own deadlines. +- Old code meeting a newer table version or a newer `json(...)` version fails the object `broken`, data intact. + +## 10. Persisted documents + +- `defineState(name, { …, persist: true, checkpointWhen })`: `checkpointWhen(value, ops, info)` required for persisted + documents (a type-level constraint), called once per commit per changed document; creation and version transitions + always store a base. The fingerprint is L1's structural fingerprint of the schema, computed once. +- Only in `local` types with empty params (P3). +- **Loading at activation:** once per object, right after the first factory, every persisted singleton and every + pinned `at(key)` with plain reads (the object holds its line; no read transaction): find, load, the fingerprint + check, `migrate` with its result checked once (R8); an absent document reads `initial()` (Q11). A storage error is + `storage_failed`; a fingerprint mismatch, a throwing `migrate` or an invalid result fail the object `broken`. +- **Keyed persisted documents** load on demand (section 4): `get(key, ctx)` (`undefined` when absent), `keys(ctx)` + (pages from the engine in code point order), `watch(key, ctx)` (`undefined` when absent), `tx.draft(handle, key)`. +- **Pins and eviction:** each loaded key counts its pins: an `at(key)` declared by the current attempt's factory, a + keyed member's `at(key)` of a consumer while it lives, each watch, each load in flight. A key with no pins not read or + written for 10 minutes leaves memory (one owner-level timer and idle structure, no timer per key; eviction runs on the + object's line). Persisted singletons stay in memory while the object lives (`handle.value` is a synchronous read). + +## 11. Keyed watches, `at(key)` and retirement + +- Every document, in memory or persisted, is watchable; a watch follows the address: + - a singleton's retirement is one call `{ value: initial(), ops: [["r", initial()]], retired: true }`; it reads + `initial()` meanwhile; a later write is an ordinary change; + - `await handle.watch(key, ctx)` is `undefined` when absent; a retirement is the same call and the watch goes on; a + later write delivers the new incarnation as `[["r", value]]` (a retire and recreate in one transaction: one call + with `retired: true` and the new value); + - watches end `ended` with their object. +- **`handle.at(key)`** in the factory: a view of the keyed handle's slot for that key (the same tracker, the same draft + in a transaction: never a second document), pinned while the declaring attempt lives, `initial()` while absent; a + state member, a derive source, an `on` source. + +## 12. Keyed state members (decision "Keyed state members, `local` contracts only") + +- `keyed(schema)` is a third member kind (`MemberKind` `"keyed"`), part of the fingerprint (normal form updated), only + in `local` contracts (`remote()` refuses it at the type level and at declaration). +- **Provider:** returns a keyed document handle for it (in memory or persisted), or re-exposes another reference's + keyed member; checked at activation as state members are. +- **Consumer on the same host:** `ref.member.at(key)` only in its factory (a `Watchable`: a pin on the provider's + per-key source while the consumer's attempt lives; loaded before the consumer goes live; `initial()` while absent; + the `retired` flag on retirement); `ref.member.get(key, ctx)` and `ref.member.watch(key, ctx)` anywhere (`watch` on an + absent key gives `undefined`). +- **Hydration:** `host.get` of a consumer waits for its declared `at(key)` sources like any state member's value; keys + read dynamically are not part of hydration. +- A key whose `migrate` fails fails its owning object `broken` (section 1.4), never the store. Versions of keyed members map per key in slice 7. +- Nothing on the wire changes: across links a view per entity is held as an ordinary remote object. + +## 13. Export and import (decisions "Export archives are operations", "Writes carry their versions; archives are lossless", small items 6-7) + +- `store.export(ctx)` yields `AsyncIterable`: the state after its `lastSeq` as writes, lossless: a + manifest `{ kind: "manifest", format: 1, lastSeq, schema }`; per table its `table` write (with `state`), its `put`s + (stored values and versions) and its `index` writes; the `document.create`s (stored version and fingerprint; for + `latest` the current value as the base); an `end` with the counts. Retired `latest` records, blobs, the outbox and + sync state are not in it; tables of plugins that are not loaded travel along. + + ``` + {"kind":"manifest","format":1,"lastSeq":4812,"schema":1} + {"kind":"table","table":"session:file-history.store:edits","version":4,"state":"ready","columns":{…},"key":["id"]} + {"kind":"put","table":"session:file-history.store:edits","row":{…},"v":4,"columns":{"meta":1}} + {"kind":"index","table":"session:file-history.store:edits","name":"by_path","columns":["path","at"]} + {"kind":"document.create","record":{…},"base":{…}} + {"kind":"end","writes":10061} + ``` + +- It takes the commit queue at its first `next()` and holds it until the iteration ends, throws, is closed + (`return()`) or `ctx` aborts (an abort listener releases it even while the consumer is suspended; a later `next()` + rejects `aborted`); it reads page by page with plain reads (no long read transaction). The guide: export to a local + file, then upload. +- `importStore(backend, parts, ctx, { outbox? })` is a trust boundary: every part checked against one compiled + `ArchivePart` schema, every name against the name rules, every row against its table's recorded columns, the order + above and the `end` counts; it refuses a non-empty backend (one with the store's tables) and parts after `end`; + applies everything in one backend transaction with the replica's applier (one code path for import, bootstrap and + batch apply) and returns the store at the manifest's `lastSeq`, its outbox (if asked) starting after it. +- The encoding is pluggable: the store yields and takes parts; it ships plain JSONL (`encodeJsonl(parts): + AsyncIterable`, `decodeJsonl(bytes: AsyncIterable)`). + +## 14. Benchmarks and tests + +- **Benchmarks** (`test/next/bench/store.ts`, results in `benchmarks.md`), the gate (decision "The commit-step + benchmark"): the store without outbox no slower than Durable's `SqliteStorage` per commit on (a) one document delta, + (b) one table `put` (against `SqliteStorage`'s entry append), (c) 1,000 commits back to back, with and without a + watched document; bursts (median and p95 over several runs), memory after GC, the same pragmas and file system. The + outbox's extra cost reported separately; also query pages, a 10k-row `updateWhere` (its step time: bulk writes block + every owner's commits), text per row, a migration chunk. A failed gate goes to the owner before the build continues. +- **Cost guards** (in-memory telemetry adapter): validations (`migrate` results), materializations (document loads), + store commits and statements per commit; none on the in-memory-only path; a cached keyed `get` touches no SQLite. +- **Tests,** one `describe` per mechanism: facets that only declare; slots, handles and job runs on the object across + retries, failures and native restarts (R15 through old handles); the seal fence (sealed handles, both commit checks, + a queued commit of a sealed attempt rejected, an admitted one adopted into the object, the line released, cleanup); + a live object failing; the commit step (order, adoption inside the step, errors, `storage_failed` once, reads after + the stop); transactions (the line across the step, no abort once admitted, the in-memory part in the same step, + in-memory-only commits without a step); cold keyed loads on the line (a cold `get` racing a draft of the same key, a + cancelled waiter); the outbox (on, off, omitted, switching, `read` gaps, `trim`, `format`); tables (declaration and + name checks, DDL, P3 errors, values, text, nullable columns); queries (every operator, `NULL` rules, order and + tie-breaks, cursors, keys as objects, reads inside and outside transactions, W10, bounded SQL texts); writes (all + five, `set` forms, the swap, upserts, counts, their outbox writes); migrations (`json(...)` on access, fingerprints, + eager steps, chunks, resume, indexes, the paused deadline with a fake clock, a timer firing during a long migration); + the documents engine (rules, ids, pruning, the singleton address); persisted documents (load once per object, + fingerprints, `migrate`, keyed `get`/`keys`/eviction and pins, checkpoints, the migrated base written back); + retirement (singleton, keyed, `at(key)`, `self.on`, derives carrying the flag, the wire with a full window); keyed + state members (pins, hydration, retirement, two consumers of one key); export and import (round trips with old + versions, mid-migration and uncheckpointed deltas, every refusal, the queue released on abort while suspended); GC + (evicted documents, unpinned keys, ended objects, stopped watches, export iterators closed early, abort listeners). + Examples: `20-tables.ts`, `21-persisted-documents.ts`, `22-migrations.ts`, `23-keyed-watches.ts`, + `24-keyed-members.ts`, `25-export-import.ts`. + +## 15. Rules from the second spec review (decision "Second spec review: the majority") + +The owner delegated the second review's findings to the reviewers' majority (`/tmp/s6a-review2-{astra,fable,opus,sol}.md`; +a point raised by one reviewer only takes that reviewer's recommendation). These rules complete the sections above; +where a section above says otherwise, this section wins. + +**Objects, attempts and the fence** +1. **Slot versions:** a slot keeps `valueVersion` (the version of its value in memory, the handle fence, R15) and + `storedVersion` (the version on disk, for write preparation), with their fingerprints, as Durable's + `LoadedDocument`. An old handle whose version is below the slot's fails that access, not the owner; a stored version + newer than the current definition fails the owner (1.4). 6a builds the version comparison and the in-memory + `migrate` (tested through a retry whose factory declares a newer definition); slice 7 wires it to reload. + `initial()` is retaken when the declaring definition's version or fingerprint changes, not on every attempt. +2. **The store's admission check:** `store.commit(writes, { admit, adopt }, ctx)` (internal): `admit()` is called at + the step's entry (false: the store rejects `released` and drops the entry, never writes it); `adopt(seq, ids)` + after `COMMIT`, `ids` mapping the commit's `document.create` writes by index to their ids. 6b's `StoreStorage` + passes `admit: () => true`. +3. **A transaction at the seal** is discarded: drafts aborted, every later `tx.*` throws `released`, a callback that + returns afterwards is rejected `released` without a commit, and the line is released. As Durable's `Transaction`, + it tracks its pending async `tx.*` operations and refuses a callback that settles with one still pending. +4. **Sealed handles** may still read synchronously (`.value`, cached keyed reads); anything taking the line (`change`, + `retire`, a cold `get`/`watch`/`keys`) is refused `released`. A live failure of an already sealed attempt is ignored. +5. **Jobs at the seal:** staged changes of an admitted commit are adopted into the object (section 1.3). A run cut off + by its attempt's seal ends `failed { aborted, reason: "restart" }` (design §15's reason, now also "the attempt + ended while the object continues"); its outcome transaction runs on the object's line fenced by the run's own token + (a transaction path owned by the object, not an attempt). A `finish` sealed mid-way is refused at the step and + retried on the next live attempt, not counted as one of the five failures. A run's state-change admission closes + when `run` returns; changes already admitted drain, then the outcome is recorded. +6. **Pins between attempts:** the object holds the factory's `at(key)` pins until the next factory returns (the moment + unused in-memory slots are dropped). Each attempt loads its declared singletons and pins not in memory (line jobs) + before it goes live. A consumer whose pin load fails the provider fails as a dependency failure (`transient`, R18). +7. **One host per store:** `createHost` refuses a store attached to an open host (`invalid/args`). + +**The commit step and errors** +8. **Serialization before `BEGIN`:** every write is serialized when it is staged (table writes) or prepared (documents, + before the queue); a value `JSON.stringify` cannot encode (a cycle, a `BigInt`) fails there (`invalid/args` or the + transaction), never in the step. +9. **Order inside one commit:** retirements before creations (the partial unique index on the live address), then + changes, rows, the outbox commit, `lastSeq`. +10. **Integers:** an `int` column has a SQL `CHECK` keeping it within `±(2^53 - 1)`; an expression result outside it + (`add` past the safe range) fails the step, which is `storage_failed` (accepted, as the 2^63 case was); the JS + evaluator (the `tx.get` overlay) checks the same range and refuses out-of-range results. `add`/`sub` take `int` + columns only; `max`/`min` take `int` or `num`. +11. **Reads after a failure:** as Durable's `GuardedStorage`, a read that settles after the store stopped rejects + `unavailable/storage_failed`, never returns a value. A shared cold load runs under the owner's context, not its + first waiter's. +12. **Open and import failures:** `openStore` and `importStore` close the backend they were given when they reject; + a refused import rolls back every table it created. An import whose input iterator stalls is cancelled by `ctx` + without holding the backend transaction past it. + +**Tables, queries, writes** +13. **Key columns are immutable** in `update`, `updateWhere` and migration patches (types and staging, `invalid/args`); + moving a row is `delete` plus `put`. +14. **Canonical writes:** both `set` forms compile at staging into one data-only expression vocabulary, with an + explicit `null` literal for a clear (`{ note: undefined }` would serialize to `{}`); the outbox and archives carry + that form, never callbacks or `undefined`. An empty `set` is `invalid/args`. +15. **Defaults:** a column with a `default` may be omitted in a `put` (SQL `DEFAULT`); reads always return it. +16. **Operands:** comparing an `opt` column with `undefined` is `invalid/args` (ask with `isNull`). +17. **Pages:** `limit` is a positive safe integer; `rows()` pages internally by 100; `next` comes from fetching + `limit + 1`; a cursor is a position in the order (one from another `where` with the same order is accepted); + duplicate `orderBy` columns are refused and the key tie-break is not repeated. +18. **The prepared-statement cache** of `chord/sqlite/node` is a fixed LRU (pi change, upstream first, before the + build): arbitrary query shapes cannot grow it without bound. + +**Documents and keyed members** +19. **Retiring a `latest` record** deletes its row and revisions (Chord never reads history; a key retired and + recreated often leaves no dead rows); `full` records keep them (6b). The record's counter is `revision` (not + "version") in the conditional `UPDATE`. +20. **Absent keys:** a keyed `get` of an absent key is not cached (a line job plus a read each time; accepted, stated in + the guide); `watch(key)` on an absent key returns `undefined`, and a later creation is seen through `at(key)` or a + new watch. +21. **Retired derives:** a derive over a retired source publishes `retired: true` even when its value is unchanged and + its ops are empty (`Derive.deliver`'s identity and empty-diff shortcuts must not swallow the flag). +22. **Keyed members:** a re-exposed keyed member resolves to the ultimate provider's source (no second tracker); pins + are released exactly once (failed consumer activation, retry, release, re-exposure); `ref.member.get/watch` while + the provider is loading or failed behave as calls (park while loading, reject with the cause when failed); keyed + members have no `keys` (a method lists keys); `keyed(...)` adds a node to the fingerprint without bumping + `NORMAL_FORM_VERSION`. +23. **Eviction's idle structure** is Durable's (`#touchDocument`'s granule rule: one move per tenth of the retention, + O(1) per touch); one owner-level timer. + +**Export and import** +24. **The manifest carries `nextDocumentId`** (the id high-water mark), restored atomically and checked against the + imported records; import keeps ids and `createdAt` seqs. +25. **Import validation** also covers record ids and seq ranges, one live record per address, versions, duplicate + definitions and agreement with the manifest; `json(...)` cells are checked as JSON text only (no plugin code runs). +26. **Export cancellation** releases the queue and its page buffers and metadata references (GC-tested with a retained, + aborted iterator). Code that commits while consuming its own export waits until `ctx` aborts (the guide; a test). + +**Benchmarks and cost** +27. A fourth gate case, (d) a commit with one in-memory document only (the store adds nothing to the W12 path); queue + wait reported separately from step time; late migration chunks measured as well as the first. + +**Carry-over (to slices.md):** slice 7: remote holds and remote objects leaving `live` (`Hold.moved`, +`RemoteObject.failed`; not reachable in 6a, P3), old table handles' version fence, the routine handover fenced per +routine invocation, holds re-bound by member name, keyed members' versions per key, `rebuild`. + +## Races + +1. Two objects commit at once: their steps queue; each adopts inside its own step, in seq order. +2. A transaction discarded after its callback while another owner's step runs: it never entered the queue. +3. The store fails during a step: that commit rejects; later commits and reads reject `unavailable/storage_failed`; + the host closes. +4. An object's attempt is sealed while its commit waits in the queue: rejected at the step's entry; one already in the + step lands and is adopted into the object's slots; the next attempt drafts from them. +5. A cold `get` and a transaction's draft of the same key: both on the line; the load installs only while absent. +6. A keyed document evicted while a `get` for it is in flight: eviction runs on the line and skips pinned keys. +7. A migration crashes in the middle: the next activation finds `version N, migrating` and continues. +8. An export holds the queue while commits and migration chunks wait; the export's `ctx` aborts while its consumer is + suspended: the queue is released. +9. A retire and a recreate of one key in one transaction: one call with `retired: true` and the new value. +10. Old code (after a rollback) meets a newer table, column or document version: the object fails `broken`, data + intact; a newer outbox `format`: `outbox.read` rejects, data intact. +11. The outbox is switched on for a store with data: the metadata says from which seq it starts. + +## Cost + +| Work | Cost | +| --- | --- | +| commit of one document delta | one step: about three statements, plus one with an outbox, plus one on a checkpoint | +| commit of in-memory documents only | as today: no step, no seq, no await | +| table put | two statements in the step (the row, `lastSeq`), plus its write in the outbox commit | +| `updateWhere` over N rows | one statement, one write; holds the queue for its duration | +| read of a page | one query | +| keyed `get` (cached) | no SQLite; one touch of the idle structure (Durable's granule rule); (cold) a line job and one or two reads | +| activation of an object with persisted singletons | plain reads, once per object | +| a `json(...)` column read at the current version | one `JSON.parse`; an older version adds `migrate` and one check | +| `rows` migration | about 15 µs per row, one commit per 100 rows | +| a keyed commit with no pins on the key | one `Map.get` | + +## Carry-over taken + +From `slices.md` "Slice 6a": facets only declare (1.1), documents, line and job runs on the object (1.2), keyed state +members (12), W12 and W7 (sections 3, 5, 8), retirement in Chord (1.5, 11). + +## Carry-over left + +- Slice 6b: persisted jobs, `StoreStorage` (Durable's record tables in the store: declared tables or internal), reads + at an older seq, `history: "rewindable"` export, `durableView`. +- Slice 7: in-memory `migrate` on reload; the routine handover fenced per routine invocation; holds unsubscribed at + seal and re-bound by member name; keyed members' versions per key; a native restart drains first (decided); `rebuild` + (key or column-type changes) if a case needs it. +- The sync layer: upload, acknowledgement and `outbox.trim`, epochs, pull-back, the split, snapshot reads for large + exports. +- Keyed members across links (Fable's design or selector holds) when a real case needs them. + +## Size + +About 3,000-3,500 source lines (the store core about 1,650, persisted documents about 400, tables and migrations +about 600, retirement and keyed watches about 200, keyed members about 300-350, the object/attempt/fence rework about +300), minus the facet code removed, plus the tests. diff --git a/packages/chord/docs/next/slice-6b.md b/packages/chord/docs/next/slice-6b.md new file mode 100644 index 000000000..9879fb104 --- /dev/null +++ b/packages/chord/docs/next/slice-6b.md @@ -0,0 +1,244 @@ +# Slice 6b: Durable and jobs on the store (spec) + +Status: **draft, open points marked "Decision needed"** (2026-10-27), written by the redesign session after reading +Durable's `Storage` interface (`packages/durable/src/types.ts`), its SQLite storage, schema and migrations, the storage +conformance suite and benchmark, `session/session.ts`, `session/transaction.ts`, `session/forks.ts`, and Chord v2's +store engine, applier, schema, commit step and jobs, in full. Durable's memory, JSONL and IndexedDB storages were not +read (they stay as they are, S15). + +The single source for building 6b, together with: `design.md` read in full; the decision log +(`packages/chord/docs/next/decisions.md`, later entries win), in particular "The store", S1, S2, S4, S10, S11, S13, S15, +W1, W12, W13, "Slice 6 first decisions", "Durable observers", "Durable's synchronous observers", "Text is plain UTF-8", +"Failures are watch calls", the "One vocabulary" entries, "The performance and memory bar"; `slice-6a.md` (what 6a +built); the feature map and the carry-over of `slices.md` (slice 6b); the root `AGENTS.md` "Chord v2 Code Style". + +6b puts Durable on the store (`StoreStorage`), follows Durable's views as Chord sources (`durableView`), persists jobs, +and extends the applier to every write kind. After 6b a worker has one store: Chord's objects and Durable's Session +commit through one queue, and a commit of either is one seq in one outbox. + +## What an app can do after this slice + +```ts +// the start code of a session's worker (p3 base code) +const store = await openStore(sqliteBackend(db), ctx); +const worker = createHost

({ address: Session({ id }), principal, store, onError }); + +// the core facet: the harness object opens Durable on the same store +host.serve(Harness, (_, self) => { + const durable = self.open(async (context) => + Durable.open(storeStorage(store, { owner: "session:harness" }), { …, paused: true }, context), + (h, _reason, context) => h.close(context)); + const view = self.follow(...) // Decision needed (D9): name and shape + return { … }; +}); + +// a persisted job: survives a crash +const ImportJob = defineJob("notes.import", { + version: 1, args: obj({ path: str() }), result: obj({ count: int() }), + state: ImportState, persist: { resume: "rerun" }, +}); +const imports = self.job(ImportJob, { run, finish }); +await imports.abort(id, ctx); // stored; Decision needed (D7): the result +``` + +## 1. `StoreStorage`: Durable's `Storage` on the store + +Durable's `Storage` (types.ts, "Atomic persistence boundary for Session records") implemented over a `ChordStore`, so +Durable's Session runs unchanged on it and passes Durable's storage conformance suite +(`createStorageConformance`), across reopen too. + +**Where it lives (D1, owner, 2026-10-27: "they are only used if durable is used"):** `StoreStorage` and +`durableView` belong to Durable: `packages/durable/src/next/`, private-only like `packages/chord/src/next`, importing +`@earendil-works/chord/next` (Chord's `./next` package export, private-only, returns for it). Durable already depends +on Chord and Chord imports nothing of Durable (S13); Durable's conformance suite runs in Durable's tests; the folder +goes upstream with Chord v2. + +### 1.1 Records + +Durable's four record tables (conversations, entries, tasks, submissions) and its ID namespace. + +> **Decision needed (D2): how Durable's records are stored.** +> (a) **Declared tables** in the store, written as ordinary `put` writes: `defineTable`s that `StoreStorage` declares +> for itself (SQL names `":

"`, e.g. `"session:harness:entries"`), each row the record's indexed columns +> (`id`, `conversation_id`, `kind`, `status`, …) plus the whole record in a `json(...)` column. The outbox, export, +> import and the applier handle them as they handle every table; Durable's layout changes later through +> `evolveTable` (Durable's own SQLite migrations, such as v2's `sleeping`, become table versions). Durable's +> secondary indexes map to declared indexes; the partial head index becomes an index on `(conversation_id, has_head, +> id)` (index columns are non-`opt`). Indexed strings that must stay lossless (`kind`, `requestId`, document kinds and +> keys) keep Durable's JSON encoding in their columns (decision "Text is plain UTF-8"). +> (b) **Internal tables** of `StoreStorage` with write kinds of their own (`durable.record`), which the outbox, +> export, import and the applier would each learn. More code, a second table mechanism. +> Proposal: (a). + +- **Writes:** every `StorageWrite` of a table record becomes a `put` of its row (Durable replaces complete records; + conversations and entries are immutable and written once). `record_ids` is a declared table too (`id`, + `record_type`), one `put` per new id (decision "Slice 6 first decisions" (3)). +- **Reads:** `conversation`, `entry` (with its `commitSeq`: a column), `findLatestHeadMarker`, `scanEntries` (both + orders, the fork chain walked as `SqliteStorage` does), `task`, `scanTasks`, `submission`, `scanSubmissions`, + `submissionByRequest`, `scanConversations`: queries over the declared tables, Durable's cursors (`{ after, order }`, + `scan.ts`) kept as they are (built from the last id, not from Chord's table cursors). +- **Validation before the step** (D4 below): what `SqliteStorage` checks inside its transaction + (`checkGlobalIds`, `prepareDocumentActions`, `checkDocumentActions`) runs before `store.commit`, with plain reads: + only Durable's Session writes these tables and it serializes its commits, so the reads see the state the step will + write on (the argument of S11). + +### 1.2 IDs + +> **Decision needed (D3): one id counter or two.** Durable mints ids before a commit (`mintId`), one namespace across +> conversations, entries, tasks, submissions and documents ("greater than every id minted since this storage was +> opened and every id stored, also across reopen", S10); its documents carry their id into the commit. Chord's store +> assigns its documents' ids inside the step from its own counter (`next_document_id`), and the engine's +> `chord_documents.id` is the primary key for both. Options: (a) **one counter per store**: `mintId` takes from the +> store's counter in memory (persisted as a high-water mark with every commit), Chord's step takes from it too; Chord's +> document ids are then no longer monotonic with seq (6a small item 11) and a Durable id minted but never committed +> is skipped; (b) **two counters**: Durable's from `record_ids` (`coalesce(max(id), 1) + 1` at open, decided), Chord's +> as today, and the engine's documents table keyed by `(namespace, id)` (Durable's document ids live in Durable's +> namespace). Proposal: (b): Durable keeps its id contract unchanged and Chord its step-assigned ids; the composite key +> costs nothing measurable and keeps the two adapters apart (S13: the engine is shared, the namespaces are not). + +### 1.3 Documents + +Durable's documents on the documents engine (S13): address `(namespace "durable", owner, kind, key?)`, owner the +scope as a string (`session`, `conversation:7`, `task:31`), kind and key JSON-encoded as Durable's indexed strings. +6a built `latest`; 6b adds what Durable needs: + +- **`rewindable` records** keep every revision and are not deleted at retirement: `retired_at` is set, revisions + stay (decision "Writes carry their versions" already defined the archive fields). +- **Reads at a point:** `findDocument(address, at)`, `document(id, at)` (the base at or below `at` plus the deltas up + to it, in one read transaction, S4), `scanDocuments(scope, at, kind?)`, with Durable's half-open lifetime + (`createdAt <= at < retiredAt`); a historical read of a `latest` (or session or task) record is a + `StorageRequestError` ("does not retain historical content"). +- **`document.copy`** (S11): `StoreStorage` reads the source at its point before the step and enters the step with a + plain create carrying the base; the conformance rules (source not changed in the copy batch, a matching record) are + checked there. +- **Stored version and `deltasSinceBase`** as `StoredDocument` reports them; Durable's records carry no fingerprint: + the engine's fingerprint column takes `""` for Durable records. +- **Export** carries every revision of a `rewindable` record, retired ones included, with its seq (format 1's + fields); import and the applier take them. + +### 1.4 Commits + +- `StoreStorage.commit(writes, adopt, ctx)`: validates (1.1), resolves copies (1.3), stages the writes (table `put`s, + document creates, changes and retirements) and calls `store.commit(staged, { admit: () => true, adopt: (seq) => + adopt(seq) }, ctx)`. The store's step runs `adopt` inside `host.batch.commit(...)` when the store has a host, so + Durable's observers mark Chord sources in the step (W1, "Slice 6 first decisions" (1)); `adopt` never throws + (Durable's rule). +- **Durable's seq is the store's seq** (gaps allowed: Chord's commits take seqs too). +- **Errors:** + +> **Decision needed (D4): what a refused Durable commit does to the store.** The store's rule: any backend error in a +> step is `storage_failed`, the store stops (Durable's own rule for its Session). Durable's conformance suite commits +> invalid batches on purpose ("ID 1 already belongs to conversation", "already has a current incarnation", "version +> transition requires a base", a mismatched copy) and expects the storage to go on. Proposal: those checks run before +> the step (1.1) and reject there, the store untouched; only a backend failure inside the step stops the store. In +> production the Session fails on any commit error either way (`GuardedStorage`), so the difference is only that the +> store, shared with Chord's objects, does not stop for a Durable bug caught by its checks. + +- `mintId`, `close(context)` (closes nothing of the store: the app owns it; later calls reject "closed"), reads after + a store failure reject as Durable's `GuardedStorage` expects. + +### 1.5 Statements per commit + +A target to measure, as 6a's: an entry append is the `record_ids` put, the entry put, `lastSeq` (plus the outbox); a +document delta is 6a's three statements. `SqliteStorage` does the same work today (`record_ids`, the row, metadata). + +## 2. `durableView`: Durable's views as Chord sources + +A Chord source over Durable's synchronous observers (`observeConversation`, `observeTaskGraph`, `observeDoc`; +decision "Durable's synchronous observers"): `advance(call, ctx)` records the call; on the store, `adopt` runs inside +the host's batch, so the source is marked in the step and its frame goes out in the flush with the rest of the commit +(one message per link per commit). On a Durable storage that is not the store (tests, other apps), calls arrive outside +any batch: the source opens one batch per Durable commit on the next microtask (carry-over). Durable's failures are +final: `end({ kind: "failed" })` ends the source's watches `failed`; `ended` ends them `ended`. A derive over two +Durable views of one commit sees both changes in one computation. + +> **Decision needed (D9): the API.** The carry-over names it `self.follow` (working name): `self.follow(observe)` in +> a factory, `observe` attaching an observer, returning a `Watchable` usable as a state member, a derive source and an +> `on` source, detached when the attempt ends. Options: (a) `self.follow((observer, ctx) => +> harness.observeConversation(id, observer, ctx, window))`; (b) a plain function `durableView(attach)` returning a +> native the object opens (`self.open`). Proposal: (a), named `self.follow`: the attempt owns the attachment as it +> owns a derive. + +## 3. Persisted jobs + +Design §15 "Persistence": `persist: { resume: "rerun" | "abandon" }`, only in `local` objects with empty params (P3, +at declaration); a run's record and state document in the owner's store; a recorded outcome resumes only `finish`; an +abort request is persisted; a restart limit; the job fingerprint stored with every record; the start trace carrier. + +- **Where records live:** + +> **Decision needed (D5).** (a) **Documents of the object:** each run's record is a keyed persisted document of the +> object (an internal definition per job, key the run id: `{ version, fingerprint, args, key?, phase, outcome?, +> abort?, restarts, trace? }`), its state document a second keyed document; both written in the transactions that +> already change them (start, state change, outcome, finish, block, drop), loaded at the object's first activation +> (all unsettled runs: `running` needs every one). (b) A table per job. Proposal: (a): the records are few, change +> in the same transactions as today, and reuse persisted documents (versions, checkpoints, export). + +- **Resume at activation:** a run recorded `finishing` resumes `finish` only; a run recorded `running` was cut off by + a crash: `rerun` invokes `run` again with the stored args and state, `abandon` records `failed { aborted, reason: + "restart" }` and finishes. +- **Restart limit** (aligned with Durable's `restarts`): + +> **Decision needed (D6).** Durable counts how many times in a row a crash cut a task off without the task changing +> its state since: open counts, a close does not, and at `restartLimit` the task ends visibly. Proposal for Chord: +> the same rule: at activation each run recorded `running` gets `restarts + 1` stored; a `ctx.state.change` resets it +> to 0 (progress); at 5 the run ends `failed { aborted, reason: "restart" }` with a `job_failed` report naming the +> limit (no `blocked { restart_limit }`: Durable dropped parking in #11139, and a blocked run would need a retry API); +> a clean close (`host.close`, a reload's native restart) records nothing: those runs end `aborted` (`stop`/`reload`) +> and do not resume. (Supersedes design §15's "then `blocked { restart_limit }`".) + +- **Abort** (owner, 2026-10-27: async and stored in 6b): + +> **Decision needed (D7): `job.abort(id, ctx)`'s result.** Cases: a running run (its signal fires, the abort is +> stored so a rerun after a crash does not run again but finishes `aborted`); a finishing run (its outcome is decided, +> nothing to do); a blocked run (dropped); an unknown or settled id. Durable's `abortTask` returns `"marked" | +> "terminal"` and throws for an unknown id. Proposal: `Promise<"marked" | "finishing" | "dropped">`, rejecting `gone` +> for an unknown or settled id (as `outcome(id)` does). + +- **Deferred state persistence:** + +> **Decision needed (D8).** Design §15: "a persisted state document may use the deferred persistence option (stored +> within about a second, flushed before the outcome is recorded)", as Durable's `CommitPersistence` `deferred`. +> Options: (a) per change: `ctx.state.change(fn, ctx, { persistence: { kind: "deferred", withinMs? } })`, as Durable's +> commit option; (b) per definition: `defineState(..., { persistDeferred: 1000 })`. Proposal: (a), Durable's shape: a +> deferred change is adopted and published at once, stored with the next immediate change of the document, before +> the outcome, at close, or after `withinMs` (default 1,000 ms); a crash loses it (acceptable for progress output). + +- **Fingerprint:** the job's fingerprint (args, result and state schemas) stored with every record; a record of the + same version with another fingerprint is `blocked { version }` (slice 7's family; the carry-over's "a job + fingerprint over args, result and state"). +- **Trace carrier:** the start's carrier (design §29) stored with the record; used in slice 9. + +## 4. The applier for batch apply + +6a's applier applies an archive's writes; 6b extends it to every `StoreWrite` kind (`alter`, `dropIndex`, `update`, +`updateWhere`, `delete`, `deleteWhere` with their recorded counts and versions, `document.change`, `document.retire`), +checked against the store's records (a trust boundary), one code path with import. A test replays a store's outbox +into an empty store and compares the two stores' exports. Applying a batch with a different count than recorded stops +applying (S17). + +## 5. Benchmarks and tests + +- **The gate** (decisions "The commit-step benchmark", "The performance and memory bar"): Durable's own storage + benchmark (`test/storage.bench.ts`: the write and read cases of `testing/storage-benchmark.ts`) on `StoreStorage` + against `SqliteStorage`, and Durable's `test/conversation-view-bench.ts` on both: each hot path within about 5%, + memory (allocation per op, retained after GC) no worse beyond noise; persisted job starts and state changes against + in-memory ones. +- **Tests:** Durable's storage conformance suite on `StoreStorage` (memory `:memory:` and across reopen); Durable's + Session and harness test files that take a storage, run on `StoreStorage`; durableView (one frame per commit, a + derive over two views, outside the store, failure and end); persisted jobs (rerun, abandon, a recorded outcome + resumes `finish`, restart counting and reset, the limit, abort stored, deferred state, fingerprints); the applier + (every kind, the outbox replay, refused counts); export and import with Durable records and `rewindable` history; + GC (observers detached with their attempt). +- **Examples:** `26-durable-on-the-store.ts`, `27-durable-views.ts`, `28-persisted-jobs.ts`. + +## Carry-over taken + +From `slices.md` "Slice 6b": persisted jobs (all of it), `StoreStorage` (ids, fork copies, `adopt` in the batch), +`durableView` (the batch on and off the store, `self.follow`), the applier for batch apply; `job.abort` async and +stored (decision of 2026-10-27). + +## Carry-over left + +Slice 7: a job fingerprint mismatch's `blocked { version }` resolution on reload; runs of jobs the code no longer +declares (with D5's records). Slice 9: the trace carrier's use. diff --git a/packages/chord/docs/next/slices.md b/packages/chord/docs/next/slices.md new file mode 100644 index 000000000..ea6d73093 --- /dev/null +++ b/packages/chord/docs/next/slices.md @@ -0,0 +1,269 @@ +# Chord v2: the slice plan + +Status: approved (owner, 2026-10-22); kept current. `design.md` states the rules; this file says which slice builds +what, and (section "Carry-over") what a slice deferred to a later one. + +## Why slices + +Building bottom-up made each layer guess the needs of callers that did not exist yet; L2 grew generic machinery +(nodes, sinks, deliveries, publications) that nobody could follow, and was deleted. A **slice** is the thinnest +end-to-end path through every layer a feature needs, so every API has a real caller from the first line. Later slices +extend earlier code; reworking it when a real need appears is expected, guessing ahead is not. + +Per slice (owner, 2026-10-22): a one-page spec written by this session (`slice-N.md`, including the carry-over +entries it takes), the owner's approval, a build by a fresh Opus 5.5 subagent, reviews of the uncommitted code by this +session and by the reviewers Astra and Fable (`v2-review-astra`, `v2-review-fable`), fixes by the builder, the owner's +review of the uncommitted diff, commit. Deferred work goes into the section "Carry-over" below, under the slice +that must do it, in the same change (owner, 2026-10-22; moved here from `carry-over.md` on 2026-10-24). Every slice that touches a hot path (links, the store, relays) also adds a light benchmark script +(`packages/chord/test/next/bench/`, run by hand) for timing, memory and wire size of realistic payloads, with results +in `benchmarks.md` (owner, 2026-10-22). Every slice adds one or more runnable +examples (`packages/chord/test/next/examples/NN-topic.ts`, numbered across slices, in the format of Durable's +`test/examples/`), each on one topic: for the owner to run and dig around in, and as references for Chord's users +(owner, 2026-10-22). Implementers and reviewers read +the slice spec, the feature map below and the whole of `design.md` in full, and check the work against all three; a +slice spec's section citations are for orientation, not a reading limit (owner, 2026-10-22). + +## What exists + +- **L1** (`src/next/contracts`, `src/next/data`): contracts, params and identity, argument checks, principal keys, + errors. `index.ts` for apps, `internal.ts` for Chord's own code. +- **Slices 1, 2, 3a, 3b, 4a, 4b, 5, 6a** (committed; 5 `4a5b45c45`, 6a 2026-10-27): `src/next/documents`, `state`, `host`, `jobs`, `link`, + `remote`, `store`; examples 01-25; benchmarks in `benchmarks.md`. +- **Chord's shared code** (upstream pi): `src/watch.ts` (`Watch`, `NativeWatch`, `WatchHandle`, `WatchEnd`, + `NativeWatchable`), Delta (`track`, `diffRevisions`, `applyImmutable`, encoders, `PendingOps`, `appendText`), + context, JSON, output, SQLite drivers. + +## The slices + +Order (owner, 2026-10-22: routines and jobs before the store and reload). + +| # | Slice | In short | +| --- | --- | --- | +| 1 | **One host, local objects** | host, facets, local objects, references, in-memory documents, transactions, derives, watches | +| 2 | **Failures and natives** | `self.open`, activation failures and recovery, backoff, the drain, timeouts | +| 3a | **Remote objects over one link** | remote contracts, principals, variants, a memory link, the message format, holds, calls, one message per commit (no relay, no drops) | +| 3b | **Surviving drops** | re-holds, parking, `ref.state`, `host.hold`, `ref.retry()`, `linger` | +| 4a | **Links and relays** | host kinds, placement, routes, two-sided links, relays and principals, IPC and WebSocket, flow control, heartbeat | +| 4b | **Authority and housekeeping** | revocation, cycles across hosts, `host.references`, `host.activity`, size limits, compression | +| 5 | **Routines and jobs** | `self.routines`, `self.job` | +| 6a | **The store** | the store (backends, tables, queries, writes, commit queue, outbox, documents engine, migrations, export), persisted documents, `self.table`, keyed watches and retirement | +| 6b | **Durable and jobs on the store** | persisted jobs, Durable's `StoreStorage`, the Chord source over Durable's views (`durableView`) | +| 7 | **Reload and rollout** | plugin loading, staging, swaps and restarts, revisions, `accepts` in use | +| 8 | **Registries and the harness** | registries, `self.writable`, Durable's harness, `chord(fn, context)` | +| 9 | **Telemetry** | spans, events, metrics across hosts | + +The playground (`apps/chord-playground`) moves to the new code once slice 4 exists. Round-3 cuts (C3-C16) and +carry-over entries are decided in the slice that first needs them. + +## Feature map + +Every feature of `design.md`, by section, assigned to the slice that builds it. A slice's spec covers every entry +assigned to it; **implementers and reviewers check the slice spec, this map and the whole design** (owner, 2026-10-22). A feature missing here is a planning bug: add it before the slice that needs it. "Doc" means a documented +pattern, no code. Decided cuts are noted where they apply. + +| § | Feature | Slice | +| --- | --- | --- | +| 3 | `createHost` with `telemetry`, `onError`; host without address | 1 | +| 3 | `createHost` with `principal`, `principalKey` (3a), `address` (4a), `revokedBy` (4b), the store (6a) | 3a, 4a, 4b, 6a | +| 3 | `load([{ name, facet }])` with static facets; start barrier (host-lifetime objects live before `load` resolves) | 1 (failed plugins reported: 2) | +| 3 | `connect`, `accept`, memory link pair, incarnations in the handshake, protocol-version handshake | 3a | +| 3 | host kinds, addresses, `route` (`authorize`, `resolve`), relayed holds and calls, re-resolve after a drop | 4a | +| 3 | IPC and WebSocket links, trusted links, the ack-clocked window per origin (K unacked state messages, pending queues, call window N per origin), relays with flow control per origin (`origin`, `closed`, senders FIFO), heartbeat | 4a | +| 3 | timeouts: `closeTimeout`, call drain, `readyTimeout` (C13: two host timeouts) | 2 | +| 3 | provider-side `linger` (C7: kept) | 3b | +| 3 | park deadline (= `readyTimeout`, C13) | 3b | +| 3 | `host.activity` (keep-alive counts) | 4b | +| 3 | `host.root(fn, …)` for code outside objects | 8 | +| 4 | principal type `P`, links stamp principals, principal keys in identity | 3a | +| 4 | revocation (`revokedBy`, `host.revoke`, propagation to children) | 4b | +| 5 | `local` types: params, members, checked arguments (Q15), results unchecked | 1 | +| 5 | version and fingerprint match at acquisition; reserved member names (`then`, `release`, `state`, `retry`) | 1 | +| 5 | `remote` types, variants chosen by the factory (`{ variant, members }`), identity with principal key, calls checked against the binding's variant | 3a | +| 5 | `accepts` mappings at runtime (old consumers), `contract_mismatch`/`updating` | 7 | +| 6 | lifecycle, one object per identity, lifetime from kind and params (Q-A), factory, activation, ending | 1 | +| 6 | failure causes, backoff per identity, dependency recovery (R18) | 2 | +| 6 | `retryOn` | 8 (first independently living source: registries) | +| 6 | cycle rejection on one host (Q5) | 1 | +| 6 | cycle rejection across hosts (activation chain in hold headers) | 4b | +| 6 | shared store with per-principal views; worker-wide native owners; natives following a relocation | doc (1, 3, 2) | +| 7 | `self.get`, `host.get`, hydrated references, members as watchables, `release` | 1 | +| 7 | per-reference watches (released with their reference) | 1 | +| 7 | `ctx.get` (call-scoped references) | 3a | +| 7 | `ref.state` (tagged), `ref.retry()`, `host.hold`, calls parking, failed references | 3b | +| 7 | `ref.variant` as discriminant, `RefIn` (C15: no `inVariant`/`expectVariant`) | 3a | +| 7 | `state: false` (C8: cut) | - | +| 7 | optional dependencies as registries | doc (8) | +| 8 | `ctx` with telemetry and signal | 1 | +| 8 | `ctx.principal`, `ctx.incarnation` (R20), `ctx.get` (3a); relayed incarnations (Q13, 4a) | 3a, 4a | +| 8 | `ctx.trace` (C11) | 9 | +| 8 | a job's `ctx.jobId`, `ctx.state` | 5 | +| 9 | `defineState`, singleton and keyed handles, `self.state`, in-memory documents (P2) | 1 | +| 9 | `self.tx`, `tx.draft`, `tx.retire`, drafts revoked (R12), no existence check (R13), lines per object | 1 | +| 9 | `handle.change`/`handle.retire` shorthands (C14: kept) | 1 | +| 9 | `initial()` checked once per definition (R8, D3) | 1 | +| 9 | `persist`, `checkpointWhen` (required for persisted documents), persisted only in local empty-params objects (P3), eviction | 6a | +| 9 | keyed watches (`handle.watch(key, ctx)`), `handle.at(key)`, retirement as a flag (decision "One retirement mechanism") | 6a | +| 9 | versions, `migrate`, schema fingerprints on records | 6a (in-memory migrate on reload: 7) | +| 9 | ~~facet documents, a served object's transaction spanning its facet's documents~~ (removed: C16 accepted, owner, 2026-10-26) | 1, removed in 6a | +| 9 | rules across owners, reconciling routines | doc (5) | +| 10 | the store: backends, tables, query runner, writes, commit queue, outbox, migrations, export/import | 6a | +| 11 | `Watchable`, watches (Chord's `Watch`), the batch (W1), reentrancy | 1 | +| 11 | `NativeWatchable`, `self.writable` | 8 | +| 11 | Durable's views as sources (`durableView` over Durable's `observeXXX`) | 6b | +| 12 | `self.derive`, failures per member, lazy when unwatched, `mapShared` | 1 | +| 12 | container-width benchmark (R24) | 9 | +| 13 | `self.open`, close reasons, order, restart patterns (R16) | 2 (restarts on reload: 7) | +| 14 | `self.on` (W2b), `self.onLive`, `sleep`, `untilAborted` (C4 decided); the handover per object on reload | 5 (handover: 7) | +| 15 | `defineJob`, `self.job`, intent/run/finish, state documents, dedupe (Q9), `busy`, `running`, blocked `finish` (C3 decided); persistence, resume, restart limit; `accepts` in use | 5 (persistence: 6b; `accepts` and reload: 7) | +| 16 | `defineFacet

`, facet as declarations only (documents and routines removed in 6a) | 1, 6a | +| 17 | registries: `defineRegistry`, contributions, `compose`, versions (Q8), use-site guards | 8 | +| 18 | `constant(value)`, start data as a core object (`SessionInfo`) | 1 (core facet pattern: 4a) | +| 18 | platform bindings as core objects | doc (8) | +| 19 | `host.references` (presence) | 4b | +| 20 | flows kept by their owner, the `steps` helper, the Questions service | doc (5); deferred decisions #9, #11 | +| 21 | loading plugins (loader D), staging, atomic commit, pure swaps, native restarts, rollout, dev loop | 7 | +| 22 | message format, header schema, unknown ids, one message per commit per link, consumers apply ops, plugin errors | 3a | +| 22 | relays, size limits per part, compression at the origin, adapted bindings get diffs | 4a (relays), 4b (limits, compression), 7 | +| 22 | trace carriers (R5) | 9 | +| 23 | HTTP, channels, chunked transfers | after slice 9 (with the first real proof) | +| 24 | Durable integration: harness object, `chord(fn, context)` (Q6), model and env registries | 8 | +| 24 | `StoreStorage` | 6b | +| 27 | versions table: contracts (L1 done), registries 8, documents 6, jobs 5, link protocol 3a | as listed | +| 29 | telemetry spans, events, metrics, carriers | 9 (cost counters: every slice) | +| 30 | cost guards per hop | every slice | + +## Size + +About 9-10k source lines in total, as estimated for the layers (owner: fine, on the order of Durable). + +## Carry-over + +Obligations a slice deliberately left to a later one, under the slice that must do them, with where they came from. +Rules (owner, 2026-10-22): whoever defers something (this session, a subagent, a reviewer) adds it here in the same +change; a slice's spec includes every entry of its section, and its build and review check them off; a done entry is +deleted (the slice's spec and code are the record). Moved here from `carry-over.md` (owner, 2026-10-24). + +### Any slice (protocol and checks) + +- **Bump `PROTOCOL_VERSION`** for any message or reason change once p3 has shipped (slice 3b added `retry`, `loading` + and `failed`'s `retryIn`; slice 4a parts, the list `bound`, `hello`'s `calls`, `ack`, `ping`, `pong`, `closed`, + `origin` and the hold's `address`, `principal` and `incarnation`; all without a bump: nothing is deployed). (Owner, + slice 3b.) +- **The park deadline's per-type override** is not built: a parked call waits the reference's `readyTimeout`. Build + it only if a real case needs it. (Slice 3b spec.) +- Failed-check paths are TypeBox's instance paths (ambiguous for keys containing `/` or `~`); documented, not fixed. + (L1.) +- No CBOR; revisit only if benchmarks show encoding as a bottleneck (slice 3a's and 4a's do not). (L2 decisions.) +- **One N on every host:** a relay forwards calls at once, so a worker announcing a smaller N than the relay's could + see more running calls of one origin than it accepts (a protocol error). Every host announces 256; per-link N, if + ever needed, must bound the relay's forwarding too. (Slice 4a rework.) +- **The Durable conversation view's size on re-hold** is a known cost until pi #11327 (windowed view) lands. (Slice 4.) +- **`map(bytes())` is refused at declaration** ("unsupported keyword `~refine`"): TypeBox's `Record` copies the bytes + schema's check as an enumerable key, which the snapshot walk refuses. An L1 bug found in slice 3a, not fixed yet. + +### Slice 6b: Durable and jobs on the store (from slice 5 and L3a-d) + +- **Persisted jobs** (from slice 5): `persist: { resume: "rerun" | "abandon" }` built (record and state document in + the owner's store; only in `local` objects with empty params, P3, enforced at declaration); outcomes and abort + requests persisted (a recorded outcome resumes only `finish`); the restart limit (five crash interruptions, + persisted, clean closes not counted; whether a state change resets the count; what a clean close records); deferred + persistence for state documents (flushed before the outcome is recorded); the `finish` failure count persisted; the + job fingerprint stored with every record; the run's start trace carrier (§29). +- **`StoreStorage`**: Durable's `Storage` on the store, passing Durable's storage conformance suite (memory and across + reopen); ids from a `record_ids` table (S10, W13); fork copies resolved before the step (S11); Durable's `adopt` run + inside the host's batch. +- **`durableView`**: a Chord source over Durable's synchronous observers (`observeConversation`, `observeTaskGraph`, + `observeDoc`; decision "Durable's synchronous observers"), marked inside Chord's batch in the store and grouped into + one batch per Durable commit on the next microtask outside it; `self.follow` (working name) and the Durable harness + carried in a registry entry. The task graph's `blocked` refresh arrives outside any commit. +- **The applier for batch apply** (slice 6a builder, 2026-10-27): 6a's applier (`store/applier.ts`) applies the + writes an archive holds (`table`, `put`, `index`, `document.create`), checked against the store's records; the + replica's batch apply extends it to every `StoreWrite` kind (`alter`, `dropIndex`, `update`/`updateWhere` with their + recorded counts and `v`, `delete`/`deleteWhere`, `document.change`, `document.retire`), one code path with import. +- Not carry-over any more: Durable's synchronous view-change hook is the upstream "Durable observers" change + (publication at adoption, `observeXXX`), which lands before 6a. + +### Slice 7: reload and rollout + +- **A native restart drains first** (owner, 2026-10-26): park new calls, await running calls (bounded by + `closeTimeout`), then close the natives and restart; a call outliving the drain is cut off by the seal fence. Running + job runs on a native restart: decide with the runs now owned by the object. +- **From slice 6a's second spec review** (decision "Second spec review: the majority"): remote holds and remote objects + leaving `live` (`Hold.moved` for bound and binding holds, `RemoteObject.failed` for a live object; not reachable in + 6a, P3); old table handles' version fence (old code issuing SQL for a dropped column); the routine handover fenced + per routine invocation; holds unsubscribed at seal and re-bound by member name; keyed members' versions mapped per + key; in-memory `migrate` wired to reload (6a builds the slot version check and the migrate); `rebuild` (key and + column-type changes) if a case needs it. +- **A run whose job the new code no longer declares** (slice 6a builder, 2026-10-27): 6a keeps it in its phase, + waiting, not invoked (unreachable without reload); slice 7 decides `blocked` with a reason (design §15's + `blocked { version }` family). +- **A job's new version on a live object** (slice 6a build review, 2026-10-27): 6a keeps the first declaration's job + definition in the object's records and refuses a later declaration with another version, so a host-lifetime object + (`local`, empty params) stays broken until the host restarts; slice 7 maps the records to the new version (or blocks + the old runs) on reload. + +- **Routines on reload** (from slice 5): the handover per object, bounded by `closeTimeout`; calls do not wait for it. +- **Jobs on reload** (from slice 5): `accepts` in use (args and result mapped forwards once per run; `blocked + { version }` without an adapter or for a newer version); when an older run's state document may migrate while its + old code still writes it, and an older run keeping its old handles valid until it settles; a native restart aborts + running runs (`aborted`, reason `reload`) and `finish` runs on the new code; a `finish` in flight across a reload; + (run records and state documents belong to the object: built in 6a); + blocked runs retried on reload; a job declaration removed while runs exist; a job fingerprint over args, result and + state (a schema change without a version bump); an `on` routine whose source failed and recovers after a provider + reload. +- **Instance ids on restart** (from slice 4b): a retry keeps an object's instance id; a reload that restarts an object + must give the new instance a new id, so an old chain never matches it. +- **`host.references` and removed types** (from slice 4b): a type's presence list lives from its first + `host.references(Type)` for the host's life; a reload that removes the type drops it. + +- **Holds across reloads** (from slice 3a): a hold's indexes belong to its bound version. A reload that still accepts + that version keeps the hold: a pure object re-subscribes the hold through the version's mappings and sends + `[["r", value]]` for each member whose value changed; a restarted object sends a fresh snapshot for the same hold (a + second `bound` on a bound hold, which 4a's consumer applies as a re-binding). A served hold whose object leaves + `live` while its `bound` waits for a slot goes back to pending (cut from 4a, where nothing leaves `live`; owner, + 2026-10-24: slice 7 builds and tests it in `Hold.moved` and `ProviderSide.unbind`). A reload + that drops the version ends its holds with `gone` (`contract_mismatch`) before the new code serves anything. A + consumer whose own code moves to a new version re-holds (new hold id) and releases the old. +- **Old-version calls** (L1): `checkArgs` returns `{ kind: "method"; args; result? }` or `{ kind: "call"; args; call }`; + invoke the handler or the provider's `call`, and map the result back with `result` for an old-version caller. +- **Derives on reload:** derives have no names (owner, 2026-10-22); exposed derives are matched by their member name, + internal ones are rebuilt; add matching only if a real need appears. (W1.) +- **Retrying broken failures:** reload retries a broken object as `ref.retry()` does (`ChordObject.retry`: the + backoff skipped, an attempt now; holders are told `loading`). (Slices 2, 3b.) + +### Slice 8: registries and the harness + +- **`host.root(fn, context, { principal? })`**: a call scope for code outside objects, built with its first users + (Durable hooks, `chord(fn, context)`); decide what an aborted, uncooperative `fn` means for its references (match + the handler and `CallContext` rules). (Slice 3b, owner.) +- **`retryOn`** retries a broken object as `ref.retry()` does. (Slice 2.) +- **`host.activity` on the facet's host** (`FacetHost`), if the harness or a core facet needs it; today only + `references` is there. (Slice 4b.) +- Registries recompose inside the owner's flush, never inside another owner's commit step; changing contributions are + native watchables (`self.writable`). (W4.) + +### Slice 9: telemetry + +- Spans for routine calls and job runs (from slice 5). + +- Round-3 cut to decide: C11 (`ctx.trace`). + +### When v2 goes upstream + +- `mapShared` lives in `src/next/state/derive.ts` and moves into Delta. (L2 decisions.) +- `withLinkedAbortSignal` lives in `src/next/host/ctx.ts` and moves next to `withAbortSignal` in `src/context` (pi + first): a call's context linked to long-lived abort signals until the call settles, instead of `AbortSignal.any`, + which keeps cleanup entries per call while its sources live before Node 24.16.0 (nodejs/node#62367). (Slice 4a.) +- The `./next` package export returns when the new code has an entry point (removed with the frozen-kernel move). +- A `./next/node` export for `nodeCompression` (`src/next-node/`), with `./next`. (Slice 4b.) + +### The sync layer (not a slice; when Radius sync is built) + +- Upload, acknowledgement and `store.outbox.trim`, epochs, pull-back, the split (S5, S7); a store opened with an + outbox (decision "The outbox is optional, internal and versioned") starts its sync from an archive. +- Snapshot reads for large exports on a second SQLite connection (S4), if exports get large. + +### Outside the slices + +- The playground's runtime tests fail on a stale `node_modules` copy of chord 1.3.0; relink with + `npm install --ignore-scripts` when the playground is used again. (Frozen-kernel move.) diff --git a/packages/chord/docs/next/store/book.md b/packages/chord/docs/next/store/book.md new file mode 100644 index 000000000..4f8a53f69 --- /dev/null +++ b/packages/chord/docs/next/store/book.md @@ -0,0 +1,665 @@ +--- +title: One Store: how a p3 session keeps, exports and syncs everything it owns +cover: One / Store +kicker: A design book · Chord v2 and p3 · 2026-10-21 +subtitle: Why Durable's transcript, Chord's documents and every plugin's own tables live in one transactional store, and how that store moves a session to another machine without coming apart +stats: + - 17 | review items decided (S1–S17) + - 1 | commit queue per store, one seq for every owner + - ~150 B | outbox write for renaming 10,000 rows (instead of ~40 MB) + - 0 | lines of plugin code a replica needs +note: | + Written for the owner from the discussion of 2026-10-21 and the store review by Fable and Astra. Every decision is in + the decision log (`packages/chord/docs/next/decisions.md`, entries "The store", "S1" to "S17", "Context last"); the + review is summarized in `apps/p2/docs/p3-reviews/store-review-summary.md`, bulk operations and migrations in + `store-bulk-migrations.md`, the earlier discussion with measurements in `session-data-scenarios.md`. Numbers were + measured on slayer with Node 24.4.1 and `node:sqlite` unless marked as estimates. Nothing here is built yet. Notes + marked *review* are points the reviewers raised that the design answers. +colophon: One Store · first edition · 2026-10-21 · based on the redesign decision log and the store review +--- + +# Before we begin {.preface} + +Mario works in session S1 on his laptop. The agent edits files, and a plugin called **file-history** remembers every +edit: which file, which conversation entry made it, the content before and after. It keeps those rows in its own table, +because it needs queries: every version of `src/app.ts`, every file the last turn touched, every edit under `a/`. When +Mario renames the directory `a/` to `b/`, ten thousand rows change. + +In the evening he wants to continue S1 on his desktop. Another day he exports S1 to a file and imports it into a +colleague's p3. Each time the other machine must end up with the conversation (Durable), the plugins' documents (Chord) +and the file-history table, and all three must agree: a row that says "entry 812 made this edit" must find entry 812, +the same entry 812. + +Copying files does not do that, and neither does syncing each storage on its own. This book explains the problem, the +solutions that were tried and dropped, and the design that came out: one transactional store per session, under +Durable, Chord and every plugin. + +## Eight things to know + +:::findings +1. **A session is self-contained.** Everything it owns lives in its store and its directory; it can be exported from + the session alone, imported elsewhere, synced and continued on another machine (P1). +2. **One store holds everything.** Durable's records (through an adapter), Chord's persisted documents and plugin + tables commit into one store, through one commit queue, numbered by one seq. +3. **Order is guaranteed where it is observable.** Disk, outbox and export follow the seq; in-memory adoption runs + inside the commit step (S1), so no reader ever sees seq 41 without seq 40. +4. **The outbox records what changed, not code.** Rows, column updates, bulk operations and table changes, written in + the same transaction as the data. A replica applies them without any plugin code. +5. **Tables are real tables with a small query language.** Keys and indexed columns are typed; text is UTF-8 and + compares by code point on every backend. +6. **Bulk writes stay small.** `updateWhere` records one operation instead of one row per change, and every replica runs + the same operation. +7. **Migrations run only where the code is.** Changes inside `json(...)` columns migrate on access; key and index changes run eagerly on the + driver and ship as ordinary writes. +8. **Losing control never rolls back.** A device that learns another one took over turns its copy into a separate + session. +::: + +## Words used throughout + +A **session** is one agent workspace: its conversations, tasks and plugin data. An **owner** is the code that writes a +piece of data: Durable's Session, a Chord object, a plugin's object. The **store** is the transactional store of one +session; a **backend** is what it runs on (SQLite, including in the browser as WebAssembly; tests use SQLite `:memory:`). A **table** is a set of +rows owned by one object. A **commit** is one atomic write; the **commit step** is the short, serialized part of a commit +in which the store checks, writes and numbers it; the **seq** is that number. The **outbox** holds a record of every +commit, written in the same backend transaction. The **driver** is the machine that writes a session; a **replica** is +a copy on another machine that applies the driver's records. **Adoption** is an owner updating its in-memory state to a +committed write. + +# A Session Is More Than Its Transcript {#problem lede="What has to travel with a session, and why separate storages cannot keep it consistent."} + +A p3 session holds three kinds of data written by three kinds of owners. Durable writes the transcript: conversations, +entries, tasks and its own documents. Chord objects write persisted documents: settings, drafts, plugin state. And a +plugin may need data shaped for queries, like file-history's edits. + +The owner's rule for p3 (**P1**) is that a session is self-contained. Everything of it lives in its storage plus its +directory; the server only mirrors it (the session catalog) and keeps instance data (shares, devices, settings). That +makes three things possible: export a session from its files alone, import it into another p3, and sync it to Radius +so another device can continue it. + +Durable alone already supports that shape: one storage, every change in an atomic commit, numbered by a seq. Armin's +sync design builds on it: a decorator captures each committed batch into an outbox, batches go to Radius, a replica +replays them. It breaks as soon as a session has a second storage. + +:::steps | One edit, two storages, synced separately ++++ On the laptop +The edit tool records the edit in file-history's own `history.sqlite` (row: path `x.ts`, entry 812). Durable commits +the tool's result. Both exist and agree. ++++ The history arrives first +The desktop has received `history.sqlite` up to the edit, but Durable's log only up to the commit before. The row says +entry 812 made the edit; the desktop has no entry 812. ++++ Mario takes over now +The laptop is closed; the desktop becomes the driver and writes its next entry, which gets id 812: a different +message. The row now points to the wrong entry, for good. +::: + +Each storage is consistent on its own. What is missing is one order across them: which change of `history.sqlite` +happened before which commit of the transcript. Every solution below is judged by whether it provides that order, +lets a replica work without the plugin's code, and keeps plugins able to model what they need. + +# What We Tried and Dropped {#dropped lede="The solutions discussed on the way, and the reason each one failed."} + +The first answers tried to avoid giving plugins storage at all. The later ones gave them storage and tried to sync it. + +| Idea | Why it was dropped | +| --- | --- | +| Plugins' own databases hold only derived data (indexes, caches), rebuilt after an import | too limiting: some plugins need their own write model and queries (owner) | +| Export plugin directories as opaque files | needs a stopped session; a changing database cannot sync in small pieces | +| Plugin export and import hooks | every plugin would build its own incremental log for sync | +| Plugin record logs in the session log, the plugin's database rebuilt from them by an `apply` function | the plugin must keep `apply` deterministic and every record version forever | +| Several storages, each with an outbox inside its own transaction, ordered by one shared position counter | works (prototype), but many outboxes and a "safe cut" rule; SQLite's row-change records are per row (a 10k-row rename is 259–418 KB); schema changes are not captured | +| A SQLite store that records the plugin's SQL statements, checked by a hash of their effect | SQLite only; arbitrary SQL can give different results on replicas (`random()`, unordered `INSERT … SELECT`, SQLite versions) | +| Raw SQL reads with controlled writes | ties every backend to SQLite and needs a guard against SQL that writes | +| An IndexedDB-shaped API only (keys, indexes, ranges) | too weak for real queries (owner) | +| Chord's documents inside Durable's existing storage | covers Chord, not plugin tables | + +The store design kept something from almost each: one order for everything (from the shared counter), recorded +operations instead of rows where possible (from the statement idea, made safe because the operations are the store's +own), and a small query language (from the IndexedDB shape, but over real tables). + +Inside the store, more choices were dropped during the review: + +| Choice | Why dropped | +| --- | --- | +| all rows of all tables in one generic table `rows(tbl, key, v, row JSON)` | the owner wants real tables; with table definitions in the outbox a replica needs no plugin code anyway | +| index entries maintained by the store in a separate table | an extra row per index per write; SQLite could no longer filter natively | +| ~~code point order (SQLite's native order) with lone surrogates replaced~~ | rejected at first, then chosen (owner, 2026-10-26, "Text is plain UTF-8, in code point order") | +| text only for equality, order only over integers (Durable's own approach) | ordered text, ranges and prefixes must be possible (owner) | +| JSON-field operands in bulk updates | they lean on SQLite's JSON functions; added only when a real case needs them | + +# The Store {#store lede="Layers, tables owned by objects, and the one transaction API."} + +The store sits between the backends and everything that persists data in a session. + +:::flow wide | The layers. Durable keeps its existing storages and gets one more, the adapter; Chord and plugins use the store directly. +row "Owners" blue +box durable "Durable Session" "`StoreStorage` adapter" +box chord "Chord objects" "persisted documents" +box plugin "plugin objects" "`self.table(...)`" +row "Store" accent +box docs "documents engine" "records, revisions, versions" accent +box tables "tables" "query language, writes" accent +box commit "commit step" "seq, outbox, adoption" accent +row "Backends" muted +box sqlite "SQLite" "Node, Durable Objects, browser (WebAssembly)" +box memory "SQLite :memory:" "tests" +arrow durable -> docs +arrow chord -> docs +arrow plugin -> tables +arrow docs -> commit +arrow tables -> commit +arrow commit -> sqlite +note "Durable's memory, SQLite and JSONL storages stay (S15), with only the adopt argument added (S1); StoreStorage joins them." +::: + +**A table belongs to one object**, like a document: only that object reads and writes it; its address is (host kind, +object type, table name), never a host instance such as the session id (W7), so a params change, a split or an import +never orphans data; other plugins see the data only through the object's +state and methods. + +```ts +const Edits = defineTable("edits", { + version: 2, + columns: { id: str(), path: str(), entry: int(), at: int(), meta: json(EditMeta, { version: 1 }) }, // flat + key: ["id"], + indexes: { by_path: ["path", "at"], by_entry: ["entry"] }, +}); + +host.serve(FileHistoryStore, (_, self) => { // a local object with empty params + const edits = self.table(Edits); + return { + record: (edit, ctx) => self.tx((tx) => { tx.put(edits, edit); }, ctx), + versions: ({ path }, ctx) => + edits.page(query(Edits).where((e) => eq(e.path, path)).orderBy((e) => e.at, "desc"), 20, ctx), + }; +}); +``` + +- A table's `columns` are a flat record: each column is a scalar (`str`, `int`, `bool`, `num`) or `json(...)`; nesting + exists only inside `json(...)`. Every column is a real SQLite column. Scalars can be filtered and ordered; keys and + indexed columns must be `str` or `int`. A `json(...)` column holds JSON text: typed when read, never queryable or + indexable (the type checker refuses it). +- Tables and documents of an object are written in the same transaction API, `self.tx(fn, ctx)` (S14). The context + comes last in every signature (owner's rule). +- Index **definitions** are recorded like other table changes: when the owning object starts on the driver, the store + creates or drops the difference between declared and existing indexes, and every replica applies it. Index contents + never travel; every backend maintains them from the rows. So every copy of the store is the same file. + +# Writing It Down {#commit lede="One commit queue per store, one seq for every owner, and adoption inside the step."} + +Two different things are serialized, and they must not be confused. A **line** serializes a whole transaction +callback: read, decide, write. Chord has one line per object, Durable one per Session. The **commit queue** serializes +only the final step: checks, writes, seq, the outbox's commit. It is short and there is one per store. + +:::sequence animate | Two Chord objects and a Durable turn commit at the same time. Their callbacks never wait for each other; only the short commit steps queue. +actor a "object A" "its own line" +actor b "object B" "its own line" +actor d "Durable Session" "its own line" +actor q "commit queue" "one per store" accent +a -> a: draft notes … +b -> b: draft drafts … +d -> d: phase writes … +a -> q: commit +q => q: seq 41: write, outbox, adopt A +b -> q: commit +q => q: seq 42: write, outbox, adopt B +d -> q: commit +q => q: seq 43: write, outbox, adopt Durable +::: + +Durable's seq is the store's seq. Durable sees gaps (other owners' commits in between), and that is harmless: every use +of a seq in Durable compares (`created_at <= at`, `seq > base`); nothing computes `seq + 1` to find a neighbour (checked +in the storages and the Session). + +## Adoption inside the step (S1) + +After a commit is on disk, each owner swaps its in-memory state to the committed version. If that happens later, in +the owner's own continuation, the order can flip: + +``` +disk: 40 file-history, 41 Durable +memory: Durable adopts 41 first (one await fewer) + a routine on "entry 812 committed" reads file-history → still at 39 +``` + +So adoption runs inside the commit step, synchronously, before the next commit starts. Durable's storage interface +carries it as a new required argument: + +```ts +commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise; +``` + +Every storage calls `adopt(seq)` once its commit is durable and before its next commit. The Session catches its own +adoption errors; a rejection after adoption fails the Session without discarding the adopted changes. + +## Adopt and mark in the step, flush right after (W1) + +The step itself stays short: it writes, adopts (pointer swaps of every changed value, including Durable's views the +commit touched) and **marks** (dependent derives dirty, ops appended by reference to each touched hold's batch and each +watch's queue). Right after the step, synchronously and before any other code runs, the committing owner **flushes**: +it recomputes the dirty derives someone holds or watches, encodes one part per touched hold into one message per link, +and queues watch callbacks. So a source and the derives over it always travel in one message; a throwing derive fails +only its own member, never the commit. + +``` +step 40: write · adopt notes · mark count dirty, notes' ops into tab 1's batch +flush 40: count = 3 → its ops join the batch · ONE message to tab 1 { notes: +1, count: 3 } +step 41: write · adopt Durable's view · mark lastReply … +``` + +Watch listeners run afterwards, each in order; their order across owners may differ from the seq, but every listener +reads a complete prefix of the log. + +## When a commit counts as done (S3) + +The step ends only on the backend's terminal success: SQLite's `COMMIT` returned (with an asynchronous backend, the +transaction's final completion event, not its individual requests). Any backend error in the step stops the store +(`storage_failed`, Durable's rule): an ambiguous `COMMIT` cannot be told from a failed statement, so nothing more is +written or published. + +:::review | Review: request success is not commit completion +Astra's trace: IndexedDB reports the puts of seq 40 successful, the store publishes 40, the transaction aborts later +(quota), 50 commits on top: memory has 40, disk does not. The rule above is the answer. +::: + +## Ids + +Each owner's id counter starts, when the store opens, above that owner's highest stored id: one index lookup per table +(S10). Chord's document ids are assigned in the commit step; for Durable's `mintId` (6b), an id handed out to a +transaction that never committed may be handed out again after a restart. Nobody can +notice, because nothing leaves a transaction before it commits (S2), so Durable's `mintId` contract is narrowed to say +exactly that. + +# Reading {#reading lede="A small query language, one text order on every backend, and cursors."} + +Queries are immutable data built in TanStack DB's style; S12 starts without joins, grouping and projections: + +```ts +query(Edits).where((e) => eq(e.path, "src/app.ts")).orderBy((e) => e.at, "desc") +query(Edits).where((e) => and(gte(e.at, from), lt(e.at, to))) +query(Edits).where((e) => prefix(e.path, "a/")) +``` + +`where` takes `eq ne lt lte gt gte inArray isNull prefix and or not`. `limit` and the cursor are arguments of the +runner, not part of the query. `count(q)` is the one aggregate; others are computed in JavaScript over indexed scans +until a real case needs more. + +## One text order + +Text compares by Unicode code point on every backend (owner, 2026-10-26; this replaced the first design, UTF-16 BLOBs +in JavaScript order, S16 and W8). That is the order SQLite compares UTF-8 text in natively, and Durable Objects, +sqlite-wasm, Postgres (`COLLATE "C"`) and MySQL (`utf8mb4_0900_bin`) do too. Every `str()` column is plain UTF-8 +`TEXT`: strings go to the driver and come back as they are, readable in `sqlite3` and in any other tool. + +It differs from JavaScript's `<` and `sort()` in one range only: characters U+E000-U+FFFF (fullwidth forms such as +`～`, private use, U+FFFD) against characters above U+FFFF (emoji): + +``` +JS sort(): "测试😀" < "测试（一）" +store order: "测试（一）" < "测试😀" +``` + +The store exports `compareText(a, b)` for plugins that sort store text themselves. Page with the store's cursors and use `prefix()`; a hand-built bound such as +`p + "\uffff"` misses every `p` followed by an emoji. + +UTF-8 cannot hold a lone surrogate, so every `str()` value written, and every query operand, goes through +`toWellFormed()`: a lone surrogate becomes U+FFFD, as `TextEncoder` does. A string cut in the middle of an emoji by +`slice()` is stored with U+FFFD in its place; cut text with `[...s].slice()` to keep it whole. A `str()` value may not +contain NUL (`invalid/args`); content belongs in `json(...)` columns, which escape both. There is no Unicode +normalization: "é" written as one code point and "e" plus a combining accent are two keys. + +| 100k path-like keys | UTF-16 BLOB (the first design) | UTF-8 TEXT | +| --- | --- | --- | +| insert, one transaction | 125 ms | 113 ms | +| read | 106 ms | 32 ms | +| bytes of an 18-character ASCII path | 36 | 18 | + +## Running queries + +A query is immutable data; the table handle runs it, and the context comes last: + +```ts +const edits = self.table(Edits); +const byPath = query(Edits).where((e) => prefix(e.path, "src/")).orderBy((e) => e.path); + +const page = await edits.page(byPath, 100, ctx, { after: cursor }); // { rows, next? } +for await (const row of edits.rows(byPath, ctx)) { … } // all pages +const e1 = await edits.get({ id: "e1" }, ctx); // keys are objects of the key columns + +await self.tx(async (tx) => { + for await (const row of tx.rows(edits, byPath)) { // committed rows, stable during the tx + tx.update(edits, { id: row.id }, { dir: dirname(row.path) }); // writes are synchronous, buffered + } + const again = await tx.get(edits, { id: "e1" }); // sees this transaction's own writes +}, ctx); +``` + +Inside a transaction, `tx.page` and `tx.rows` return committed rows: a stable view, because only the owning object +writes its tables and its transactions run one at a time. They do not show the transaction's own writes; `tx.get` does. +The rule in one sentence: queries inside a transaction see the state before it; your own writes become visible after +the commit. After an `updateWhere` or `deleteWhere` on a table, `tx.get` on that table throws `read_after_bulk` +(W10): the store would need a second evaluator to know what a bulk operation did to one row. + +## Cursors and consistent reads + +Every index ends with the primary key, so order is total and cursors are stable. A cursor is opaque, carries its +direction and its last key as logical values, is bound to its query's order, and is refused by another query or the +other direction. + +Reads outside a transaction see committed state, one query at a time. A read that needs several queries, like a +document assembled from its base and later revisions outside its owner's line (Durable's reads, 6b), runs in one backend +read transaction (S4); Chord's own loads run on the object's line and need none. Two separately +awaited reads in object code are not a snapshot, and the guide says so. + +# Documents {#documents lede="One documents engine in the store, used by Durable's adapter and by Chord."} + +Durable and Chord both persist documents the same way: a record per incarnation, revisions made of a base and later +deltas, versions, `checkpointWhen`, one live document per address. The store implements it once (S13): + +```ts +type DocAddress = { namespace: string; owner: string; kind: string; key?: string }; + +interface Documents { // reads, committed state + find(address: DocAddress, at: At, context: Context): Promise; + load(id: number, at: At, context: Context): Promise; // base + deltas, applied once + list(query: DocListQuery, limit: number, cursor: Cursor | undefined, context: Context): Promise>; +} +interface DocumentWrites { // inside the commit step + create(record: NewDocRecord, base: BaseContent): number; + change(id: number, content: DocContent): void; + retire(id: number): void; +} +``` + +The `owner` string is opaque to the engine; each adapter fills it: + +| | namespace | owner | kind | key | +| --- | --- | --- | --- | --- | +| Durable session document | `pi-durable` | `session` | `pi.settings` | | +| Durable conversation document | `pi-durable` | `conversation:7` | `pi.provider` | | +| Durable task document | `pi-durable` | `task:31` | `pi.codemode` | | +| Chord persisted document | `chord` | `session:notes.store` | `notes` | `n-42` | + +What the owner string means (a conversation's documents fork and rewind with it, a task's are retired with the task) +stays in the adapter. The engine keeps nothing in memory; Durable's Session and Chord's objects keep their Delta +trackers, cache, pin and evict exactly as Durable does today. A fork's document copies are reconstructed by +`StoreStorage` before it enters the commit step (S11), so the step never replays revisions while every owner waits; +the outbox only ever carries the resulting documents. + +# Writing Many Rows {#bulk lede="Bulk operations recorded as operations, and migrations that run only where the code is."} + +A full-row put per changed row makes the rename of `a/` to `b/` over ten thousand 4 KB rows cost about 40 MB in the +outbox. The store's writes are therefore five: + +| Write | Records | +| --- | --- | +| `tx.put(table, row)` | the whole row | +| `tx.update(table, key, set)` | the changed columns of one row | +| `tx.delete(table, key)` | the key | +| `tx.updateWhere(table, where, set)` | one operation, plus the affected row count | +| `tx.deleteWhere(table, where)` | one operation, plus the affected row count | + +```ts +await self.tx((tx) => { + tx.updateWhere(edits, (e) => prefix(e.path, "a/"), (e) => ({ path: replacePrefix(e.path, "a/", "b/") })); +}, ctx); +``` + +```sql +-- on SQLite: one statement; path is UTF-8 TEXT, "a/" is 2 code points +UPDATE "session:file-history.store:edits" SET path = :to || substr(path, 3) WHERE path >= :lo AND path < :hi +``` + +``` +outbox, seq 57 (about 150 bytes): +{ kind: "updateWhere", table: "session:file-history.store:edits", where: { prefix: ["path", "a/"] }, + set: { path: { replacePrefix: ["a/", "b/"] } }, count: 10000 } +``` + +A replica runs the same operation, without plugin code, and compares the number of affected rows. A different number +means divergence: it stops applying that store and restores it from a snapshot. The `set` expressions touch only real +columns (plain values, or column references in a callback): `add`, `sub`, `max`, `min`, `concat`, `replacePrefix`, +`coalesce`, `when`. They are the store's +own, specified once and checked on every backend by one conformance suite, which is why there are no SQL subqueries, +joins, case folding or floats: each of those behaves differently somewhere. + +## Migrations (S17) + +Adding, renaming or dropping a column is a recorded table change (`alter`); a new column gets its default without a +bulk rewrite. A change of the shape **inside a `json(...)` column** (a field added, renamed, restructured) **migrates on +access** (W11): each `json(...)` column has its own version and a value-only migration, +`json(MetaV2, { version: 2, migrate: (old, from) => … })`, stored with its value; reading an older value migrates it, +writing the column stores the new shape and version. Nothing is rewritten in bulk and indexes are unaffected. The row's +`$v` counts only eager steps (below); `update`, `updateWhere` and `deleteWhere` never set it. Old code refuses a newer +table version or a newer JSON column version. + +A change to a key or an indexed column must be **eager**, because an index over a column filled only on access would +silently miss rows. It runs when the owning object starts on the driver, before the object goes live: + +```ts +const EditsV4 = evolveTable(EditsV3, (t) => + t.add("dir", str({ default: "" })).rows((row) => ({ dir: dirname(row.path) })).index("by_dir", ["dir"])); +const EditsV5 = evolveTable(EditsV4, (t) => + t.add("size", int({ default: 0 })).update({ where: (e) => lt(e.size, 0), set: { size: 0 } })); +``` + +Each version is its own materialized definition (as contracts' `evolve`), so a store at version 3 replays 4 and then 5, +each with exactly its own columns. + +:::steps | Migration 4 on the driver ++++ The table change +Commit 80 records `alter: add column dir` and marks table version 4 as `migrating`. Every replica adds the column +from the record. ++++ The scan +The store reads rows with `$v < 4` in key order, 100 at a time (about 1 ms per commit step), applies `rows` to each and writes the changed columns +plus `$v = 4` in the same write. One commit per chunk, so other owners' commits interleave and the commit step stays +short. (A full-row `put` also sets `$v`; partial and bulk writes never do.) ++++ Ready +The last chunk marks version 4 `ready`. The store creates the `byDir` index and records it, and `FileHistoryStore` +goes live. ++++ After a crash in the middle +The next start finds `migrating`, scans again from the first key and skips rows with `$v >= 4`. Each row is +transformed exactly once, even when the transform is not idempotent. A key change is never a `rows` step: it is a +recorded `rebuild` that every replica performs itself. +::: + +A replica, with or without the plugin, applies the recorded column changes and the column updates; it never runs a +migration itself. Chord documents keep their own per-document `migrate` on access, and +Durable versions its records itself. + +# The Easy Parts {#easy lede="Blobs and derived data: why they need none of the store's ordering machinery."} + +Two kinds of session data stay outside the store's tables, and both are easy for the same reason: nothing about them +ever changes in place. A blob is final once written; derived data has no data of its own. + +## Blobs + +Artifact versions, uploads and sent files are **blobs**: files in the session directory, named by the hash of their +content (`blobs/sha256/9f/9f3a…`), never changed. They are app layer, not store: any process may write one, because a +file named by its content cannot conflict with another writer. The order is the whole trick: the blob is written first, +then the row or document that names it. + +:::sequence animate | Saving artifact "chart" version 3. The blob exists before anything refers to it. +actor tool "artifact tool" +actor blobs "blobs/" muted +actor store "store" blue +tool -> blobs: write sha256:9f3a… (tmp, fsync, rename) +tool => store: seq 41: chart.versions[2] = { blob: "sha256:9f3a…" } +tool => store: seq 42: entry 812 "chart v3 saved" (Durable) +::: + +- **Export** copies `blobs/` together with the store snapshot. Every blob a record up to the export's `lastSeq` names was + written before that record, so it is in the directory. **Import** copies them back and checks each hash. +- **Sync:** before uploading a batch, the sync layer uploads every blob file that is present and not yet uploaded. No + reference tracking is needed: a blob is always written before the commit that names it, so a replica never sees a + hash whose file is missing. +- **Deleting:** no blob is deleted before its session is deleted. A blob that nothing references any more (a crash + between the write and the commit, a retired document) only costs disk; a garbage collector would have to know every + reference in every table. + +## Derived data + +A full-text index, a cache, a plugin's own search database: **derived data** can be recomputed from the store and the +blobs, so it is never exported or synced. Its owner keeps it current with a routine that compares it with the store. +The artifacts plugin's full-text index in its own SQLite file: + +```ts +host.serve(ArtifactStore, (_, self) => { + const artifacts = self.state(Artifact); // keyed documents in the store + const blobs = self.get(SessionBlobs, {}); + const index = self.open(async () => openIndex(join(dir, "derived/artifacts/index.sqlite")), (db) => db.close()); + + self.onLive(async (ctx) => { // the index repairs itself + for await (const name of artifacts.keys(ctx)) { + const a = await artifacts.get(name, ctx); + if (a && index.value.versions(name) !== a.versions.length) + index.value.put(name, a.versions.length, await blobs.read(a.versions.at(-1)!.blob, ctx)); + } + }); + … +}); +``` + +:::steps | One routine, three situations ++++ Normal save +`save` writes the blob, commits the document, then updates the index. The routine finds nothing to do. ++++ Crash after the commit +The process dies between the commit and the index update. The document says version 3, the index still has version 2. +On the next start the routine sees the difference, reads the blob of version 3 and indexes it. ++++ Import or sync +On the desktop `derived/artifacts/` does not exist. The routine finds every artifact missing and indexes them all from +the documents and blobs that came with the session. +::: + +This routine visits every artifact on each start: fine for hundreds, slow for millions. A large derived store keeps +the last seq it applied and reads only what changed after it. + +# Moving a Session {#sync lede="Outbox, replicas (no compaction yet), takeover, export and import."} + +With an outbox (optional: a store opened with `{ outbox: true }`), every commit's writes go into it, written in the +same backend transaction as its data, so a crash keeps both or neither. Sync is a layer above the store; it follows Armin's transport and fencing with the rules the review added. + +- **Uploading:** the driver may upload batches in parallel, resubmits each until Radius acknowledges it, and trims a + batch from its outbox only after its acknowledgement (S5). +- **Applying:** every batch carries `from` and `to`. A replica applies a batch only when its `from` equals the + replica's `lastSeq` (the seq of the last commit it applied), as a whole, in one backend transaction together with + its new `lastSeq`. A later batch + waits for a missing one. A crash leaves all of a batch or none of it. +- **No compaction at first (W9):** batches carry the ordered operations exactly as committed. "Keep the last write per + row" would lose rows built from `put` plus partial `update`s; a later compaction must merge those and treat bulk + operations as barriers. +- **After a power loss** the driver's local `lastSeq` may be behind what Radius has; it first pulls back its own uploaded + batches, then continues (S3). +- **Blobs** go up before the batch that may name them: every blob file present and not yet uploaded (chapter VIII). + +## Losing the driver role (S7) + +:::sequence animate | The laptop learns the desktop took over. Nothing is rolled back. +actor laptop "laptop" "driver, epoch 3" +actor radius "Radius" blue +actor desktop "desktop" +desktop -> radius: take over S1 +radius => desktop: epoch 4 +laptop -> laptop: commits 40, 41 locally (a routine, a late tool result) +laptop -> radius: upload 40–41 (epoch 3) +radius --> laptop: refused: epoch 4 +laptop => laptop: the copy becomes session S1′ (new id) +note laptop..desktop: S1 continues on the desktop; S1′ on the laptop; they never sync again +::: + +Rolling back is not possible: tools ran, files were written, natives and derived data followed the tail. The laptop's +copy becomes a separate session with a new id and keeps running; commits in flight land in it. The catalog shows both, +the split-off copy marked with its origin. + +## Export and import (S4) + +An export holds the commit queue while it reads, so the archive is the state after one seq, its `lastSeq`: Durable's records, Chord's +documents and every plugin table, including tables of plugins that are not installed, plus the session directory's +blobs (chapter VIII). The archive is written in the outbox's own operations (a manifest, `table`, `index`, `put` and `document.create` +records, an `end` record), as one batch from 0 to its `lastSeq`, so an import is applied by the same code as a +replica's batches. An import restores the archive into a fresh store in one transaction; the server then mirrors the +session into its catalog. The encoding is pluggable (the store yields and takes records); plain JSONL ships first. +Derived data (full-text indexes, caches) is never exported; it is rebuilt where it is needed. + +# Rules for Plugin Authors {#rules lede="What the store does not do for you."} + +**Nothing leaves a transaction before it commits (S2).** Ids are handed out inside the transaction that creates them. +Reference what is committed: the file-history plugin references the tool-call entry, which is committed before the tool +runs, never the tool-result entry, which does not exist yet. + +**Work that finishes out of order is yours (S8).** A routine that starts a slow job per change can let an old result +overwrite a newer one. The handler calls of one routine never overlap, so either chain through the routine or drop +stale results inside the destination's transaction: + +```ts +self.on(conversation.entries, async ({ value: entries }, ctx) => { + const run = await summarize.start({ upTo: entries.at(-1)!.id }, ctx); + await run.outcome(ctx); // the next handler waits for this run +}); +``` + +**Text compares by code point.** Do not sort keys yourself to page through a table; use the store's cursors, or +`compareText`. Normalize text yourself if two spellings must be one key. + +**Bulk changes are operations.** Prefer `updateWhere` and `deleteWhere` over loops of puts; a loop records every row. + +# What's Next {.epilogue lede="What is decided, and the order in which it gets built."} + +The direction and all seventeen review items are decided (decision log, 2026-10-21). The store is new code; the parts +outside it are small changes in pi: + +- **pi, one change:** `Storage.commit(writes, adopt, context)` with every storage calling `adopt` (S1), the + Session's handling of rejections after adoption, the narrowed `mintId` comment (S10), `traceIn(options, fn, + context)` (context last), Durable's `WatchHandle`/`WatchEnd` moved into Chord with generic end reasons and explicit + retirement (W2), and Durable's state getters and the `replicatedState` family removed (W14). +- **The store** in `packages/chord/src/next`: commit step and outbox, tables and the query language, the SQLite + backend with the text encoding, the documents engine, bulk operations and migrations, export and import; + about 1,650 lines by Fable's estimate, plus its tests. +- **`StoreStorage`**, passing Durable's existing storage conformance suite. +- **The sync layer** (upload, apply, pull-back, epochs, splitting a session) on top, when Radius sync is built. + +Open, to settle while building: how a split-off session gets its new id (store metadata, directory, catalog), and a Postgres backend once a serverless target needs one (W14). + +# Decisions {.appendix} + +| # | Decision | +| --- | --- | +| The store | one transactional store per session under Durable (adapter), Chord documents and plugin tables | +| S1 | adoption inside the commit step; `Storage.commit(writes, adopt, context)` | +| S2 | nothing leaves a transaction before it commits; no mechanism | +| S3 | a commit is done on the backend's terminal success; ambiguous outcomes stop the store; pull-back after restart | +| S4 | export holds the commit queue; multi-query reads in one read transaction; no global snapshots | +| S5 | parallel uploads, resubmit until acknowledged, trim after acknowledgement; apply whole batches in order | +| S6 | (superseded by W9) | +| S7 | losing the driver role splits the session; no rollback | +| S8 | out-of-order work is an app concern; routine handlers never overlap | +| S9 | `lastSeq` (the seq of the last commit) is persisted with every commit | +| S10 | id counters derived at open; `mintId` contract narrowed | +| S11 | fork copies reconstructed by `StoreStorage` before the commit step | +| S12 | the query language starts without joins, grouping and projections | +| S13 | one documents engine in the store, opaque owner | +| S14 | one transaction API, `self.tx(fn, ctx)` | +| S15 | Durable's existing storages are kept | +| S16, text | real tables; text as plain UTF-8 in code point order (replaced W8's UTF-16 BLOBs) | +| S17, W11 | five writes, bulk operations recorded with a row count; JSON columns migrate on access with their own version; eager steps by scan with row `v` | +| W1 | adopt and mark in the commit step, one synchronous flush right after (derives, wire, watches) | +| W6, W7 | workers connect before loading; store addresses carry the host kind, never an instance | +| W9, W10 | no compaction at first; no `tx.get` on a table after a bulk operation in the same transaction | +| W14 | no IndexedDB backend (SQLite in WebAssembly in the browser); no `replicatedState` | +| index definitions | recorded in the outbox and applied by every replica; contents never travel | +| blobs, derived data | blobs are app-layer files named by hash, written before what names them, uploaded before each batch, deleted only with the session; derived data is rebuilt, never shipped | + +# Glossary {.appendix} + +**Adoption:** an owner updating its in-memory state to a committed write. **Blob:** a file in the session directory +named by the hash of its content, never changed. **Derived data:** data recomputed from the store and blobs, never +shipped. **Backend:** what the store runs on (SQLite, +memory). **Write:** one change of the store (`put`, `update`, `updateWhere`, `table`, `document.create`, …), the same in the +outbox and in an archive. **Commit** (in the outbox): the writes of one seq. **Batch:** the commits from `from` to `to`, +uploaded and applied as a whole. **Archive:** a manifest, the writes from 0 to `lastSeq`, and an `end`. +**Commit step:** the short, serialized part of a commit in which the store checks, writes, numbers and records it, and +owners adopt. **Driver:** the machine that writes a session. **Line:** the serialization of one owner's transactions +(per Chord object, per Durable Session). **Outbox:** the record of every commit, written in the same transaction. +**Owner:** the code that writes a piece of data. **Replica:** a copy of a session that applies the driver's batches. +**Seq:** the number of a commit, one sequence per store. **Store:** the transactional store of one session. +**Table:** a set of rows owned by one object, with a flat record of typed columns (scalars or `json(...)`). diff --git a/packages/chord/docs/next/store/index.html b/packages/chord/docs/next/store/index.html new file mode 100644 index 000000000..067a77772 --- /dev/null +++ b/packages/chord/docs/next/store/index.html @@ -0,0 +1,861 @@ + + +One Store: how a p3 session keeps, exports and syncs everything it owns + + + + +

+
+
A design book · Chord v2 and p3 · 2026-10-21
+

One
Store

+

Why Durable's transcript, Chord's documents and every plugin's own tables live in one transactional store, and how that store moves a session to another machine without coming apart

+
17review items decided (S1–S17)
1commit queue per store, one seq for every owner
~150 Boutbox record for renaming 10,000 rows (instead of ~40 MB)
0lines of plugin code a replica needs
+

Written for the owner from the discussion of 2026-10-21 and the store review by Fable and Astra. Every decision is in +the decision log (packages/chord/docs/next/decisions.md, entries "The store", "S1" to "S17", "Context last"); the +review is summarized in apps/p2/docs/p3-reviews/store-review-summary.md, bulk operations and migrations in +store-bulk-migrations.md, the earlier discussion with measurements in session-data-scenarios.md. Numbers were +measured on slayer with Node 24.4.1 and node:sqlite unless marked as estimates. Nothing here is built yet. Notes +marked review are points the reviewers raised that the design answers.

+
+
+ + +
+

Before we begin

+

Mario works in session S1 on his laptop. The agent edits files, and a plugin called file-history remembers every +edit: which file, which conversation entry made it, the content before and after. It keeps those rows in its own table, +because it needs queries: every version of src/app.ts, every file the last turn touched, every edit under a/. When +Mario renames the directory a/ to b/, ten thousand rows change.

+

In the evening he wants to continue S1 on his desktop. Another day he exports S1 to a file and imports it into a +colleague's p3. Each time the other machine must end up with the conversation (Durable), the plugins' documents (Chord) +and the file-history table, and all three must agree: a row that says "entry 812 made this edit" must find entry 812, +the same entry 812.

+

Copying files does not do that, and neither does syncing each storage on its own. This book explains the problem, the +solutions that were tried and dropped, and the design that came out: one transactional store per session, under +Durable, Chord and every plugin.

+

Eight things to know

+
    +
  1. A session is self-contained. Everything it owns lives in its store and its directory; it can be exported from +the session alone, imported elsewhere, synced and continued on another machine (P1).
    2. +
  2. One store holds everything. Durable's records (through an adapter), Chord's persisted documents and plugin +tables commit into one store, through one commit queue, numbered by one seq.
    3. +
  3. Order is guaranteed where it is observable. Disk, outbox and export follow the seq; in-memory adoption runs +inside the commit step (S1), so no reader ever sees seq 41 without seq 40.
    4. +
  4. The outbox records what changed, not code. Rows, column updates, bulk operations and table changes, written in +the same transaction as the data. A replica applies them without any plugin code.
    5. +
  5. Tables are real tables with a small query language. Keys and indexed columns are typed; text is UTF-8 and +compares by code point on every backend.
    6. +
  6. Bulk writes stay small. updateWhere records one operation instead of one row per change, and every replica runs +the same operation.
    7. +
  7. Migrations run only where the code is. Changes inside json(...) columns migrate on access; key and index changes run eagerly on the +driver and ship as ordinary writes.
    8. +
  8. Losing control never rolls back. A device that learns another one took over turns its copy into a separate +session.
    9. +
+

Words used throughout

+

A session is one agent workspace: its conversations, tasks and plugin data. An owner is the code that writes a +piece of data: Durable's Session, a Chord object, a plugin's object. The store is the transactional store of one +session; a backend is what it runs on (SQLite, including in the browser as WebAssembly, or a memory backend for tests). A table is a set of +rows owned by one object. A commit is one atomic write; the commit step is the short, serialized part of a commit +in which the store checks, writes and numbers it; the seq is that number. The outbox holds a record of every +commit, written in the same backend transaction. The driver is the machine that writes a session; a replica is +a copy on another machine that applies the driver's records. Adoption is an owner updating its in-memory state to a +committed write.

+
+ +
I

A Session Is More Than Its Transcript

What has to travel with a session, and why separate storages cannot keep it consistent.

+

A p3 session holds three kinds of data written by three kinds of owners. Durable writes the transcript: conversations, +entries, tasks and its own documents. Chord objects write persisted documents: settings, drafts, plugin state. And a +plugin may need data shaped for queries, like file-history's edits.

+

The owner's rule for p3 (P1) is that a session is self-contained. Everything of it lives in its storage plus its +directory; the server only mirrors it (the session catalog) and keeps instance data (shares, devices, settings). That +makes three things possible: export a session from its files alone, import it into another p3, and sync it to Radius +so another device can continue it.

+

Durable alone already supports that shape: one storage, every change in an atomic commit, numbered by a seq. Armin's +sync design builds on it: a decorator captures each committed batch into an outbox, batches go to Radius, a replica +replays them. It breaks as soon as a session has a second storage.

+
One edit, two storages, synced separately

On the laptop

The edit tool records the edit in file-history's own history.sqlite (row: path x.ts, entry 812). Durable commits +the tool's result. Both exist and agree.

+

The history arrives first

The desktop has received history.sqlite up to the edit, but Durable's log only up to the commit before. The row says +entry 812 made the edit; the desktop has no entry 812.

+

Mario takes over now

The laptop is closed; the desktop becomes the driver and writes its next entry, which gets id 812: a different +message. The row now points to the wrong entry, for good.

+
+

Each storage is consistent on its own. What is missing is one order across them: which change of history.sqlite +happened before which commit of the transcript. Every solution below is judged by whether it provides that order, +lets a replica work without the plugin's code, and keeps plugins able to model what they need.

+
+ +
II

What We Tried and Dropped

The solutions discussed on the way, and the reason each one failed.

+

The first answers tried to avoid giving plugins storage at all. The later ones gave them storage and tried to sync it.

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
IdeaWhy it was dropped
Plugins' own databases hold only derived data (indexes, caches), rebuilt after an importtoo limiting: some plugins need their own write model and queries (owner)
Export plugin directories as opaque filesneeds a stopped session; a changing database cannot sync in small pieces
Plugin export and import hooksevery plugin would build its own incremental log for sync
Plugin record logs in the session log, the plugin's database rebuilt from them by an apply functionthe plugin must keep apply deterministic and every record version forever
Several storages, each with an outbox inside its own transaction, ordered by one shared position counterworks (prototype), but many outboxes and a "safe cut" rule; SQLite's row-change records are per row (a 10k-row rename is 259–418 KB); schema changes are not captured
A SQLite store that records the plugin's SQL statements, checked by a hash of their effectSQLite only; arbitrary SQL can give different results on replicas (random(), unordered INSERT … SELECT, SQLite versions)
Raw SQL reads with controlled writesties every backend to SQLite and needs a guard against SQL that writes
An IndexedDB-shaped API only (keys, indexes, ranges)too weak for real queries (owner)
Chord's documents inside Durable's existing storagecovers Chord, not plugin tables
+

The store design kept something from almost each: one order for everything (from the shared counter), recorded +operations instead of rows where possible (from the statement idea, made safe because the operations are the store's +own), and a small query language (from the IndexedDB shape, but over real tables).

+

Inside the store, more choices were dropped during the review:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
ChoiceWhy dropped
all rows of all tables in one generic table rows(tbl, key, v, row JSON)the owner wants real tables; with table definitions in the outbox a replica needs no plugin code anyway
index entries maintained by the store in a separate tablean extra row per index per write; SQLite could no longer filter natively
code point order (SQLite's native order) with lone surrogates replacedlossy ("\uD800" and "\uFFFD" collide) and needs a comparator in JS
text only for equality, order only over integers (Durable's own approach)ordered text, ranges and prefixes must be possible (owner)
JSON-field operands in bulk updatesthey lean on SQLite's JSON functions; added only when a real case needs them
+ + +
III

The Store

Layers, tables owned by objects, and the one transaction API.

+

The store sits between the backends and everything that persists data in a session.

+
OWNERSSTOREBACKENDSDurable Session`StoreStorage` adapterChord objectspersisted documentsplugin objectsself.table(...)documents enginerecords, revisions, versionstablesquery language, writescommit stepseq, outbox, adoptionSQLiteNode, Durable Objects, browser(WebAssembly)memorytestsDurable's memory, SQLite and JSONL storages stay (S15), with only the adopt argument added (S1); StoreStorage joinsthem.
Figure 1. The layers. Durable keeps its existing storages and gets one more, the adapter; Chord and plugins use the store directly.
+

A table belongs to one object, like a document: only that object reads and writes it; its address is (host kind, +object type, table name), never a host instance such as the session id (W7), so a params change, a split or an import +never orphans data; other plugins see the data only through the object's +state and methods.

+
const Edits = defineTable("edits", {
+  version: 2,
+  columns: { id: str(), path: str(), entry: int(), at: int(), meta: json<EditMeta>() },   // flat
+  key: ["id"],
+  indexes: { byPath: ["path", "at"], byEntry: ["entry"] },
+});
+
+host.serve(FileHistoryStore, (_, self) => {          // a local object with empty params
+  const edits = self.table(Edits);
+  return {
+    record: (edit, ctx) => self.tx(async (tx) => { await tx.put(edits, edit); }, ctx),
+    versions: ({ path }, ctx) =>
+      edits.page(query(Edits).where((e) => eq(e.path, path)).orderBy((e) => e.at, "desc"), 20, ctx),
+  };
+});
+
    +
  • A table's columns are a flat record: each column is a scalar (str, int, bool, num) or json(...); nesting +exists only inside json(...). Every column is a real SQLite column. Scalars can be filtered and ordered; keys and +indexed columns must be str or int. A json(...) column holds JSON text: typed when read, never queryable or +indexable (the type checker refuses it).
    • +
  • Tables and documents of an object are written in the same transaction API, self.tx(fn, ctx) (S14). The context +comes last in every signature (owner's rule).
    • +
  • Index definitions are recorded like other table changes: when the owning object starts on the driver, the store +creates or drops the difference between declared and existing indexes, and every replica applies it. Index contents +never travel; every backend maintains them from the rows. So every copy of the store is the same file.
    • +
+
+ +
IV

Writing It Down

One commit queue per store, one seq for every owner, and adoption inside the step.

+

Two different things are serialized, and they must not be confused. A line serializes a whole transaction +callback: read, decide, write. Chord has one line per object, Durable one per Session. The commit queue serializes +only the final step: checks, writes, seq, outbox record. It is short and there is one per store.

+
object Aits own lineobject Bits own lineDurable Sessionits own linecommit queueone per storedraft notes …draft drafts …phase writes …commitseq 41: write, outbox, adopt Acommitseq 42: write, outbox, adopt Bcommitseq 43: write, outbox, adopt Durable
Figure 2. Two Chord objects and a Durable turn commit at the same time. Their callbacks never wait for each other; only the short commit steps queue.
+

Durable's seq is the store's seq. Durable sees gaps (other owners' commits in between), and that is harmless: every use +of a seq in Durable compares (created_at <= at, seq > base); nothing computes seq + 1 to find a neighbour (checked +in the storages and the Session).

+

Adoption inside the step (S1)

+

After a commit is on disk, each owner swaps its in-memory state to the committed version. If that happens later, in +the owner's own continuation, the order can flip:

+
disk:    40 file-history, 41 Durable
+memory:  Durable adopts 41 first (one await fewer)
+         a routine on "entry 812 committed" reads file-history → still at 39
+

So adoption runs inside the commit step, synchronously, before the next commit starts. Durable's storage interface +carries it as a new required argument:

+
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq>;
+

Every storage calls adopt(seq) once its commit is durable and before its next commit. The Session catches its own +adoption errors; a rejection after adoption fails the Session without discarding the adopted changes.

+

Adopt and mark in the step, flush right after (W1)

+

The step itself stays short: it writes, adopts (pointer swaps of every changed value, including Durable's views the +commit touched) and marks (dependent derives dirty, ops appended by reference to each touched hold's batch and each +watch's queue). Right after the step, synchronously and before any other code runs, the committing owner flushes: +it recomputes the dirty derives someone holds or watches, encodes one part per touched hold into one message per link, +and queues watch callbacks. So a source and the derives over it always travel in one message; a throwing derive fails +only its own member, never the commit.

+
step 40:  write · adopt notes · mark count dirty, notes' ops into tab 1's batch
+flush 40: count = 3 → its ops join the batch · ONE message to tab 1 { notes: +1, count: 3 }
+step 41:  write · adopt Durable's view · mark lastReply …
+

Watch listeners run afterwards, each in order; their order across owners may differ from the seq, but every listener +reads a complete prefix of the log.

+

When a commit counts as done (S3)

+

The step ends only on the backend's terminal success: SQLite's COMMIT returned (with an asynchronous backend, the +transaction's final completion event, not its individual requests). The memory backend stages a +commit and swaps it in at the end. An ambiguous outcome, an error during COMMIT, stops the store +(storage_failed): nobody knows whether the commit landed, so nothing more is written or published.

+
Review: request success is not commit completion

Astra's trace: IndexedDB reports the puts of seq 40 successful, the store publishes 40, the transaction aborts later +(quota), 50 commits on top: memory has 40, disk does not. The rule above is the answer.

+
+

Ids

+

Each owner's id counter starts, when the store opens, above that owner's highest stored id: one index lookup per table +(S10). An id handed out to a transaction that never committed may be handed out again after a restart. Nobody can +notice, because nothing leaves a transaction before it commits (S2), so Durable's mintId contract is narrowed to say +exactly that.

+
+ +
V

Reading

A small query language, one text order on every backend, and cursors.

+

Queries are immutable data built in TanStack DB's style; S12 starts without joins, grouping and projections:

+
query(Edits).where((e) => eq(e.path, "src/app.ts")).orderBy((e) => e.at, "desc")
+query(Edits).where((e) => and(gte(e.at, from), lt(e.at, to)))
+query(Edits).where((e) => prefix(e.path, "a/"))
+

where takes eq ne lt lte gt gte inArray isNull prefix and or not. limit and the cursor are arguments of the +runner, not part of the query. count(q) is the one aggregate; others are computed in JavaScript over indexed scans +until a real case needs more.

+

One text order

+

Text compares by Unicode code point on every backend (owner, 2026-10-26; this replaced the first design, UTF-16 BLOBs +in JavaScript order, S16 and W8). That is the order SQLite compares UTF-8 text in natively, and Durable Objects, +sqlite-wasm, Postgres (COLLATE "C") and MySQL (utf8mb4_0900_bin) do too. Every str() column is plain UTF-8 +TEXT: strings go to the driver and come back as they are, readable in sqlite3 and in any other tool.

+

It differs from JavaScript's < and sort() in one range only: characters U+E000-U+FFFF (fullwidth forms such as +～, private use, U+FFFD) against characters above U+FFFF (emoji):

+
JS sort():    "测试😀" < "测试（一）"
+store order:  "测试（一）" < "测试😀"
+

The store exports compareText(a, b) for plugins that sort store text themselves; the memory backend uses it for +ordering, ranges, max and min. Page with the store's cursors and use prefix(); a hand-built bound such as +p + "\uffff" misses every p followed by an emoji.

+

UTF-8 cannot hold a lone surrogate, so every str() value written, and every query operand, goes through +toWellFormed(): a lone surrogate becomes U+FFFD, as TextEncoder does. A string cut in the middle of an emoji by +slice() is stored with U+FFFD in its place; cut text with [...s].slice() to keep it whole. A str() value may not +contain NUL (invalid/args); content belongs in json(...) columns, which escape both. There is no Unicode +normalization: "é" written as one code point and "e" plus a combining accent are two keys.

+ + + + + + + + + + + + + + + + + + + + + + + +
100k path-like keysUTF-16 BLOB (the first design)UTF-8 TEXT
insert, one transaction125 ms113 ms
read106 ms32 ms
bytes of an 18-character ASCII path3618
+

Running queries

+

A query is immutable data; the table handle runs it, and the context comes last:

+
const edits = self.table(Edits);
+const byPath = query(Edits).where((e) => prefix(e.path, "src/")).orderBy((e) => e.path);
+
+const page = await edits.page(byPath, 100, ctx, { after: cursor });   // { rows, next? }
+for await (const row of edits.rows(byPath, ctx)) { … }                 // all pages
+const e1 = await edits.get(["e1"], ctx);
+
+await self.tx(async (tx) => {
+  for await (const row of tx.rows(edits, byPath)) {                    // committed rows, stable during the tx
+    await tx.update(edits, [row.id], { dir: dirname(row.path) });
+  }
+  const again = await tx.get(edits, ["e1"]);                           // sees this transaction's own writes
+}, ctx);
+

Inside a transaction, tx.page and tx.rows return committed rows: a stable view, because only the owning object +writes its tables and its transactions run one at a time. They do not show the transaction's own writes; tx.get does. +The rule in one sentence: queries inside a transaction see the state before it; your own writes become visible after +the commit. After an updateWhere or deleteWhere on a table, tx.get on that table throws read_after_bulk +(W10): the store would need a second evaluator to know what a bulk operation did to one row.

+

Cursors and consistent reads

+

Every index ends with the primary key, so order is total and cursors are stable. A cursor is opaque, carries its +direction and its last key as logical values, is bound to its query's order, and is refused by another query or the +other direction.

+

Reads outside a transaction see committed state, one query at a time. A read that needs several queries, like a +document assembled from its base and later revisions, runs in one backend read transaction (S4). Two separately +awaited reads in object code are not a snapshot, and the guide says so.

+
+ +
VI

Documents

One documents engine in the store, used by Durable's adapter and by Chord.

+

Durable and Chord both persist documents the same way: a record per incarnation, revisions made of a base and later +deltas, versions, checkpointWhen, one live document per address. The store implements it once (S13):

+
type DocAddress = { namespace: string; owner: string; kind: string; key?: string };
+
+interface Documents {                                   // reads, committed state
+  find(address: DocAddress, at: At, context: Context): Promise<DocRecord | undefined>;
+  load(id: number, at: At, context: Context): Promise<StoredDoc | undefined>;   // base + deltas, applied once
+  list(query: DocListQuery, limit: number, cursor: Cursor | undefined, context: Context): Promise<Page<DocRecord>>;
+}
+interface DocumentWrites {                              // inside the commit step
+  create(record: NewDocRecord, base: BaseContent): number;
+  change(id: number, content: DocContent): void;
+  retire(id: number): void;
+}
+

The owner string is opaque to the engine; each adapter fills it:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
namespaceownerkindkey
Durable session documentpi-durablesessionpi.settings
Durable conversation documentpi-durableconversation:7pi.provider
Durable task documentpi-durabletask:31pi.codemode
Chord persisted documentchordsession/notes.storenotesn-42
+

What the owner string means (a conversation's documents fork and rewind with it, a task's are retired with the task) +stays in the adapter. The engine keeps nothing in memory; Durable's Session and Chord's objects keep their Delta +trackers, cache, pin and evict exactly as Durable does today. A fork's document copies are reconstructed by +StoreStorage before it enters the commit step (S11), so the step never replays revisions while every owner waits; +the outbox only ever carries the resulting documents.

+
+ +
VII

Writing Many Rows

Bulk operations recorded as operations, and migrations that run only where the code is.

+

A full-row put per changed row makes the rename of a/ to b/ over ten thousand 4 KB rows cost about 40 MB in the +outbox. The store's writes are therefore five:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
WriteRecords
tx.put(table, row)the whole row
tx.update(table, key, set)the changed columns of one row
tx.delete(table, key)the key
tx.updateWhere(table, where, set)one operation, plus the affected row count
tx.deleteWhere(table, where)one operation, plus the affected row count
+
await self.tx(async (tx) => {
+  await tx.updateWhere(Edits, (e) => prefix(e.path, "a/"), { path: replacePrefix("a/", "b/") });
+}, ctx);
+
-- on SQLite: one statement; path is UTF-8 TEXT, "a/" is 2 code points
+UPDATE "file-history/edits" SET path = :to || substr(path, 3) WHERE path >= :lo AND path < :hi
+
outbox, seq 57 (about 150 bytes):
+{ op: "update", table: "file-history/edits", where: { prefix: ["path", "a/"] },
+  set: { path: { replacePrefix: ["a/", "b/"] } }, count: 10000 }
+

A replica runs the same operation, without plugin code, and compares the number of affected rows. A different number +means divergence: it stops applying that store and restores it from a snapshot. The set expressions touch only real +columns: value, col, add, sub, max, min, concat, replacePrefix, coalesce, when. They are the store's +own, specified once and checked on every backend by one conformance suite, which is why there are no SQL subqueries, +joins, case folding or floats: each of those behaves differently somewhere.

+

Migrations (S17)

+

Adding, renaming or dropping a column is a recorded table change (alter); a new column gets its default without a +bulk rewrite. A change of the shape inside a json(...) column (a field added, renamed, restructured) migrates on +access (W11): each json(...) column has its own version and a value-only migration, +json(MetaV2, { version: 2, migrate: (old, from) => … }), stored with its value; reading an older value migrates it, +writing the column stores the new shape and version. Nothing is rewritten in bulk and indexes are unaffected. The row's +v counts only eager steps (below); update, updateWhere and deleteWhere never set it. Old code refuses a newer +row v or a newer JSON column version.

+

A change to a key or an indexed column must be eager, because an index over a column filled only on access would +silently miss rows. It runs when the owning object starts on the driver, before the object goes live:

+
migrations: {
+  4: { alter: { add: { dir: { type: "text" } } },
+       rows: (row) => ({ dir: dirname(row.path) }) },                         // scan + tx.update, in chunks
+  5: { alter: { add: { size: { type: "int", default: 0 } } },
+       update: { where: (e) => isNull(e.size), set: { size: value(0) } } },     // one updateWhere
+},
+
Migration 4 on the driver

The table change

Commit 80 records alter: add column dir and marks table version 4 as migrating. Every replica adds the column +from the record.

+

The scan

The store reads rows with v < 4 in key order, 500 at a time, applies rows to each and writes the changed columns +plus v = 4 in the same write. One commit per chunk, so other owners' commits interleave and the commit step stays +short. (A full-row put also sets v; partial and bulk writes never do.)

+

Ready

The last chunk marks version 4 ready. The store creates the byDir index and records it, and FileHistoryStore +goes live.

+

After a crash in the middle

The next start finds migrating, scans again from the first key and skips rows with v >= 4. Each row is +transformed exactly once, even when the transform is not idempotent. A key change is never a rows step: it is a +recorded rebuild that every replica performs itself.

+
+

A replica without the plugin applies the column change and the column updates; a replica with the plugin's code treats +the table as not ready while it is migrating. Chord documents keep their own per-document migrate on access, and +Durable versions its records itself.

+
+ +
VIII

The Easy Parts

Blobs and derived data: why they need none of the store's ordering machinery.

+

Two kinds of session data stay outside the store's tables, and both are easy for the same reason: nothing about them +ever changes in place. A blob is final once written; derived data has no data of its own.

+

Blobs

+

Artifact versions, uploads and sent files are blobs: files in the session directory, named by the hash of their +content (blobs/sha256/9f/9f3a…), never changed. They are app layer, not store: any process may write one, because a +file named by its content cannot conflict with another writer. The order is the whole trick: the blob is written first, +then the row or document that names it.

+
artifact toolblobs/storewrite sha256:9f3a… (tmp, fsync, rename)seq 41: chart.versions[2] = { blob: "sha256:9f3a…" }seq 42: entry 812 "chart v3 saved" (Durable)
Figure 3. Saving artifact "chart" version 3. The blob exists before anything refers to it.
+
    +
  • Export copies blobs/ together with the store snapshot. Every blob a record up to the export's head names was +written before that record, so it is in the directory. Import copies them back and checks each hash.
    • +
  • Sync: before uploading a batch, the sync layer uploads every blob file that is present and not yet uploaded. No +reference tracking is needed: a blob is always written before the commit that names it, so a replica never sees a +hash whose file is missing.
    • +
  • Deleting: no blob is deleted before its session is deleted. A blob that nothing references any more (a crash +between the write and the commit, a retired document) only costs disk; a garbage collector would have to know every +reference in every table.
    • +
+

Derived data

+

A full-text index, a cache, a plugin's own search database: derived data can be recomputed from the store and the +blobs, so it is never exported or synced. Its owner keeps it current with a routine that compares it with the store. +The artifacts plugin's full-text index in its own SQLite file:

+
host.serve(ArtifactStore, (_, self) => {
+  const artifacts = self.state(Artifact);                         // keyed documents in the store
+  const blobs = self.get(SessionBlobs, {});
+  const index = self.open(async () => openIndex(join(dir, "derived/artifacts/index.sqlite")), (db) => db.close());
+
+  self.routines.loop("catch-up", async (ctx) => {                 // the index repairs itself
+    for await (const name of artifacts.keys()) {
+      const a = await artifacts.get(name);
+      if (a && index.value.versions(name) !== a.versions.length)
+        index.value.put(name, a.versions.length, await blobs.read(a.versions.at(-1)!.blob, ctx));
+    }
+  });
+  …
+});
+
One routine, three situations

Normal save

save writes the blob, commits the document, then updates the index. The routine finds nothing to do.

+

Crash after the commit

The process dies between the commit and the index update. The document says version 3, the index still has version 2. +On the next start the routine sees the difference, reads the blob of version 3 and indexes it.

+

Import or sync

On the desktop derived/artifacts/ does not exist. The routine finds every artifact missing and indexes them all from +the documents and blobs that came with the session.

+
+

This routine visits every artifact on each start: fine for hundreds, slow for millions. A large derived store keeps +the last seq it applied and reads only what changed after it.

+
+ +
IX

Moving a Session

Outbox, replicas (no compaction yet), takeover, export and import.

+

Every commit's outbox record is written in the same backend transaction as its data, so a crash keeps both or +neither. Sync is a layer above the store; it follows Armin's transport and fencing with the rules the review added.

+
    +
  • Uploading: the driver may upload batches in parallel, resubmits each until Radius acknowledges it, and trims a +batch from its outbox only after its acknowledgement (S5).
    • +
  • Applying: every batch carries from and to. A replica applies a batch only when its from equals the +replica's applied head, as a whole, in one backend transaction together with the new applied head. A later batch +waits for a missing one. A crash leaves all of a batch or none of it.
    • +
  • No compaction at first (W9): batches carry the ordered operations exactly as committed. "Keep the last write per +row" would lose rows built from put plus partial updates; a later compaction must merge those and treat bulk +operations as barriers.
    • +
  • After a power loss the driver's local head may be behind what Radius has; it first pulls back its own uploaded +batches, then continues (S3).
    • +
  • Blobs go up before the batch that may name them: every blob file present and not yet uploaded (chapter VIII).
    • +
+

Losing the driver role (S7)

+
laptopdriver, epoch 3Radiusdesktoptake over S1epoch 4commits 40, 41 locally (a routine, a late tool result)upload 40–41 (epoch 3)refused: epoch 4the copy becomes session S1′ (new id)S1 continues on the desktop; S1′ on the laptop; they never sync again
Figure 4. The laptop learns the desktop took over. Nothing is rolled back.
+

Rolling back is not possible: tools ran, files were written, natives and derived data followed the tail. The laptop's +copy becomes a separate session with a new id and keeps running; commits in flight land in it. The catalog shows both, +the split-off copy marked with its origin.

+

Export and import (S4)

+

An export holds the commit queue while it reads, so the archive is the state at one head: Durable's records, Chord's +documents and every plugin table, including tables of plugins that are not installed, plus the session directory's +blobs (chapter VIII). An import restores the archive into a fresh store; the server then mirrors the session into its catalog. +Derived data (full-text indexes, caches) is never exported; it is rebuilt where it is needed.

+
+ +
X

Rules for Plugin Authors

What the store does not do for you.

+

Nothing leaves a transaction before it commits (S2). Ids are handed out inside the transaction that creates them. +Reference what is committed: the file-history plugin references the tool-call entry, which is committed before the tool +runs, never the tool-result entry, which does not exist yet.

+

Work that finishes out of order is yours (S8). A routine that starts a slow job per change can let an old result +overwrite a newer one. The handler calls of one routine never overlap, so either chain through the routine or drop +stale results inside the destination's transaction:

+
self.routines.on(conversation.entries, async (entries, ops, ctx) => {
+  const run = await summarize.start({ upTo: entries.at(-1)!.id }, ctx);
+  await run.outcome(ctx);                                  // the next handler waits for this run
+});
+

Text compares by code point. Do not sort keys yourself to page through a table; use the store's cursors, or +compareText. Normalize text yourself if two spellings must be one key.

+

Bulk changes are operations. Prefer updateWhere and deleteWhere over loops of puts; a loop records every row.

+
+ +
∴

What's Next

What is decided, and the order in which it gets built.

+

The direction and all seventeen review items are decided (decision log, 2026-10-21). The store is new code; the parts +outside it are small changes in pi:

+
    +
  • pi, one change: Storage.commit(writes, adopt, context) with every storage calling adopt (S1), the +Session's handling of rejections after adoption, the narrowed mintId comment (S10), traceIn(options, fn, context) (context last), Durable's WatchHandle/WatchEnd moved into Chord with generic end reasons and explicit +retirement (W2), and Durable's state getters and the replicatedState family removed (W14).
    • +
  • The store in packages/chord/src/next: commit step and outbox, tables and the query language, the SQLite and +memory backends with the text encoding, the documents engine, bulk operations and migrations, export and import; +about 1,650 lines by Fable's estimate, plus the conformance suite.
    • +
  • StoreStorage, passing Durable's existing storage conformance suite.
    • +
  • The sync layer (upload, apply, pull-back, epochs, splitting a session) on top, when Radius sync is built.
    • +
+

Open, to settle while building: how a split-off session gets its new id (store metadata, directory, catalog), the +chunk size of row migrations, and a Postgres backend once a serverless target needs one (W14).

+
+ +
A

Decisions

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
#Decision
The storeone transactional store per session under Durable (adapter), Chord documents and plugin tables
S1adoption inside the commit step; Storage.commit(writes, adopt, context)
S2nothing leaves a transaction before it commits; no mechanism
S3a commit is done on the backend's terminal success; ambiguous outcomes stop the store; pull-back after restart
S4export holds the commit queue; multi-query reads in one read transaction; no global snapshots
S5parallel uploads, resubmit until acknowledged, trim after acknowledgement; apply whole batches in order
S6(superseded by W9)
S7losing the driver role splits the session; no rollback
S8out-of-order work is an app concern; routine handlers never overlap
S9the head is persisted with every commit
S10id counters derived at open; mintId contract narrowed
S11fork copies reconstructed by StoreStorage before the commit step
S12the query language starts without joins, grouping and projections
S13one documents engine in the store, opaque owner
S14one transaction API, self.tx(fn, ctx)
S15Durable's existing storages are kept
S16, textreal tables; text as plain UTF-8 in code point order (replaced W8's UTF-16 BLOBs)
S17, W11five writes, bulk operations recorded with a row count; JSON columns migrate on access with their own version; eager steps by scan with row v
W1adopt and mark in the commit step, one synchronous flush right after (derives, wire, watches)
W6, W7workers connect before loading; store addresses carry the host kind, never an instance
W9, W10no compaction at first; no tx.get on a table after a bulk operation in the same transaction
W14no IndexedDB backend (SQLite in WebAssembly in the browser); no replicatedState
index definitionsrecorded in the outbox and applied by every replica; contents never travel
blobs, derived datablobs are app-layer files named by hash, written before what names them, uploaded before each batch, deleted only with the session; derived data is rebuilt, never shipped
+
+ +
B

Glossary

+

Adoption: an owner updating its in-memory state to a committed write. Blob: a file in the session directory +named by the hash of its content, never changed. Derived data: data recomputed from the store and blobs, never +shipped. Backend: what the store runs on (SQLite, +memory). Batch: a range of outbox records from from to to, uploaded and applied as a whole. +Commit step: the short, serialized part of a commit in which the store checks, writes, numbers and records it, and +owners adopt. Driver: the machine that writes a session. Line: the serialization of one owner's transactions +(per Chord object, per Durable Session). Outbox: the record of every commit, written in the same transaction. +Owner: the code that writes a piece of data. Replica: a copy of a session that applies the driver's batches. +Seq: the number of a commit, one sequence per store. Store: the transactional store of one session. +Table: a set of rows owned by one object, with a flat record of typed columns (scalars or json(...)).

+
+ +

One Store · first edition · 2026-10-21 · based on the redesign decision log and the store review

+ + + diff --git a/packages/chord/docs/next/store/walkthrough.md b/packages/chord/docs/next/store/walkthrough.md new file mode 100644 index 000000000..c682fe3e3 --- /dev/null +++ b/packages/chord/docs/next/store/walkthrough.md @@ -0,0 +1,193 @@ +--- +title: One Store +kind: walkthrough +kicker: A walkthrough · Chord v2 and p3 · 2026-10-21 +subtitle: How one transactional store keeps a p3 session's transcript, documents and plugin tables consistent, taught one concept at a time +model: anthropic/claude-sonnet-5-5 +sources: + - packages/chord/docs/next/store/book.md + - packages/chord/docs/next/decisions.md + - apps/p2/docs/p3-reviews/store-review-summary.md + - apps/p2/docs/p3-reviews/store-bulk-migrations.md + - apps/p2/docs/p3-reviews/session-data-scenarios.md + - apps/p2/docs/p3-reviews/store-proposal.md + - packages/durable/src/storage/sqlite/storage.ts + - packages/durable/src/session/session.ts +truth: The decision log (redesign-decisions.md, entries "The store", "S1" to "S17", "Context last") wins over everything; the book (book.md) is written from it; store-proposal.md and session-data-scenarios.md are earlier drafts and the discussion, superseded where the log differs; the Durable sources show today's code, before the S1 change. +--- + +:::tutor +Audience: the owner of p3 and Chord, who decided this design in one long discussion and wants to understand it end to +end, or a developer joining the work who knows TypeScript, SQLite and pi-durable but not this store. Use the book's +running example throughout: Mario's session S1 on his laptop, the file-history plugin with its `edits` table, entry +812, the rename of `a/` to `b/` over 10,000 rows, continuing on the desktop. Read the matching book chapter before each +concept and check decisions in the log. Be concrete: code, short traces with seq numbers, small tables. Short messages; +one figure per concept at most. When the reader asks why something was not done another way, answer from the book's +"What We Tried and Dropped" chapter and the log. +::: + +:::intro +#### What this is about + +A p3 session must be self-contained: everything it owns lives in its own store and directory, so it can be exported, +imported into another p3, synced to Radius and continued on another device. Three kinds of owners write into a +session: Durable (the transcript), Chord objects (persisted documents) and plugins (their own tables). This walkthrough +explains the store they all share and why it is built that way. + +#### The example + +Mario works in session S1 on his laptop. A plugin called **file-history** records every file edit in its table +`edits`: path, the conversation entry that made it (say entry 812), the time. He renames the directory `a/` to `b/` +(10,000 rows change). In the evening he continues S1 on his desktop; another day he exports it and a colleague imports +it. Every concept follows this session through one mechanism. + +#### How it progresses + +Eleven concepts in four parts: the problem and what was dropped; how the store writes (commit queue, seq, adoption, +completion); how it reads and stores documents and tables (text order, the documents engine, bulk operations and +migrations; blobs and derived data); and how a session moves (sync, takeover, export) plus the rules left to plugin authors. +::: + +# The problem {#problem part="Why one store"} +Book "Before we begin" and chapter I. Must cover: P1 (a session is self-contained; the server only mirrors); three +owners (Durable, Chord objects, plugins); the trace where file-history's own `history.sqlite` and Durable's log sync +separately and entry 812 dangles or points to the wrong entry after takeover; what is missing is one order across +storages. A short trace or a steps figure helps. + +:::check +1. Both of S1's storages are consistent on their own. Why can the desktop still end up in a state the laptop never had? ++++ answers +1. Because nothing orders the changes of one storage relative to the other: the desktop can receive history.sqlite up to the edit but Durable's log only up to the commit before it (or the reverse), a combination that never existed on the laptop. +::: + +# What was dropped {#dropped} +Book chapter II; log entry "The store" (its list of rejected options) and S16. Must cover the main dropped options and +the reason for each: derived-only plugin data, opaque directories, export hooks, plugin record logs with apply, several +storages with a shared position counter, recorded SQLite statements, raw SQL reads, an IndexedDB-only API; inside the +store: generic JSON rows, store-maintained index entries, code point order with replaced surrogates, text equality only. +Point out what the final design kept from them (one order, recorded operations, a small query language). + +:::check +1. Why was "record the plugin's SQL statements and replay them on the replica" dropped, while bulk operations are now recorded as operations and replayed? ++++ answers +1. Arbitrary SQL can produce different results on a replica (random(), datetime('now'), unordered INSERT ... SELECT, other SQLite versions); the store's bulk operations use only the store's own expressions, specified once and identical on every backend, plus a row count check. +::: + +# The store and its tables {#store part="Writing"} +Book chapter III; log "The store", S14, S16. Must cover: layers (owners, store with documents engine, tables and commit +step, backends); a table belongs to one object (address host, object type, table name); `defineTable` with typed key +and indexed columns (`str` or `int`) in a flat `columns` record of real columns, nesting only inside `json()` columns (typed but not queryable); one transaction API +`self.tx(fn, ctx)`; index definitions are recorded in the outbox (contents never travel); Durable keeps its storages and gains `StoreStorage`; context last. + +:::check +1. In `edits`, which fields can a query filter on, and where does the rest of the row live? ++++ answers +1. Every scalar column (str, int, bool, num) is a real column and can be filtered; keys and indexes need str or int; json(...) columns hold JSON text, typed when read but not queryable or indexable. +::: + +# Lines, the commit queue and the seq {#commit} +Book chapter IV; log S1, S3, S9, S10. Must cover: lines (per Chord object, per Durable Session) serialize whole +transactions, the commit queue serializes only the short final step; one seq per store, Durable sees gaps and only +compares seqs; adoption inside the step with `Storage.commit(writes, adopt, context)`; why (the trace where Durable +adopts 41 before file-history adopts 40); listeners still run afterwards and read a complete prefix; a commit is done +only on the backend's terminal success; ambiguous outcomes stop the store; ids derived at open. The book's sequence +figure of two objects and a Durable turn helps. + +:::check +1. Object A commits seq 40 and Durable commits 41. Without S1, how could a routine see 41 but not 40? +2. Why is it harmless that Durable's seqs now have gaps? ++++ answers +1. Each owner adopted in its own continuation after the disk commit; Durable's continuation ran first, so a routine triggered by 41 read A's in-memory state still at 39. With S1 both adoptions run inside the commit step, in seq order. +2. Durable only compares seqs (created_at <= at, seq > base); it never computes seq + 1 to find a neighbouring commit. +::: + +# Reading and text order {#reading part="Reading and storing"} +Book chapter V; log S4, S12, S16. Must cover: the small query language (where operators, orderBy, limit, cursor; no +joins, grouping, projections at first); JavaScript and SQLite order strings differently (the ～ vs 😀 example); the +decision: JavaScript order everywhere, SQLite stores key and indexed text as UTF-16 BLOBs (big-endian), lossless, +measured costs; no normalization; cursors carry direction and logical values; multi-query reads in one read +transaction, no global snapshot. + +:::check +1. A plugin pages through `edits` by path with a cursor. Why would it break if SQLite stored paths as ordinary TEXT while the memory backend compared JS strings? +2. What does the UTF-16 BLOB encoding cost, according to the measurements? ++++ answers +1. SQLite would order by UTF-8 bytes and JS by UTF-16 code units; they disagree for characters between U+E000 and U+FFFF versus emoji, so a cursor taken from one order skips or repeats rows in the other (and tests on the memory backend would not show what SQLite does). +2. Mostly JS encoding and decoding plus twice the bytes for ASCII: about +40% on bulk inserts, +62% file size in the test; SQLite's own range scans are about 4% slower when values are not decoded from the BLOBs. +::: + +# The documents engine {#documents} +Book chapter VI; log S11, S13. Must cover: Durable and Chord persist documents the same way, so the store has one +engine; address (namespace, owner, kind, key) with an opaque owner string filled by each adapter (Durable: session, +conversation:7, task:31; Chord: host/object type); find, load, list outside the step, create, change, retire inside it; +load reconstructs base plus deltas once, the engine keeps nothing in memory; Durable's Session caches and evicts as +today; fork copies are reconstructed by StoreStorage before the commit step. + +:::check +1. Who decides that a conversation's documents fork and rewind with it: the documents engine or Durable's adapter? ++++ answers +1. Durable's adapter (StoreStorage): the engine only sees an opaque owner string such as "conversation:7"; the meaning of that owner stays with Durable. +::: + +# Bulk operations and migrations {#bulk} +Book chapter VII; log S17; store-bulk-migrations.md. Must cover: the five writes (put, update, delete, updateWhere, +deleteWhere); the rename of a/ to b/ as one recorded operation with a row count (about 150 bytes instead of about 40 +MB), compiled to one SQL UPDATE on the UTF-16 BLOB column; the replica runs the same operation and compares the count; +the set expressions and why they are limited (identical behaviour on every backend; no JSON operands yet); migrations: +changes inside json(...) columns on access via migrate(row, from), column changes as recorded alters, key and indexed columns eagerly by recorded table changes plus an update +step or a rows step (scan + tx.update in chunks, migrating then ready, resume after a crash). The book's migration steps +figure helps. + +:::check +1. Version 4 of `edits` adds an indexed column `dir` computed with dirname(path). Why can this not migrate on access, and how does it run instead? +2. A replica applies the rename and affects 9,998 rows instead of 10,000. What happens? ++++ answers +1. An index over a column filled only on access would silently miss unmigrated rows; so it runs eagerly on the driver before the object goes live: a recorded alter adds the column, then a rows step scans in key order and writes dir with tx.update, chunk by chunk, until the table is marked ready. +2. The counts differ, so the replica stops applying that store and restores it from a snapshot: divergence is detected, not silent. +::: + +# Blobs and derived data {#easy} +Book chapter VIII "The Easy Parts"; log "Blobs and derived data". Must cover: blobs are app-layer files named by the +hash of their content, written before the row or document that names them; export copies them, import checks hashes; +sync uploads every blob present and not yet uploaded before each batch (no reference tracking); no deletion before the +session is deleted; derived data is never shipped and is repaired by a catch-up routine (the artifacts full-text index +example: normal save, crash after commit, import). + +:::check +1. Why does the sync layer not need to know which rows reference which blobs? +2. The process dies between the artifact document's commit and the index update. What fixes the index? ++++ answers +1. A blob is always written before the commit that names it, so uploading every blob present before each batch guarantees that any hash a batch names has its file on Radius already. +2. The catch-up routine on the next start: it compares the index with the documents, finds version 3 missing and indexes it from the blob. +::: + +# Sync and takeover {#sync part="Moving a session"} +Book chapter IX; log S3, S5, S6, S7. Must cover: the outbox record is written in the same backend transaction; +parallel uploads, resubmit until acknowledged, trim after acknowledgement; a replica applies whole batches in order +(from equals its applied head), later batches wait; compaction is safe because batches apply whole; pull-back after a +power loss; losing the driver role splits the session (new id) without rollback, and why rollback is impossible. + +:::check +1. The laptop committed 40 and 41 locally after the desktop took over. What happens to those commits? ++++ answers +1. Nothing is rolled back: when Radius refuses the laptop's upload for an old epoch, the laptop's copy becomes a separate session with a new id, keeps 40 and 41, and never syncs with S1 again; S1 continues on the desktop. +::: + +# Export, import and the rules left to plugins {#export} +Book chapters IX ("Export and import") and X; log S2, S4, S8. Must cover: export holds the commit queue so the +archive is one head, includes tables of plugins that are not installed, never derived data; import restores into a +fresh store and the server mirrors the catalog; rules: nothing leaves a transaction before it commits (reference the +tool-call entry, not the tool-result entry); work that finishes out of order is the app's concern (routine handlers +never overlap: chain or drop stale results); text compares like JS strings; prefer updateWhere over loops of puts. + +:::check +1. The file-history plugin wants to store which entry made an edit. Which entry can it safely reference, and why? ++++ answers +1. The assistant entry with the tool call: it is committed before the tool runs. The tool-result entry does not exist yet while the tool runs, and its id is handed out only inside the harness's later transaction. +::: + +# What comes next {#next part="Where it stands"} +Book "What's Next"; log entries. Must cover: everything decided on 2026-10-21; the pi change (adopt argument, Session +rejection handling, mintId wording, traceIn context last); the store in packages/chord/src/next (about 1,650 lines +estimated); StoreStorage passing Durable's conformance suite; the sync layer later; open points (new id of a split +session, chunk size, whether IndexedDB is needed while sqlite-wasm covers the browser). diff --git a/packages/chord/src/next-node/compression.ts b/packages/chord/src/next-node/compression.ts new file mode 100644 index 000000000..d1b9e7006 --- /dev/null +++ b/packages/chord/src/next-node/compression.ts @@ -0,0 +1,24 @@ +// Compression for Chord v2 hosts on Node (§22, slice 4b): the synchronous compressor `createHost({ compression })` +// takes, over Node's zlib. Node-only, so it lives outside `src/next`, which imports no Node module. + +import { constants as buffer } from "node:buffer"; +import { deflateRawSync, inflateRawSync } from "node:zlib"; +import type { Compression } from "../next/link/index.ts"; + +/** + * Raw deflate at level 1 (`deflateRawSync`), and inflate bounded by `maxOutputLength` (`inflateRawSync`): past the + * limit `too_large`, data that is no raw deflate stream `corrupt`. + */ +export const nodeCompression: Compression = { + deflate: (bytes) => deflateRawSync(bytes, { level: 1 }), + inflate: (bytes, limit) => { + try { + // zlib refuses a bound of 0: inflate at most one byte, and one byte is past a limit of 0. + const inflated = inflateRawSync(bytes, { maxOutputLength: Math.max(1, Math.min(limit, buffer.MAX_LENGTH)) }); + return inflated.byteLength > limit ? { kind: "too_large" } : { kind: "bytes", bytes: inflated }; + } catch (error) { + if ((error as { readonly code?: unknown }).code === "ERR_BUFFER_TOO_LARGE") return { kind: "too_large" }; + return { kind: "corrupt", error }; + } + }, +}; diff --git a/packages/chord/src/next/contracts/README.md b/packages/chord/src/next/contracts/README.md new file mode 100644 index 000000000..a6e2a06ba --- /dev/null +++ b/packages/chord/src/next/contracts/README.md @@ -0,0 +1,80 @@ +# contracts + +Declarations of Chord object types: what an object's state and methods look like, version by version. + +## For apps (`index.ts`) + +```ts +import { arr, evolve, hostKind, type Implementation, method, obj, remote, str } from "./index.ts"; + +export const Session = hostKind("session", { id: str() }); // a kind of addressable host, params like a contract's +const Note = obj({ text: str() }); +export const NotesV1 = remote("notes", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), // where an object lives; not in the fingerprint + variants: { + editor: { state: { notes: arr(Note) }, methods: { add: method(Note, str()), clear: method() } }, + reader: { state: { notes: arr(Note) } }, + }, +}); +export const Notes = evolve(NotesV1, (e) => e.args("add", obj({ text: str(), tag: str() }))); +type NotesProvider = Implementation; // { variant: "editor"; members: { notes, add(args, context), clear(context) } } | { variant: "reader"; ... } +``` + +- `remote(name, { version, params, placement?, variants })`: per-principal variants, each with `state` (schemas) and `methods` + (`method(args?, result?)`; an omitted side is `null`, and a method whose arguments are `null` omits them: + `clear(context)`). `local(name, { version, params, members })`: one flat map, + where a schema is state and `method(...)` a method. +- `placement(params) => Address` (remote only, optional): the address of the host that serves the object for these + params (`host/`'s routing table). The caller computes it and sends it with its hold, so a relay routes by the address + alone. It is not part of the normal form or the fingerprint and `evolve` carries it. +- `hostKind(name, params?)`: a kind of addressable host (`Server = hostKind("server")`, `Session = hostKind("session", + { id: str() })`); `params` follow the params policy below. Calling it makes an `Address { kind, params, key }`, its + key built like an identity key (`session|S1`); two addresses are equal when their keys are. The params are checked + where the address is routed or served (`invalid/placement`), not when it is made. No host-name strings in app code. +- Schema helpers: `str`, `num`, `int`, `bool`, `nul`, `lit`, `oneOf`, `union`, `obj`, `arr`, `map`, `opt`, `bytes`, + `unknown`, and `Static`. They return ordinary TypeBox schemas. +- `evolve(previous, (e) => e...)`: the next version from structured edits (`state`, `addState`, `removeState`, `args`, + `result`, `method`, `addMethod`, `removeMethod`); it must return the editor it was given. +- Types: `Contract`, `Implementation` (what a factory returns: for `local` a `Watchable` per state member and a + function per method, `(args, context)` or `(context)` without arguments; for `remote` one variant, `{ variant, + members }`; `X` is the context type the methods get), `ParamsOf`, `AcceptsTable` (the older versions a provider serves, with their mappings), + `HostKind`, `Address`, `KindShape` and `ParamsOfKind`. + +Rules: + +- Declarations are checked once and fail with an `Error` naming the path (`notes.editor.methods.add.args`). A contract + keeps its own copies of its schemas: changing a schema object after declaring it changes nothing. +- A member name has one kind (state or method) in every variant and for the whole history; a method has one + signature in every variant. State schemas may differ per variant. +- `then`, `release`, `variant`, `state` and `retry` are refused as member names: a reference has them (`then` would + make an awaited reference thenable). +- `$bytes` is reserved only in the values of a method whose argument or result schema has a `bytes()` leaf: there it + marks a byte segment on a link (an own key `$bytes` fails `invalid/args` when it crosses one); everywhere else it is + ordinary data. +- Every object inside a method's arguments is closed (`additionalProperties: false`); `obj()` always is, records + (`map()`) are allowed. +- Params are required `str()`, `int()`, `bool()` or `oneOf(...)` of strings. Params and the variant set never change by + evolution: changing them is a new history base, a new declaration with no accepted older versions. +- Mappings in an `AcceptsTable` are direct (old version to current); when a version is added, update every accepted + version's mappings. A mapping is required per edit after the accepted version, even for a no-op edit. +- Contracts are branded: a spread or hand-built copy is refused everywhere. + +## For Chord's layers (`internal.ts`) + +- `checkAccepts(contract, accepts)`: validates an accepts table without calling a mapping; returns the accepted + versions (version and fingerprint). +- `contractData(contract)`: the Chord-owned data behind a handle (its `placement`, its versions with their + `VariantData`, `MethodData` and schema snapshots, its `ParamsData`); a local contract's members are the variant + `LOCAL`. Type-level helpers: `ShapeOf`, `CurrentVariants`, `ContractShape`, `Params` and `Method`. +- `currentVariants(contract)`: the current version's variants (a local contract has the one variant `LOCAL`); + `takesNoArgs(method)`: whether its argument schema is `null`, so references and implementations omit the argument; + `VariantImplementation` is one variant's members as a type; `variantNames` and `variantOrder`: the names the wire's + indexes point into. +- A method's `args.bytes` and `result.bytes`: whether the schema has a `bytes()` leaf, decided once at declaration (a + link then encodes the method's values with byte segments). +- `kindParams(kind)`: a declared host kind's `ParamsData` (a route checks addresses with it). +- `NORMAL_FORM_VERSION`: changing the normal form changes it and the fingerprints; it is a link-protocol change. + Fingerprints are a portable hash, not a security boundary. +- Nothing in L1 is frozen: Chord-owned data is immutable by convention. diff --git a/packages/chord/src/next/contracts/accepts.ts b/packages/chord/src/next/contracts/accepts.ts new file mode 100644 index 000000000..1596f0132 --- /dev/null +++ b/packages/chord/src/next/contracts/accepts.ts @@ -0,0 +1,244 @@ +// Accepted versions: the older versions a provider still serves, with the direct mappings their edits require. + +import type { Context } from "../../context/index.ts"; +import { + type Contract, + type ContractData, + type ContractShape, + contractData, + type Kind, + LOCAL, + type ShapeOf, +} from "./contract.ts"; +import { checkKeys, entriesOf } from "./declare.ts"; +import type { + Change, + ChangesAfter, + CurrentStep, + MemberNames, + MethodAt, + OlderVersions, + StepAt, + VariantShape, + Variants, + VariantsWith, +} from "./history.ts"; +import type { Method, Static } from "./schema.ts"; + +/** Per accepted older version, exactly the mappings its later edits require (direct: old to current). */ +export type AcceptsTable = AcceptsOf>; + +type AcceptsOf = { + readonly [V in OlderVersions]?: VersionMappings< + S["kind"], + StepAt["variants"], + CurrentStep["variants"], + ChangesAfter + >; +}; + +type VersionMappings = Section< + "state", + K extends "local" ? StateMappings : RemoteStateMappings +> & + Section<"methods", MethodMappings>; + +type Local = typeof LOCAL; + +/** A section is required when it holds a required mapping; otherwise it may be absent or empty. */ +type Section = [keyof T] extends [never] + ? { readonly [K in Key]?: { readonly [member: string]: never } } + : { readonly [K in Key]: T }; + +/** `T` without its `never` properties: the members that need no mapping. */ +type OmitNever = { readonly [K in keyof T as [T[K]] extends [never] ? never : K]: T[K] }; + +type RemoteStateMappings = OmitNever<{ + [X in keyof Old & keyof Current]: [keyof StateMappings] extends [never] + ? never + : StateMappings; +}>; + +type StateMappings = OmitNever<{ + [S in keyof Old["state"]]: StateMapping; +}>; + +/** A changed member maps the current member back; a removed one maps the variant's whole current state, or refuses. */ +type StateMapping< + Old extends VariantShape, + Current extends VariantShape, + X, + S extends keyof Old["state"], + C, +> = S extends keyof Current["state"] + ? [X, S, "state"] extends C + ? (current: Static) => Static + : never + : ((state: { [M in keyof Current["state"]]: Static }) => Static) | "refuse"; + +type MethodMappings = OmitNever<{ + [M in MemberNames]: [RemovedFrom, M>] extends [never] + ? ChangedMethod< + Changed, M, "args">, + Changed, M, "result">, + MethodAt, + MethodAt + > + : Replacement>; +}>; + +/** The variants among `Xs` that no longer have method `M`. */ +type RemovedFrom = { + [X in Xs]: M extends keyof Current[X]["methods"] ? never : X; +}[Xs]; + +/** `true` when an edit changed `Side` of `M` in one of `Xs`. */ +type Changed = Xs extends unknown ? ([Xs, M, Side] extends C ? true : never) : never; + +/** Mappings for the changed sides of a method (`A`, `R` are `true` when changed), or a replacement. */ +type ChangedMethod = [A | R] extends [never] + ? never + : + | ({ readonly call?: never } & ([A] extends [never] + ? { readonly args?: never } + : { readonly args: (old: Static) => Static }) & + ([R] extends [never] + ? { readonly result?: never } + : { readonly result: (current: Static) => Static })) + | Replacement; + +/** A removed (or changed) method served by the provider's own function, or refused. */ +type Replacement = + | { + readonly call: (args: Static, context: Context) => Promise>; + readonly args?: never; + readonly result?: never; + } + | "refuse"; + +/** An accepted older version and its fingerprint. */ +export type AcceptedVersion = { readonly version: number; readonly fingerprint: string }; + +/** + * Validates an accepts table without calling any mapping: known older versions only, exactly the required mappings + * with the right shapes. Fails with an Error naming the path (`notes accepts[1].methods.add`). + */ +export function checkAccepts( + contract: C, + accepts: AcceptsTable, +): { readonly versions: readonly AcceptedVersion[] } { + const data = contractData(contract); + // Declaration policy: JavaScript callers reach every check below. + const versions = entriesOf(accepts, `${data.name} accepts`).map(([key, mappings]) => { + const path = `${data.name} accepts[${key}]`; + const index = data.versions.findIndex(({ version }) => String(version) === key); + if (index === -1 || index === data.versions.length - 1) throw new Error(`${path}: not an older version`); + checkMappings(mappings, needs(data, index), data.kind, path); + const { version, fingerprint } = data.versions[index]!; + return { version, fingerprint }; + }); + return { versions }; +} + +/** What an accepted version needs, by edit and never by schema comparison. */ +type Needs = { + /** Per variant, per state member. */ + readonly state: ReadonlyMap>; + readonly methods: ReadonlyMap; +}; + +type StateNeed = "changed" | "removed"; + +type MethodNeed = + | { readonly kind: "removed" } + | { readonly kind: "changed"; readonly args: boolean; readonly result: boolean }; + +function needs(data: ContractData, index: number): Needs { + const old = data.versions[index]!; + const current = data.versions.at(-1)!; + const changes = new Set(data.versions.slice(index + 1).flatMap((version) => version.changes.map(changeKey))); + const changed = (...change: Change) => changes.has(changeKey(change)); + const state = new Map>(); + const methods = new Map(); + for (const [variant, { state: oldState, methods: oldMethods }] of old.variants) { + const now = current.variants.get(variant)!; + const variantNeeds = new Map(); + for (const member of oldState.keys()) { + if (!now.state.has(member)) variantNeeds.set(member, "removed"); + else if (changed(variant, member, "state")) variantNeeds.set(member, "changed"); + } + if (variantNeeds.size > 0) state.set(variant, variantNeeds); + for (const member of oldMethods.keys()) { + const need = methods.get(member) ?? { removed: false, args: false, result: false }; + need.removed ||= !now.methods.has(member); + need.args ||= changed(variant, member, "args"); + need.result ||= changed(variant, member, "result"); + methods.set(member, need); + } + } + const methodNeeds = new Map(); + for (const [member, { removed, args, result }] of methods) { + if (removed) methodNeeds.set(member, { kind: "removed" }); + else if (args || result) methodNeeds.set(member, { kind: "changed", args, result }); + } + return { state, methods: methodNeeds }; +} + +function changeKey(change: Change): string { + return JSON.stringify(change); +} + +function checkMappings(mappings: unknown, needs: Needs, kind: Kind, path: string): void { + checkKeys(mappings, ["state", "methods"], path); + const { state, methods } = mappings as { readonly state?: unknown; readonly methods?: unknown }; + const checkVariant = (variantMappings: unknown, variantNeeds: ReadonlyMap, at: string) => + checkSection(variantMappings, variantNeeds, at, checkStateMapping); + // A local table is flat: its state section is the one variant's. + if (kind === "local") checkVariant(state, needs.state.get(LOCAL) ?? new Map(), `${path}.state`); + else checkSection(state, needs.state, `${path}.state`, checkVariant); + checkSection(methods, needs.methods, `${path}.methods`, checkMethodMapping); +} + +/** Exactly the needed keys: no missing and no unnecessary mapping; absent only when nothing is needed. */ +function checkSection( + section: unknown, + needs: ReadonlyMap, + path: string, + check: (mapping: unknown, need: N, path: string) => void, +): void { + const mappings = new Map(section === undefined ? [] : entriesOf(section, path)); + for (const key of mappings.keys()) if (!needs.has(key)) throw new Error(`${path}.${key}: unnecessary mapping`); + for (const [key, need] of needs) { + if (!mappings.has(key)) throw new Error(`${path}.${key}: missing mapping`); + check(mappings.get(key), need, `${path}.${key}`); + } +} + +function checkStateMapping(mapping: unknown, need: StateNeed, path: string): void { + if (typeof mapping === "function" || (need === "removed" && mapping === "refuse")) return; + throw new Error(`${path}: expected ${need === "removed" ? 'a function or "refuse"' : "a function"}`); +} + +function checkMethodMapping(mapping: unknown, need: MethodNeed, path: string): void { + if (mapping === "refuse") return; + checkKeys(mapping, ["args", "result", "call"], path); + const { args, result, call } = mapping as { + readonly args?: unknown; + readonly result?: unknown; + readonly call?: unknown; + }; + if (call !== undefined || need.kind === "removed") { + if (typeof call !== "function" || args !== undefined || result !== undefined) { + throw new Error(`${path}: expected { call } or "refuse"`); + } + return; + } + checkSide(args, need.args, `${path}.args`); + checkSide(result, need.result, `${path}.result`); +} + +function checkSide(mapping: unknown, needed: boolean, path: string): void { + if (!needed && mapping !== undefined) throw new Error(`${path}: unnecessary mapping`); + if (needed && mapping === undefined) throw new Error(`${path}: missing mapping`); + if (needed && typeof mapping !== "function") throw new Error(`${path}: expected a function`); +} diff --git a/packages/chord/src/next/contracts/contract.ts b/packages/chord/src/next/contracts/contract.ts new file mode 100644 index 000000000..cf113e96f --- /dev/null +++ b/packages/chord/src/next/contracts/contract.ts @@ -0,0 +1,178 @@ +// Contract handles: the branded, immutable value an app declares, its type-level shape, and the Chord-owned data +// behind it. + +import type { TNull } from "typebox"; +import type { Context } from "../../context/index.ts"; +import type { Watchable, WatchHandle } from "../../watch.ts"; +import type { Change, CurrentStep, History, VariantShape } from "./history.ts"; +import type { Address } from "./host-kind.ts"; +import type { Params, ParamsData } from "./params.ts"; +import type { Method, Static } from "./schema.ts"; +import type { Snapshot } from "./snapshot.ts"; + +export type Kind = "remote" | "local"; + +/** The variant name of a local contract's members. */ +export const LOCAL = "members"; + +/** Everything the types know about a contract: kind, name, params and the whole history. */ +export type ContractShape = { + readonly kind: Kind; + readonly name: string; + readonly params: Params; + readonly history: History; +}; + +/** A declared contract: a branded handle, immutable by convention. Spread or hand-built copies are refused. */ +export interface Contract { + readonly kind: S["kind"]; + readonly name: S["name"]; + readonly version: number; + readonly fingerprint: string; + /** Type only: absent at runtime. */ + readonly "~shape"?: S; +} + +export type ShapeOf = NonNullable; + +/** The variants of a contract's current version. */ +export type CurrentVariants = CurrentStep["history"]>["variants"]; + +/** The params object of a contract. */ +export type ParamsOf = { [F in keyof ShapeOf["params"]]: Static["params"][F]> }; + +/** + * What a factory returns: for `local`, a `Watchable` per state member and a function per method, `(args, context)`, + * or `(context)` when the method takes no arguments; for `remote`, one variant `{ variant, members }` with exactly + * that variant's members. `X` is the context the methods get. + */ +export type Implementation = ImplementationOf< + ShapeOf["kind"], + CurrentVariants, + X +>; + +type ImplementationOf< + K extends Kind, + V extends { readonly [variant: string]: VariantShape }, + X extends Context, +> = K extends "local" + ? VariantImplementation + : { [N in keyof V]: { readonly variant: N; readonly members: VariantImplementation } }[keyof V]; + +/** + * One variant's members: a `Watchable` per state member, a function per method and, in a local contract, a + * `KeyedWatchable` per keyed member. + */ +export type VariantImplementation = { + [M in keyof V["state"] | keyof V["methods"] | keyof NonNullable]: M extends keyof V["methods"] + ? MethodFunction + : M extends keyof V["state"] + ? Watchable> + : M extends keyof NonNullable + ? KeyedWatchable[M]>> + : never; +}; + +/** + * A keyed state member (slice 6a §12): documents by key, each watchable on its own; a keyed document handle + * provides it. `at(key)` is valid only in a factory (it pins the key while that attempt lives). + */ +export interface KeyedWatchable { + /** The document at `key`, `undefined` when absent. */ + get(key: string, context: Context): Promise; + /** A watch of the document at `key` that follows its address, `undefined` when absent. */ + watch(key: string, context: Context): Promise | undefined>; + /** In a factory only: the document at `key` as a watchable, `initial()` while absent. */ + at(key: string): Watchable; +} + +/** A method as a function: without the argument when its argument schema is `null` (`null` stays the wire form). */ +type MethodFunction = M["args"] extends TNull + ? (context: X) => Promise> + : (args: Static, context: X) => Promise>; + +/** The Chord-owned data behind a contract handle. */ +export type ContractData = { + readonly kind: Kind; + readonly name: string; + readonly params: ParamsData; + /** A remote contract's placement: the address of the host serving an object for its params. */ + readonly placement: Placement | undefined; + /** Every version of the history, oldest first; the last is current. */ + readonly versions: readonly VersionData[]; + /** The kind of every member name ever declared in the history. */ + readonly kinds: ReadonlyMap; +}; + +export type MemberKind = "state" | "method" | "keyed"; + +/** A placement as Chord calls it: with checked params. Plugin code: what it returns is checked where it is routed. */ +export type Placement = (params: ParamsOf) => Address; + +export type VersionData = { + readonly version: number; + readonly fingerprint: string; + readonly variants: ReadonlyMap; + /** The changes that produced this version; empty for a base. */ + readonly changes: readonly Change[]; +}; + +export type VariantData = { + readonly state: ReadonlyMap; + readonly methods: ReadonlyMap; + /** Keyed state members (slice 6a §12): only a local contract has any. */ + readonly keyed: ReadonlyMap; +}; + +export type MethodData = { readonly args: Snapshot; readonly result: Snapshot }; + +/** The data behind a contract handle; refuses spread and hand-built copies. */ +export function contractData(contract: Contract): ContractData { + const data = contracts.get(contract); + if (data === undefined) { + throw new Error(`${contract.name}: not a declared contract (spread and hand-built copies are refused)`); + } + return data; +} + +const contracts = new WeakMap(); + +/** Whether a method takes no arguments: its argument schema is `null`, so its functions omit the argument. */ +export function takesNoArgs(method: MethodData): boolean { + // A schema's normal form is an object of keywords. + const normal = method.args.normal as { readonly [keyword: string]: unknown }; + return normal.type === "null" && Object.keys(normal).length === 1; +} + +/** The variants of a contract's current version; a local contract has the one variant `LOCAL`. */ +export function currentVariants(contract: Contract): ReadonlyMap { + return contractData(contract).versions.at(-1)!.variants; +} + +/** A variant's names as links index them (2b): its state members and its methods, each sorted. */ +export type VariantNames = { readonly state: readonly string[]; readonly methods: readonly string[] }; + +/** The current version's variant names in the order links index them: sorted. */ +export function variantOrder(contract: Contract): readonly string[] { + return [...currentVariants(contract).keys()].sort(); +} + +/** A variant's state members and methods in the order links index them: each sorted. */ +export function variantNames(variant: VariantData): VariantNames { + return { state: [...variant.state.keys()].sort(), methods: [...variant.methods.keys()].sort() }; +} + +/** A new contract handle for `data`; the caller states the type-level shape. */ +export function createContract(data: ContractData): C { + const current = data.versions.at(-1)!; + const contract: Contract = { + kind: data.kind, + name: data.name, + version: current.version, + fingerprint: current.fingerprint, + }; + contracts.set(contract, data); + // The shape is type-level only; every caller states the shape it built. + return contract as C; +} diff --git a/packages/chord/src/next/contracts/declare.ts b/packages/chord/src/next/contracts/declare.ts new file mode 100644 index 000000000..2e3856c28 --- /dev/null +++ b/packages/chord/src/next/contracts/declare.ts @@ -0,0 +1,237 @@ +// Declarations: `remote(...)` and `local(...)`, the checks every declared version passes, and the reading of +// untyped declaration input. + +import { + type Contract, + createContract, + type Kind, + LOCAL, + type MemberKind, + type MethodData, + type Placement, + type VariantData, +} from "./contract.ts"; +import { materialize } from "./fingerprint.ts"; +import type { MemberNames, MethodAt, VariantShape } from "./history.ts"; +import type { Address } from "./host-kind.ts"; +import { checkName } from "./names.ts"; +import { type Params, type RequiredFields, readParams } from "./params.ts"; +import { KeyedMember, Method, type Static, type TSchema } from "./schema.ts"; +import { type Snapshot, snapshotArguments, snapshotSchema } from "./snapshot.ts"; + +type VariantDeclaration = { + readonly state?: { readonly [member: string]: TSchema }; + readonly methods?: { readonly [member: string]: Method }; +}; + +type VariantDeclarations = { readonly [variant: string]: VariantDeclaration }; + +type RemoteVariants = { + [V in keyof D]: { + state: D[V]["state"] extends object ? D[V]["state"] : Record; + methods: D[V]["methods"] extends object ? D[V]["methods"] : Record; + }; +}; + +type LocalVariants = { + [LOCAL]: { + state: { [K in keyof M as M[K] extends Method | KeyedMember ? never : K]: Exclude }; + methods: { [K in keyof M as M[K] extends Method ? K : never]: Extract }; + keyed: { [K in keyof M as M[K] extends KeyedMember ? K : never]: SchemaOf }; + }; +}; + +/** A local contract's member: a state schema, `method(...)` or `keyed(schema)`. */ +type LocalMember = TSchema | Method | KeyedMember; + +type SchemaOf = M extends KeyedMember ? S : never; + +/** D11: a member name has one kind in every variant, a method one signature; no keyed members. */ +type SameMembers> = { + [X in keyof D]: { + state?: { + // Keyed members are only in local contracts (slice 6a §12). + [M in keyof D[X]["state"]]: M extends MemberNames + ? never + : D[X]["state"][M] extends KeyedMember + ? never + : D[X]["state"][M]; + }; + methods?: { + [M in keyof D[X]["methods"]]: M extends MemberNames + ? never + : [MethodAt] extends [D[X]["methods"][M]] + ? D[X]["methods"][M] + : never; + }; + }; +}; + +/** Declares a remote contract: per-principal variants of state and methods. */ +export function remote< + const N extends string, + const P extends Params, + const V extends number, + const D extends VariantDeclarations, +>( + name: N, + declaration: { + readonly version: V; + readonly params: P & NoInfer>; + /** The address of the host serving an object for these params; never part of the fingerprint. */ + readonly placement?: (params: { [F in keyof P]: Static }) => Address; + readonly variants: D & NoInfer>; + }, +): Contract<{ + kind: "remote"; + name: N; + params: P; + history: [{ version: `${V}`; variants: RemoteVariants; changes: never }]; +}> { + // Declaration policy: JavaScript callers reach every check below. + checkName(name, "A contract name"); + checkKeys(declaration, ["version", "params", "placement", "variants"], name); + const variants = entriesOf(declaration.variants, `${name}.variants`); + if (variants.length === 0) throw new Error(`${name}.variants: at least one variant is required`); + const { placement } = declaration; + if (placement !== undefined && typeof placement !== "function") { + throw new Error(`${name}.placement: expected a function of the params`); + } + const read = variants.map(([variant, members]) => [variant, readVariant(members, `${name}.${variant}`)] as const); + // The placement gets the checked params of this contract. + return declare("remote", name, declaration, new Map(read), placement as Placement | undefined); +} + +/** + * Declares a local contract: one flat map of members, where a schema is state, `method(...)` a method and + * `keyed(schema)` a keyed state member. + */ +export function local< + const N extends string, + const P extends Params, + const V extends number, + const M extends { readonly [member: string]: LocalMember }, +>( + name: N, + declaration: { readonly version: V; readonly params: P & NoInfer>; readonly members: M }, +): Contract<{ + kind: "local"; + name: N; + params: P; + history: [{ version: `${V}`; variants: LocalVariants; changes: never }]; +}> { + // Declaration policy: JavaScript callers reach every check below. + checkName(name, "A contract name"); + checkKeys(declaration, ["version", "params", "members"], name); + const state = new Map(); + const methods = new Map(); + const keyed = new Map(); + for (const [member, value] of entriesOf(declaration.members, `${name}.${LOCAL}`)) { + const path = `${name}.${LOCAL}.${member}`; + if (value instanceof Method) methods.set(member, readMethod(value, path)); + else if (value instanceof KeyedMember) keyed.set(member, snapshotSchema(value.schema, path)); + else state.set(member, snapshotSchema(value, path)); + } + return declare("local", name, declaration, new Map([[LOCAL, { state, methods, keyed }]])); +} + +function readVariant(source: unknown, path: string): VariantData { + checkKeys(source, ["state", "methods"], path); + const { state = {}, methods = {} } = source as VariantDeclaration; + const stateEntries = entriesOf(state, `${path}.state`).map(([member, schema]) => { + if (schema instanceof Method) throw new Error(`${path}.state.${member}: a method belongs in methods`); + if (schema instanceof KeyedMember) { + throw new Error(`${path}.state.${member}: keyed members are only in local contracts`); + } + return [member, snapshotSchema(schema, `${path}.state.${member}`)] as const; + }); + const methodEntries = entriesOf(methods, `${path}.methods`).map(([member, method]) => { + if (!(method instanceof Method)) throw new Error(`${path}.methods.${member}: expected method(...)`); + return [member, readMethod(method, `${path}.methods.${member}`)] as const; + }); + return { state: new Map(stateEntries), methods: new Map(methodEntries), keyed: new Map() }; +} + +/** One method declaration as Chord-owned snapshots; its arguments must be closed. */ +export function readMethod(method: Method, path: string): MethodData { + return { + args: snapshotArguments(method.args, `${path}.args`), + result: snapshotSchema(method.result, `${path}.result`), + }; +} + +function declare( + kind: Kind, + name: string, + declaration: { readonly version: number; readonly params: unknown }, + variants: ReadonlyMap, + placement?: Placement, +): C { + const { version } = declaration; + if (!Number.isSafeInteger(version) || version < 1) throw new Error(`${name}.version: expected a positive integer`); + const kinds = new Map(); + checkMembers(variants, kinds, (variant, member, memberKind) => memberPath(name, kind, variant, member, memberKind)); + const params = readParams(entriesOf(declaration.params, `${name}.params`), `${name}.params`); + const versions = [materialize(kind, params, version, variants, [])]; + return createContract({ kind, name, params, placement, versions, kinds }); +} + +/** + * Checks the members of every variant: a name keeps one kind across variants and the whole history (`kinds`, updated + * here), a method one signature across variants. + */ +export function checkMembers( + variants: ReadonlyMap, + kinds: Map, + path: (variant: string, member: string, kind: MemberKind) => string, +): void { + const signatures = new Map(); + for (const [variant, { state, methods, keyed }] of variants) { + for (const member of state.keys()) recordKind(kinds, member, "state", path(variant, member, "state")); + for (const member of keyed.keys()) recordKind(kinds, member, "keyed", path(variant, member, "keyed")); + for (const [member, { args, result }] of methods) { + recordKind(kinds, member, "method", path(variant, member, "method")); + const signature = JSON.stringify([args.normal, result.normal]); + if ((signatures.get(member) ?? signature) !== signature) { + throw new Error(`${path(variant, member, "method")}: a method has the same signature in every variant`); + } + signatures.set(member, signature); + } + } +} + +function recordKind(kinds: Map, member: string, kind: MemberKind, path: string): void { + if (RESERVED_MEMBERS.has(member)) throw new Error(`${path}: "${member}" is reserved: a reference has it`); + const known = kinds.get(member); + if (known !== undefined && known !== kind) { + throw new Error(`${path}: "${member}" is a ${known} member; its kind never changes`); + } + kinds.set(member, kind); +} + +/** + * Names a reference has itself (`release`, `variant`, and `state` and `retry` with failures), and `then`, which would + * make an awaited reference thenable. + */ +const RESERVED_MEMBERS: ReadonlySet = new Set(["then", "release", "variant", "state", "retry"]); + +/** The path of a member in errors: `notes.editor.methods.add`, `info.members.clear`. */ +export function memberPath(base: string, kind: Kind, variant: string, member: string, memberKind: MemberKind): string { + if (kind === "local") return `${base}.${LOCAL}.${member}`; + return `${base}.${variant}.${memberKind === "state" ? "state" : "methods"}.${member}`; +} + +/** Refuses a non-object and unknown keys. */ +export function checkKeys(source: unknown, allowed: readonly string[], path: string): void { + for (const [key] of entriesOf(source, path)) { + if (!allowed.includes(key)) throw new Error(`${path}: unknown key "${key}"`); + } +} + +/** The entries of an object; anything else is refused. */ +export function entriesOf(source: unknown, path: string): [string, unknown][] { + if (typeof source !== "object" || source === null || Array.isArray(source)) { + throw new Error(`${path}: expected an object`); + } + return Object.entries(source); +} diff --git a/packages/chord/src/next/contracts/evolve.ts b/packages/chord/src/next/contracts/evolve.ts new file mode 100644 index 000000000..9e405d732 --- /dev/null +++ b/packages/chord/src/next/contracts/evolve.ts @@ -0,0 +1,310 @@ +// Evolution: `evolve(previous, (e) => e...)` appends the next version of a history with structured edits. + +import { + type Contract, + type ContractData, + type ContractShape, + type CurrentVariants, + contractData, + createContract, + type Kind, + LOCAL, + type MemberKind, + type MethodData, + type ShapeOf, +} from "./contract.ts"; +import { checkMembers, memberPath, readMethod } from "./declare.ts"; +import { materialize } from "./fingerprint.ts"; +import type { Change, CurrentStep, Increment, MemberNames, MethodAt, Variants, VariantsWith } from "./history.ts"; +import { Method, type TSchema } from "./schema.ts"; +import { type Snapshot, snapshotArguments, snapshotSchema } from "./snapshot.ts"; + +/** Method edits, the same for both kinds: each changes the method in every variant that has it. */ +interface MethodEditor { + /** Type only: the variants after the edits so far. */ + readonly "~variants"?: V; + /** Type only: the changes made by the edits so far. */ + readonly "~changes"?: C; + args, A extends TSchema>( + member: M, + args: A, + ): Editor< + K, + Put, "methods", M, Method["result"]>>, + C | [VariantsWith, M, "args"] + >; + result, R extends TSchema>( + member: M, + result: R, + ): Editor< + K, + Put, "methods", M, Method["args"], R>>, + C | [VariantsWith, M, "result"] + >; + method, A extends TSchema, R extends TSchema>( + member: M, + args: A, + result: R, + ): Editor, "methods", M, Method>, C | [VariantsWith, M, "args" | "result"]>; + removeMethod>(member: M): Editor, C>; +} + +/** Edits of a remote contract. */ +interface RemoteEditor extends MethodEditor<"remote", V, C> { + state( + variant: X, + member: M, + schema: S, + ): RemoteEditor, C | [X, M, "state"]>; + addState( + variant: X, + member: M, + schema: S, + ): RemoteEditor, C | [X, M, "state"]>; + removeState( + variant: X, + member: M, + ): RemoteEditor, C>; + addMethod( + variant: X, + member: M, + args: A, + result: R, + ): RemoteEditor>, C | [X, M, "args" | "result"]>; +} + +/** Edits of a local contract: like remote ones, without a variant. */ +interface LocalEditor extends MethodEditor<"local", V, C> { + state( + member: M, + schema: S, + ): LocalEditor, C | [Local, M, "state"]>; + addState( + member: M, + schema: S, + ): LocalEditor, C | [Local, M, "state"]>; + removeState(member: M): LocalEditor, C>; + addMethod( + member: M, + args: A, + result: R, + ): LocalEditor>, C | [Local, M, "args" | "result"]>; +} + +type Local = typeof LOCAL; + +type Editor = K extends "remote" + ? RemoteEditor + : LocalEditor; + +type EditorOf = Editor["kind"], CurrentVariants, never>; + +/** Variants `X` with member `M` of `Side` set to `T`. */ +type Put = { + [Y in keyof V]: Y extends X + ? { + [S in keyof V[Y]]: S extends Side ? { [K in keyof V[Y][S] | M]: K extends M ? T : V[Y][S][K] } : V[Y][S]; + } + : V[Y]; +}; + +/** Variants `X` without member `M` of `Side`. */ +type Drop = { + [Y in keyof V]: Y extends X ? { [S in keyof V[Y]]: S extends Side ? Omit : V[Y][S] } : V[Y]; +}; + +/** The sides of a variant that edits change; keyed members are carried over unchanged (their versions: slice 7). */ +type EditedSide = "state" | "methods"; + +type Evolved> = { + kind: S["kind"]; + name: S["name"]; + params: S["params"]; + history: [ + ...S["history"], + { + version: Increment["version"]>; + variants: NonNullable; + changes: NonNullable; + }, + ]; +}; + +/** + * Appends the next version (`previous.version + 1`) with the edits `edit` makes; `edit` must return the editor it was + * given. Params and the variant set never change: a change to either is a new history base. + */ +export function evolve["kind"], Variants, Change>>( + previous: C, + edit: (editor: EditorOf) => E, +): Contract, E>> { + const draft = new Draft(contractData(previous)); + const editor = draft.kind === "local" ? localEditor(draft) : draft; + // The editor interfaces track the edits in the types; at runtime every edit returns the same editor. + editWith(editor as unknown as EditorOf, edit, `${draft.path}: evolve`); + return createContract(draft.next()); +} + +/** + * Runs `edit` on `editor` (contracts' and tables' rule): an edit returning anything but the editor it was given throws + * (`what` names the call). + */ +export function editWith(editor: E, edit: (editor: E) => unknown, what: string): void { + if (edit(editor) !== editor) throw new Error(`${what} must return the editor it was given`); +} + +/** A variant under construction; keyed members are carried over unchanged (their versions are slice 7's). */ +type DraftVariant = { + readonly state: Map; + readonly methods: Map; + readonly keyed: ReadonlyMap; +}; + +/** The next version under construction, and the remote editor. Every edit is checked as it is made. */ +class Draft { + readonly kind: Kind; + readonly path: string; + /** What edits return: the draft itself, or the local editor wrapping it. */ + editor: object = this; + readonly #data: ContractData; + readonly #version: number; + readonly #variants: Map; + readonly #kinds: Map; + readonly #changes: Change[] = []; + + constructor(data: ContractData) { + const current = data.versions.at(-1)!; + this.kind = data.kind; + this.#data = data; + this.#version = current.version + 1; + this.path = `${data.name} v${this.#version}`; + this.#variants = new Map( + [...current.variants].map(([name, { state, methods, keyed }]) => [ + name, + { state: new Map(state), methods: new Map(methods), keyed }, + ]), + ); + this.#kinds = new Map(data.kinds); + } + + state(variant: string, member: string, schema: TSchema): object { + if (!this.#variant(variant).state.has(member)) { + throw new Error(`${this.#path(variant, member, "state")}: no such state member`); + } + return this.#putState(variant, member, schema); + } + + addState(variant: string, member: string, schema: TSchema): object { + this.#refuseDeclared(variant, member, "state"); + return this.#putState(variant, member, schema); + } + + removeState(variant: string, member: string): object { + if (!this.#variant(variant).state.delete(member)) { + throw new Error(`${this.#path(variant, member, "state")}: no such state member`); + } + return this.editor; + } + + args(member: string, args: TSchema): object { + return this.#changeMethod(member, args, undefined); + } + + result(member: string, result: TSchema): object { + return this.#changeMethod(member, undefined, result); + } + + method(member: string, args: TSchema, result: TSchema): object { + return this.#changeMethod(member, args, result); + } + + addMethod(variant: string, member: string, args: TSchema, result: TSchema): object { + this.#refuseDeclared(variant, member, "method"); + this.#variant(variant).methods.set( + member, + readMethod(new Method(args, result), this.#path(variant, member, "method")), + ); + this.#changes.push([variant, member, "args"], [variant, member, "result"]); + return this.#checked(); + } + + removeMethod(member: string): object { + for (const variant of this.#variantsWith(member)) this.#variant(variant).methods.delete(member); + return this.editor; + } + + /** The contract data with this version appended. */ + next(): ContractData { + const { kind, params, versions } = this.#data; + const version = materialize(kind, params, this.#version, this.#variants, this.#changes); + return { ...this.#data, versions: [...versions, version], kinds: this.#kinds }; + } + + #putState(variant: string, member: string, schema: TSchema): object { + this.#variant(variant).state.set(member, snapshotSchema(schema, this.#path(variant, member, "state"))); + this.#changes.push([variant, member, "state"]); + return this.#checked(); + } + + /** Changes the given sides of a method in every variant that has it. */ + #changeMethod(member: string, args: TSchema | undefined, result: TSchema | undefined): object { + const variants = this.#variantsWith(member); + const path = this.#path(variants[0]!, member, "method"); + const newArgs = args === undefined ? undefined : snapshotArguments(args, `${path}.args`); + const newResult = result === undefined ? undefined : snapshotSchema(result, `${path}.result`); + for (const variant of variants) { + const { methods } = this.#variant(variant); + const old = methods.get(member)!; + methods.set(member, { args: newArgs ?? old.args, result: newResult ?? old.result }); + if (newArgs !== undefined) this.#changes.push([variant, member, "args"]); + if (newResult !== undefined) this.#changes.push([variant, member, "result"]); + } + return this.#checked(); + } + + #variant(variant: string): DraftVariant { + const found = this.#variants.get(variant); + if (found === undefined) { + throw new Error(`${this.path}.${variant}: unknown variant (the variant set never changes)`); + } + return found; + } + + #variantsWith(member: string): string[] { + const variants = [...this.#variants].filter(([, { methods }]) => methods.has(member)).map(([variant]) => variant); + if (variants.length === 0) throw new Error(`${this.path}: no method "${member}"`); + return variants; + } + + #refuseDeclared(variant: string, member: string, kind: MemberKind): void { + const { state, methods } = this.#variant(variant); + if (state.has(member) || methods.has(member)) { + throw new Error(`${this.#path(variant, member, kind)}: already declared`); + } + } + + #path(variant: string, member: string, kind: MemberKind): string { + return memberPath(this.path, this.kind, variant, member, kind); + } + + /** A member's kind never changes, also not after a removal; a method keeps one signature across variants. */ + #checked(): object { + checkMembers(this.#variants, this.#kinds, (variant, member, kind) => this.#path(variant, member, kind)); + return this.editor; + } +} + +/** The local editor: the draft's edits with the variant filled in. */ +function localEditor(draft: Draft): object { + draft.editor = { + state: (member: string, schema: TSchema) => draft.state(LOCAL, member, schema), + addState: (member: string, schema: TSchema) => draft.addState(LOCAL, member, schema), + removeState: (member: string) => draft.removeState(LOCAL, member), + addMethod: (member: string, args: TSchema, result: TSchema) => draft.addMethod(LOCAL, member, args, result), + args: (member: string, args: TSchema) => draft.args(member, args), + result: (member: string, result: TSchema) => draft.result(member, result), + method: (member: string, args: TSchema, result: TSchema) => draft.method(member, args, result), + removeMethod: (member: string) => draft.removeMethod(member), + }; + return draft.editor; +} diff --git a/packages/chord/src/next/contracts/fingerprint.ts b/packages/chord/src/next/contracts/fingerprint.ts new file mode 100644 index 000000000..2cce1b7f0 --- /dev/null +++ b/packages/chord/src/next/contracts/fingerprint.ts @@ -0,0 +1,69 @@ +// Fingerprints: the normal form of a contract version and its portable 64-bit hash. Not a security boundary. + +import type { JsonValue } from "../../json.ts"; +import { type Kind, LOCAL, type VariantData, type VersionData } from "./contract.ts"; +import type { Change } from "./history.ts"; +import type { ParamsData } from "./params.ts"; + +/** Version of the normal form; changing the normal form changes it and is a link-protocol change. */ +export const NORMAL_FORM_VERSION = 2; + +/** + * Materializes a version with its fingerprint: `hash64` of `JSON.stringify({ normalFormVersion, value })` over the + * normal form `{ kind: "remote", params, variants }` or `{ kind: "local", params, members }`. + */ +export function materialize( + kind: Kind, + params: ParamsData, + version: number, + variants: ReadonlyMap, + changes: readonly Change[], +): VersionData { + const normal: JsonValue = + kind === "local" + ? { kind, params: params.normal, members: variantNormal(variants.get(LOCAL)!) } + : { + kind, + params: params.normal, + variants: sortedDict([...variants].map(([name, v]) => [name, variantNormal(v)])), + }; + return { version, fingerprint: fingerprintOf(normal), variants, changes }; +} + +/** `hash64` of `JSON.stringify({ normalFormVersion, value: normal })`: the fingerprint of a normal form. */ +export function fingerprintOf(normal: JsonValue): string { + return hash64(JSON.stringify({ normalFormVersion: NORMAL_FORM_VERSION, value: normal })); +} + +/** + * Members as a sorted dictionary of `{ state }`, `{ args, result }` or `{ keyed }` (slice 6a §12: a node of its own, + * so contracts without keyed members keep their fingerprints and `NORMAL_FORM_VERSION` stays). + */ +function variantNormal({ state, methods, keyed }: VariantData): JsonValue { + return sortedDict([ + ...[...state].map(([member, schema]) => [member, { state: schema.normal }] as const), + ...[...keyed].map(([member, schema]) => [member, { keyed: schema.normal }] as const), + ...[...methods].map( + ([member, { args, result }]) => [member, { args: args.normal, result: result.normal }] as const, + ), + ]); +} + +function sortedDict(entries: (readonly [string, JsonValue])[]): JsonValue { + return Object.fromEntries(entries.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))); +} + +/** 16 hex characters from two 32-bit FNV-1a lanes (different offset bases and primes) over the UTF-8 bytes. */ +export function hash64(text: string): string { + let low = 0x811c9dc5; + let high = 0xcbf29ce4; + for (const byte of new TextEncoder().encode(text)) { + low = Math.imul(low ^ byte, 0x01000193); + high = Math.imul(high ^ byte, 0x01000197); + } + return hex(high) + hex(low); +} + +function hex(lane: number): string { + return (lane >>> 0).toString(16).padStart(8, "0"); +} diff --git a/packages/chord/src/next/contracts/history.ts b/packages/chord/src/next/contracts/history.ts new file mode 100644 index 000000000..24c0d7f33 --- /dev/null +++ b/packages/chord/src/next/contracts/history.ts @@ -0,0 +1,66 @@ +// The type-level history of a contract: every version's members and the changes that produced it. + +import type { Method, TSchema } from "./schema.ts"; + +/** One variant's members at one version: state schemas and method signatures. */ +export type VariantShape = { + readonly state: { readonly [member: string]: TSchema }; + readonly methods: { readonly [member: string]: Method }; + /** A local contract's keyed state members (slice 6a §12). */ + readonly keyed?: { readonly [member: string]: TSchema }; +}; + +/** Variants by name. A local contract has the single variant `members`. */ +export type Variants = { readonly [variant: string]: VariantShape }; + +/** What one edit changed: one side of one member in one variant (a union of variants when it changed several). */ +export type Change = readonly [variant: string, member: string, side: "state" | "args" | "result"]; + +/** One version: its number as digits, its variants, and the changes that produced it (`never` for a base). */ +type Step = { readonly version: string; readonly variants: Variants; readonly changes: Change }; + +/** Every version from the base, oldest first. */ +export type History = readonly Step[]; + +export type CurrentStep = H extends readonly [...Step[], infer Last extends Step] ? Last : never; + +export type StepAt = Extract; + +/** Versions before the current one. */ +export type OlderVersions = H extends readonly [...infer Older extends Step[], Step] + ? Older[number]["version"] + : never; + +/** Every change made after version `V`. */ +export type ChangesAfter = H extends readonly [ + infer First extends Step, + ...infer Rest extends Step[], +] + ? First["version"] extends V + ? Rest[number]["changes"] + : ChangesAfter + : never; + +/** Names of the state members or methods in any variant. */ +export type MemberNames = { + [X in keyof V]: keyof V[X][Side]; +}[keyof V] & + string; + +/** The variants that have method `M`. */ +export type VariantsWith = { + [X in keyof V]: M extends keyof V[X]["methods"] ? X : never; +}[keyof V] & + string; + +/** Method `M` (one signature in every variant that has it). */ +export type MethodAt = { + [X in keyof V]: M extends keyof V[X]["methods"] ? V[X]["methods"][M] : never; +}[keyof V]; + +/** `N + 1` on decimal digits, so versions beyond 9 and 99 stay literal. */ +export type Increment = N extends `${infer Init}9` + ? `${Init extends "" ? "1" : Increment}0` + : { [D in keyof NextDigit]: N extends `${infer Init}${D}` ? `${Init}${NextDigit[D]}` : never }[keyof NextDigit]; + +type NextDigit = { "0": "1"; "1": "2"; "2": "3"; "3": "4"; "4": "5"; "5": "6"; "6": "7"; "7": "8"; "8": "9" }; diff --git a/packages/chord/src/next/contracts/host-kind.ts b/packages/chord/src/next/contracts/host-kind.ts new file mode 100644 index 000000000..3f0e61055 --- /dev/null +++ b/packages/chord/src/next/contracts/host-kind.ts @@ -0,0 +1,68 @@ +// Host kinds and addresses (§3): a kind of addressable host with params like a contract's, and the address of one +// host of a kind, which a remote contract's placement computes from its params. + +import { checkName } from "./names.ts"; +import { joinKey, type Params, type ParamsData, type RequiredFields, readParams } from "./params.ts"; +import type { Static } from "./schema.ts"; + +/** What an address's type needs of its kind: the name and the params, type only. */ +export type KindShape = { readonly kind: string; readonly "~params"?: Params }; + +/** One host of a kind: the kind's name, its params and its key (`session|S1`); two addresses are equal by key. */ +export type Address = { + readonly kind: K["kind"]; + readonly params: ParamsOfKind; + readonly key: string; +}; + +/** A kind of addressable host: called with its params, it makes an address. */ +export interface HostKind { + ( + ...params: keyof P extends never ? [params?: Record] : [params: { [F in keyof P]: Static }] + ): Address>; + readonly kind: N; + /** Type only: absent at runtime. */ + readonly "~params"?: P; +} + +/** The params object of a host kind. */ +export type ParamsOfKind = { + [F in keyof NonNullable]: Static[F]>; +}; + +/** + * Declares a host kind; `params` follow the params policy of contracts (a flat record of required `str()`, `int()`, + * `bool()` and `oneOf(...)`). Making an address computes its key only: its params are checked where it is routed. + */ +export function hostKind>( + name: N, + params?: P & NoInfer>, +): HostKind { + checkName(name, "A host kind's name"); + // `client` is the host kind of a host without an address in store names (owner strings, table names). + if (name === "client") throw new Error('A host kind\'s name "client" is reserved for hosts without an address'); + const data = readParams(Object.entries(params ?? {}), `${name}.params`); + const make = (values: { readonly [field: string]: unknown } = {}) => ({ + kind: name, + params: values, + key: addressKey(name, data, values), + }); + Object.defineProperty(make, "kind", { value: name, enumerable: true }); + kinds.set(make, data); + // The function and its `kind` are the declared shape; the params type is type-level only. + return make as unknown as HostKind; +} + +const kinds = new WeakMap(); + +/** The params of a declared host kind. */ +export function kindParams(kind: { readonly kind: string }): ParamsData { + const data = kinds.get(kind); + if (data === undefined) throw new Error(`${kind.kind}: not a declared host kind`); + return data; +} + +/** An address's key: the kind's name and each field value in declared order, joined (`joinKey`). */ +export function addressKey(kind: string, params: ParamsData, values: { readonly [field: string]: unknown }): string { + return joinKey([kind, ...params.fields.map((field) => String(values[field]))]); +} diff --git a/packages/chord/src/next/contracts/index.ts b/packages/chord/src/next/contracts/index.ts new file mode 100644 index 000000000..299bac9d3 --- /dev/null +++ b/packages/chord/src/next/contracts/index.ts @@ -0,0 +1,30 @@ +// Contracts for apps: declaring, evolving and implementing Chord object types. + +export type { AcceptsTable } from "./accepts.ts"; +export type { Contract, Implementation, KeyedWatchable, ParamsOf } from "./contract.ts"; +export { local, remote } from "./declare.ts"; +export { evolve } from "./evolve.ts"; +export { type Address, type HostKind, hostKind, type KindShape, type ParamsOfKind } from "./host-kind.ts"; +export { + arr, + type Bounds, + bool, + bytes, + type Defaulted, + int, + type KeyedMember, + keyed, + lit, + map, + method, + nul, + num, + obj, + oneOf, + opt, + type Static, + str, + type TSchema, + union, + unknown, +} from "./schema.ts"; diff --git a/packages/chord/src/next/contracts/internal.ts b/packages/chord/src/next/contracts/internal.ts new file mode 100644 index 000000000..9f43298fe --- /dev/null +++ b/packages/chord/src/next/contracts/internal.ts @@ -0,0 +1,25 @@ +// Contracts for Chord's layers: accepted versions, the data behind a contract handle, and the types L4 dispatches on. + +export { checkAccepts } from "./accepts.ts"; +export { + type ContractShape, + type CurrentVariants, + contractData, + currentVariants, + LOCAL, + type MethodData, + type ShapeOf, + takesNoArgs, + type VariantData, + type VariantImplementation, + type VariantNames, + variantNames, + variantOrder, +} from "./contract.ts"; +export { editWith } from "./evolve.ts"; +export { fingerprintOf, hash64, NORMAL_FORM_VERSION } from "./fingerprint.ts"; +export type { Increment } from "./history.ts"; +export { kindParams } from "./host-kind.ts"; +export { checkName } from "./names.ts"; +export type { Params, ParamsData } from "./params.ts"; +export { Method } from "./schema.ts"; diff --git a/packages/chord/src/next/contracts/names.ts b/packages/chord/src/next/contracts/names.ts new file mode 100644 index 000000000..e31294039 --- /dev/null +++ b/packages/chord/src/next/contracts/names.ts @@ -0,0 +1,14 @@ +// Names that become SQL names or stored identities in the store (decision "Restricted names"): object types, host +// kinds, tables, state definitions and jobs use npm package-name characters, lowercase, so `::` is injective and readable, and a document's or a job record's name is stored as declared. + +/** Lowercase npm package-name characters: `:` cannot occur, so names joined with it stay apart. */ +const NAME = /^[a-z0-9@/._~-]+$/; + +/** Throws unless `name` is a nonempty lowercase name of `[a-z0-9@/._~-]` (`what` names it in the error). */ +export function checkName(name: string, what: string): void { + // Declaration policy: JavaScript callers reach this check. + if (typeof name !== "string" || !NAME.test(name)) { + throw new Error(`${what} ${JSON.stringify(name)}: expected lowercase [a-z0-9@/._~-]`); + } +} diff --git a/packages/chord/src/next/contracts/params.ts b/packages/chord/src/next/contracts/params.ts new file mode 100644 index 000000000..565c88500 --- /dev/null +++ b/packages/chord/src/next/contracts/params.ts @@ -0,0 +1,77 @@ +// Params declarations: the policy for params fields and the closed object schema checked on every get. + +import type { TBoolean, TInteger, TLiteral, TString, TUnion } from "typebox"; +import type { JsonValue } from "../../json.ts"; +import type { TSchema } from "./schema.ts"; +import { snapshotSchema } from "./snapshot.ts"; + +/** A params field: a string, an integer, a boolean or a union of string literals; constraints allowed. */ +type ParamSchema = TString | TInteger | TBoolean | TUnion[]>; + +/** Params: a flat record of required fields, in identity order. */ +export type Params = { readonly [field: string]: ParamSchema }; + +/** Refuses optional params fields, which the field types alone admit. */ +export type RequiredFields

= { + [F in keyof P]: P[F] extends { readonly "~optional": true } ? never : P[F]; +}; + +export type ParamsData = { + /** Field names in identity order. */ + readonly fields: readonly string[]; + /** The closed object schema of all fields. */ + readonly schema: TSchema; + /** `[field, normal form]` pairs in identity order. */ + readonly normal: JsonValue; +}; + +/** Reads declared params fields: each a required str(), int(), bool() or oneOf(...) of strings, constraints allowed. */ +export function readParams(declared: readonly [string, unknown][], path: string): ParamsData { + const fields = declared.map(([field, schema]) => { + const fieldPath = `${path}.${field}`; + const snapshot = snapshotSchema(schema, fieldPath); + if (Object.hasOwn(schema as object, "~optional") || !isParamSchema(snapshot.normal)) { + throw new Error(`${fieldPath}: a param is a required str(), int(), bool() or oneOf(...) of strings`); + } + return [field, snapshot] as const; + }); + const schema = { + type: "object", + properties: Object.fromEntries(fields.map(([field, snapshot]) => [field, snapshot.schema])), + required: fields.map(([field]) => field), + additionalProperties: false, + }; + return { + fields: fields.map(([field]) => field), + schema, + normal: fields.map(([field, snapshot]) => [field, snapshot.normal]), + }; +} + +/** Joins key parts with `|`, each escaped (`\\` and `|`), so two different part lists never join to one key. */ +export function joinKey(parts: readonly string[]): string { + return parts.map((part) => part.replaceAll("\\", "\\\\").replaceAll("|", "\\|")).join("|"); +} + +type SchemaNode = { readonly [keyword: string]: JsonValue }; + +function isParamSchema(normal: JsonValue): boolean { + const node = normal as SchemaNode; + if (node.anyOf !== undefined) { + return Object.keys(node).length === 1 && Array.isArray(node.anyOf) && node.anyOf.every(isStringLiteral); + } + const allowed = PARAM_KEYWORDS.get(node.type); + return allowed !== undefined && Object.keys(node).every((key) => allowed.has(key)); +} + +/** Keywords a params field may use, by type. */ +const PARAM_KEYWORDS = new Map>([ + ["string", new Set(["type", "minLength", "maxLength", "pattern", "format"])], + ["integer", new Set(["type", "minimum", "maximum", "exclusiveMinimum", "exclusiveMaximum", "multipleOf"])], + ["boolean", new Set(["type"])], +]); + +function isStringLiteral(branch: JsonValue): boolean { + const node = branch as SchemaNode; + return node.type === "string" && typeof node.const === "string" && Object.keys(node).length === 2; +} diff --git a/packages/chord/src/next/contracts/schema.ts b/packages/chord/src/next/contracts/schema.ts new file mode 100644 index 000000000..fc987dbb2 --- /dev/null +++ b/packages/chord/src/next/contracts/schema.ts @@ -0,0 +1,180 @@ +// Schema helpers: short constructors for TypeBox schemas, the bytes leaf, and method declarations. + +import Type, { + type TArray, + type TBoolean, + type TInteger, + type TLiteral, + type TNull, + type TNumber, + type TObject, + type TOptional, + type TProperties, + type TRecord, + type TSchema, + type TString, + type TUnion, + type TUnknown, + type TUnsafe, +} from "typebox"; + +export type { Static, TSchema } from "typebox"; + +/** Bounds of a string length, a number or an array length. */ +export type Bounds = { readonly min?: number; readonly max?: number }; + +/** A declared default value: a table column with one may be left out of a `put` (slice 6a §5). */ +export type Defaulted = { readonly default: T }; + +/** A string, with optional length bounds, pattern and default. */ +export function str( + options: Bounds & { readonly pattern?: string; readonly default: string }, +): TString & Defaulted; +export function str(options?: Bounds & { readonly pattern?: string }): TString; +export function str(options: Bounds & { readonly pattern?: string; readonly default?: string } = {}): TString { + const pattern = options.pattern === undefined ? {} : { pattern: options.pattern }; + return Type.String({ ...limits(options, "minLength", "maxLength"), ...pattern, ...defaultOf(options) }); +} + +/** A finite number within optional bounds, with an optional default. */ +export function num(options: Bounds & { readonly default: number }): TNumber & Defaulted; +export function num(options?: Bounds): TNumber; +export function num(options: Bounds & { readonly default?: number } = {}): TNumber { + return Type.Number({ ...limits(options, "minimum", "maximum"), ...defaultOf(options) }); +} + +/** An integer within optional bounds, with an optional default. */ +export function int(options: Bounds & { readonly default: number }): TInteger & Defaulted; +export function int(options?: Bounds): TInteger; +export function int(options: Bounds & { readonly default?: number } = {}): TInteger { + return Type.Integer({ ...limits(options, "minimum", "maximum"), ...defaultOf(options) }); +} + +/** A boolean, with an optional default. */ +export function bool(options: { readonly default: boolean }): TBoolean & Defaulted; +export function bool(options?: Record): TBoolean; +export function bool(options: { readonly default?: boolean } = {}): TBoolean { + return Type.Boolean(defaultOf(options)); +} + +/** The `default` keyword of `options`, left out when absent (snapshots refuse `undefined`). */ +function defaultOf(options: { readonly default?: string | number | boolean }): { default?: string | number | boolean } { + return options.default === undefined ? {} : { default: options.default }; +} + +/** `null`; also the omitted side of a method. */ +export function nul(): TNull { + return Type.Null(); +} + +/** Exactly the given value. */ +export function lit(value: V): TLiteral { + return Type.Literal(value); +} + +/** One of the given literals. */ +export function oneOf(...values: V): TUnion> { + // One literal schema per value, in order. + return Type.Union(values.map((value) => Type.Literal(value))) as TUnion>; +} + +type Literal = string | number | boolean; + +type Literals = V extends [infer First extends Literal, ...infer Rest extends Literal[]] + ? [TLiteral, ...Literals] + : []; + +/** Any of the given schemas. */ +export function union(schemas: [...S]): TUnion { + return Type.Union(schemas); +} + +/** A closed object: unknown fields are refused. */ +export function obj

(properties: P): TObject

{ + return Type.Object(properties, { additionalProperties: false }); +} + +/** An array, with optional length bounds. */ +export function arr(items: S, options: Bounds = {}): TArray { + return Type.Array(items, limits(options, "minItems", "maxItems")); +} + +/** A string-keyed record. */ +export function map(value: S): TRecord<"^.*$", S> { + return Type.Record(Type.String(), value); +} + +/** An optional object field. */ +export function opt(schema: S): TOptional { + return Type.Optional(schema) as TOptional; +} + +/** Any value. */ +export function unknown(): TUnknown { + return Type.Unknown(); +} + +/** + * A `Uint8Array` leaf, checked by `instanceof` and length, never enumerated. Its enumerable form is the data marker + * `{ type: "bytes", minLength?, maxLength? }`. + */ +export function bytes(options: Bounds = {}): TUnsafe { + return withBytesCheck({ type: "bytes", ...limits(options, "minLength", "maxLength") }); +} + +/** The data marker of a bytes schema. */ +export type BytesMarker = { readonly type: "bytes"; readonly minLength?: number; readonly maxLength?: number }; + +/** Attaches the TypeBox check of a bytes marker, non-enumerably so it is never copied as data. */ +export function withBytesCheck(marker: BytesMarker): TUnsafe { + const { minLength = 0, maxLength = Number.POSITIVE_INFINITY } = marker; + const refinement = { + check: (value: unknown) => value instanceof Uint8Array && value.length >= minLength && value.length <= maxLength, + error: () => "expected bytes of the declared length", + }; + Object.defineProperty(marker, "~refine", { value: [refinement] }); + // The marker object is the schema; TypeBox reads the attached check. + return marker as unknown as TUnsafe; +} + +/** TypeBox options for the given bounds; an absent bound is left out, since snapshots refuse `undefined`. */ +function limits(options: Bounds, min: string, max: string): { [keyword: string]: number } { + const result: { [keyword: string]: number } = {}; + if (options.min !== undefined) result[min] = options.min; + if (options.max !== undefined) result[max] = options.max; + return result; +} + +/** A method declaration: its argument and result schemas; an omitted side is `null` on the wire. */ +export class Method { + readonly args: A; + readonly result: R; + + constructor(args: A, result: R) { + this.args = args; + this.result = result; + } +} + +/** A keyed state member's declaration (slice 6a §12): documents of `schema` by string key; only in `local` contracts. */ +export class KeyedMember { + readonly schema: S; + + constructor(schema: S) { + this.schema = schema; + } +} + +/** Declares a keyed state member of a `local` contract: documents of `schema` by string key, each watchable. */ +export function keyed(schema: S): KeyedMember { + return new KeyedMember(schema); +} + +/** Declares a method: `method()`, `method(args)`, `method(args, result)` or `method(undefined, result)`. */ +export function method(): Method; +export function method(args: A): Method; +export function method(args: undefined, result: R): Method; +export function method(args: A, result: R): Method; +export function method(args?: TSchema, result?: TSchema): Method { + return new Method(args ?? nul(), result ?? nul()); +} diff --git a/packages/chord/src/next/contracts/snapshot.ts b/packages/chord/src/next/contracts/snapshot.ts new file mode 100644 index 000000000..a84ed79c0 --- /dev/null +++ b/packages/chord/src/next/contracts/snapshot.ts @@ -0,0 +1,206 @@ +// Schema snapshots: one schema-aware walk checks an app schema against Chord's vocabulary and produces a +// Chord-owned copy and its normal form (the fingerprint input). + +import { isJsonValue, type JsonValue } from "../../json.ts"; +import { type BytesMarker, type TSchema, withBytesCheck } from "./schema.ts"; + +/** + * A Chord-owned copy of an app schema, its normal form, and whether it has a `bytes()` leaf (values crossing a link + * then carry raw byte segments); immutable by convention. + */ +export type Snapshot = { readonly schema: TSchema; readonly normal: JsonValue; readonly bytes: boolean }; + +/** Snapshot of a state, result or params schema. Throws an Error naming `path` for anything outside the vocabulary. */ +export function snapshotSchema(source: unknown, path: string): Snapshot { + return walk(source, path, "open", new Set()); +} + +/** Snapshot of a method's arguments: every object schema inside must also be closed (records allowed). */ +export function snapshotArguments(source: unknown, path: string): Snapshot { + return walk(source, path, "closed", new Set()); +} + +/** Whether object schemas must set `additionalProperties: false`. */ +type Objects = "open" | "closed"; + +/** A copied keyword value, its normal form, and whether a schema inside it is a bytes leaf. */ +type Copy = { readonly schema: unknown; readonly normal: JsonValue; readonly bytes: boolean }; + +/** Every supported keyword, grouped by how its value is walked. */ +const KEYWORD_GROUPS = { + schema: "additionalProperties additionalItems", + schemas: "anyOf allOf prefixItems", + items: "items", + schemaMap: "properties patternProperties $defs", + type: "type", + required: "required", + enum: "enum", + data: + "const default $ref $id pattern format minLength maxLength minimum maximum exclusiveMinimum exclusiveMaximum " + + "multipleOf minItems maxItems uniqueItems minProperties maxProperties readOnly writeOnly", + annotation: "title description examples $comment deprecated", +}; + +type Keyword = keyof typeof KEYWORD_GROUPS; + +const KEYWORDS = new Map( + Object.entries(KEYWORD_GROUPS).flatMap(([keyword, names]) => + names.split(" ").map((name) => [name, keyword as Keyword]), + ), +); + +const TYPES = new Set(["null", "boolean", "number", "integer", "string", "array", "object", "bytes"]); + +/** TypeBox's executable features, attached non-enumerably. */ +const EXECUTABLE = ["~refine", "~codec", "~unsafe"]; + +const BYTES_EXECUTABLE = ["~refine", "~unsafe"]; + +const BYTES_KEYWORDS = new Set(["type", "minLength", "maxLength"]); + +function walk(source: unknown, path: string, objects: Objects, ancestors: Set): Snapshot { + if (typeof source === "boolean") return { schema: source, normal: source, bytes: false }; + if (!isPlainObject(source)) throw new Error(`${path}: expected a schema`); + if (ancestors.has(source)) throw new Error(`${path}: cyclic schema; use $defs and $ref for recursion`); + const isBytes = source.type === "bytes"; + // A bytes schema may carry its own check (rebuilt below); no other schema may carry an executable feature. + const allowed = isBytes ? BYTES_EXECUTABLE : []; + for (const key of EXECUTABLE) { + if (Object.hasOwn(source, key) && !allowed.includes(key)) { + throw new Error(`${path}: TypeBox refinements, codecs and unsafe types are not supported`); + } + } + ancestors.add(source); + const schema: [string, unknown][] = []; + const normal: [string, JsonValue][] = []; + let bytes = isBytes; + for (const [key, value] of ownEntries(source, path)) { + const keyword = KEYWORDS.get(key); + if (keyword === undefined || (isBytes && keyword !== "annotation" && !BYTES_KEYWORDS.has(key))) { + throw new Error(`${path}: unsupported keyword "${key}"`); + } + const copy = walkKeyword(keyword, value, `${path}.${key}`, objects, ancestors); + bytes ||= copy.bytes; + schema.push([key, copy.schema]); + // Annotations are dropped on schema nodes only; inside data they are ordinary keys. + if (keyword !== "annotation") normal.push([key, copy.normal]); + } + ancestors.delete(source); + const node = Object.fromEntries(schema); + if (objects === "closed" && isOpenObject(node)) { + throw new Error(`${path}: argument objects must be closed (additionalProperties: false)`); + } + if (isBytes) checkBytesBounds(node, path); + return { + schema: isBytes ? withBytesCheck(node as BytesMarker) : node, + normal: Object.fromEntries(normal), + bytes, + }; +} + +function walkKeyword(keyword: Keyword, value: unknown, path: string, objects: Objects, ancestors: Set): Copy { + switch (keyword) { + case "schema": + return walk(value, path, objects, ancestors); + case "schemas": + return walkList(value, path, objects, ancestors); + case "items": + return Array.isArray(value) + ? walkList(value, path, objects, ancestors) + : walk(value, path, objects, ancestors); + case "schemaMap": { + if (!isPlainObject(value)) throw new Error(`${path}: expected an object of schemas`); + const entries = ownEntries(value, path).map( + ([key, item]) => [key, walk(item, `${path}.${key}`, objects, ancestors)] as const, + ); + return { + schema: Object.fromEntries(entries.map(([key, item]) => [key, item.schema])), + normal: Object.fromEntries(entries.map(([key, item]) => [key, item.normal])), + bytes: entries.some(([, item]) => item.bytes), + }; + } + case "type": + if (typeof value !== "string" || !TYPES.has(value)) { + throw new Error(`${path}: unsupported type ${JSON.stringify(value)}`); + } + return { schema: value, normal: value, bytes: false }; + case "required": { + if (!Array.isArray(value) || !value.every((name) => typeof name === "string")) { + throw new Error(`${path}: expected an array of strings`); + } + const names = [...value].sort(); + return { schema: names, normal: names, bytes: false }; + } + case "enum": { + const values = data(value, path).normal; + if (!Array.isArray(values)) throw new Error(`${path}: expected an array`); + const sorted = [...values].sort((a, b) => compare(JSON.stringify(a), JSON.stringify(b))); + return { schema: sorted, normal: sorted, bytes: false }; + } + case "data": + case "annotation": + return data(value, path); + } +} + +function walkList(value: unknown, path: string, objects: Objects, ancestors: Set): Copy { + if (!Array.isArray(value)) throw new Error(`${path}: expected an array of schemas`); + const items = Array.from(value, (item, index) => walk(item, `${path}[${index}]`, objects, ancestors)); + return { + schema: items.map((item) => item.schema), + normal: items.map((item) => item.normal), + bytes: items.some((item) => item.bytes), + }; +} + +/** A JSON value as a copy with sorted keys; the same copy serves as snapshot and normal form. */ +function data(value: unknown, path: string): Copy { + if (!isJsonValue(value)) throw new Error(`${path}: expected JSON data (finite numbers, plain objects and arrays)`); + const copy = sortedCopy(value); + return { schema: copy, normal: copy, bytes: false }; +} + +function sortedCopy(value: JsonValue): JsonValue { + if (value === null || typeof value !== "object") return value; + if (Array.isArray(value)) return value.map(sortedCopy); + const keys = Object.keys(value).sort(); + return Object.fromEntries(keys.map((key) => [key, sortedCopy(value[key]!)])); +} + +/** Enumerable data entries sorted by key; symbol keys and accessors are refused. */ +function ownEntries(source: object, path: string): [string, unknown][] { + const entries: [string, unknown][] = []; + for (const key of Reflect.ownKeys(source)) { + if (typeof key === "symbol") throw new Error(`${path}: symbol keys are not supported`); + const descriptor = Object.getOwnPropertyDescriptor(source, key)!; + if (!("value" in descriptor)) throw new Error(`${path}.${key}: accessors are not supported`); + // Non-enumerable keys are TypeBox metadata (`~kind`, `~optional`), never schema data. + if (descriptor.enumerable) entries.push([key, descriptor.value]); + } + return entries.sort(([a], [b]) => compare(a, b)); +} + +function isPlainObject(value: unknown): value is Record { + if (typeof value !== "object" || value === null || Array.isArray(value)) return false; + const prototype = Object.getPrototypeOf(value); + return prototype === Object.prototype || prototype === null; +} + +/** An object schema that admits unknown fields. A record (pattern properties only) is allowed. */ +function isOpenObject(node: Record): boolean { + const isRecord = "patternProperties" in node && !("properties" in node); + const isObject = node.type === "object" || "properties" in node; + return isObject && !isRecord && node.additionalProperties !== false; +} + +function checkBytesBounds(node: Record, path: string): void { + for (const bound of [node.minLength, node.maxLength]) { + if (bound !== undefined && !(Number.isSafeInteger(bound) && (bound as number) >= 0)) { + throw new Error(`${path}: bytes bounds must be non-negative integers`); + } + } +} + +function compare(a: string, b: string): number { + return a < b ? -1 : a > b ? 1 : 0; +} diff --git a/packages/chord/src/next/data/README.md b/packages/chord/src/next/data/README.md new file mode 100644 index 000000000..1b8817496 --- /dev/null +++ b/packages/chord/src/next/data/README.md @@ -0,0 +1,60 @@ +# data + +Per-call data checks, principals and errors. Pure functions and weak caches; no I/O. + +## For apps (`index.ts`) + +```ts +import { broken, isChordError } from "./index.ts"; + +async function open(config: { path?: string }) { + if (config.path === undefined) throw broken(new Error("notes: no path configured")); // retrying will not help + // ... +} + +try { + await notes.add({ text: "hi" }); +} catch (error) { + if (isChordError(error, "invalid")) console.log(error.reason, error.message); // e.g. "args", "args/text: missing" +} +``` + +- `ChordError` with `code`, `reason`, `message`, `details?`; narrowing on `code` narrows `reason`. `isChordError(error, + code?)`, and the `ChordErrorCode`/`ChordErrorReason` types. +- `Principal` (plain JSON with a `kind`) and `PrincipalKeyFunction`. Apps declare principals as type aliases, not + interfaces (an interface does not fit `Principal`'s index signature). A custom principal key function must be the + same on every host; a different one silently splits identities. +- `broken(cause)`: thrown by an open to mark a bad configuration; `FailureCause` is what Chord reports. +- `new PluginError(code, message, details?)`: what a method throws to give its callers a code of its own. On one host + callers get the thrown value itself; over a link only `code`, `message` and JSON `details` cross and the consumer + rethrows a PluginError. Any other thrown value crosses as a PluginError with the code `internal` and its message. + +## For Chord's layers (`internal.ts`) + +- `checkParams(contract, raw, context)`: one compiled closed-object check of a get's params and their key; + `identityKey(contract, key, principalKey)` (no principal key for `local`); `checkAddress(kind, params, raw, + context)`: the same check of an address's params against its host kind's (`invalid/placement`), and its key. +- `bindProvider(contract, accepts)` once per registration, then `checkArgs(provider, method, raw, boundVersion, + context)` per call. It returns `{ kind: "method", args, result? }` (current-shape arguments, and the result mapping + back to the caller's version) or `{ kind: "call", args, call }` (old arguments for the provider's replacement). + `Value` is JSON plus bytes. +- `validatorFor(owner, context)`: the compiled validator of a Chord-owned schema, cached weakly by its owner; + `check(validator, value, reason, context)` runs it (counted) and throws `invalid/` naming the path (a job's + args, `jobs/`). +- `principalKey(principal, context, customKey?)`, `canonicalJson(value, context)`. +- `invalid`, `asChordError`, `chordErrorFromJson`, `PROTOCOL_VERSION`, `CHORD_ERROR_REASONS`; + `openFailure(thrown, label)`. + +Rules: + +- Params and arguments are the caller's own objects: never copied, frozen or defaulted. A provider must not mutate + them. Results and state are never checked. Nothing in L1 is frozen. +- A method without arguments (argument schema `null`) omits the argument in references and implementations; `null` + is its internal and wire form, which `checkArgs` checks. Unknown fields fail. +- A failed check is `invalid/args`, `invalid/params` or `invalid/placement` with `details.path`, a JSON Pointer from TypeBox's first error. + TypeBox does not escape `~` and `/`, so keys containing them make the path ambiguous. +- `checkArgs` refuses an unknown method with `invalid/member` and an unaccepted version or a refused method with + `contract_mismatch`. It never invokes a `call` replacement. +- Error codes, reasons and their pairings are link protocol: changing them bumps `PROTOCOL_VERSION`. +- Every function that works takes a required `context` (after the required arguments, before optional ones). Work is + counted in `chord.data.walks`, `chord.data.validations` and `chord.data.compilations` through its telemetry. diff --git a/packages/chord/src/next/data/args.ts b/packages/chord/src/next/data/args.ts new file mode 100644 index 000000000..88cb6f69f --- /dev/null +++ b/packages/chord/src/next/data/args.ts @@ -0,0 +1,142 @@ +// Arguments: a provider's binding (its dispatch table per accepted version) and the per-call argument check. + +import type { Validator } from "typebox/compile"; +import type { Context } from "../../context/index.ts"; +import { type AcceptedVersion, type AcceptsTable, checkAccepts } from "../contracts/accepts.ts"; +import { + type Contract, + type CurrentVariants, + contractData, + type MethodData, + type VersionData, +} from "../contracts/contract.ts"; +import type { MemberNames, MethodAt } from "../contracts/history.ts"; +import type { Static } from "../contracts/schema.ts"; +import type { Snapshot } from "../contracts/snapshot.ts"; +import { ChordError, invalid } from "./errors.ts"; +import { check, validatorFor } from "./validator.ts"; + +/** JSON plus byte leaves. Same-host values are plain data and immutable once handed over, by convention. */ +export type Value = null | boolean | number | string | Uint8Array | Value[] | { [key: string]: Value }; + +/** A provider's binding of a contract with the older versions it accepts. */ +export type Provider = { readonly contract: C; readonly accepted: readonly AcceptedVersion[] }; + +/** + * Checked arguments: a method's arguments in the current shape with the result mapping back to the caller's version, + * or the old arguments for the provider's own `call` replacement (L4 dispatches it). + */ +export type CheckedArgs = + | { readonly kind: "method"; readonly args: ArgsOf; readonly result?: (current: Value) => Value } + | { readonly kind: "call"; readonly args: Value; readonly call: (args: Value, context: Context) => Promise }; + +type ArgsOf = M extends MemberNames, "methods"> + ? Static, M>["args"]> + : Value; + +/** + * Binds a contract and the app's accepts table (validated once, kept as it is) into a provider. Builds the dispatch + * table: accepted version -> method -> schema and mapping; nothing is compiled yet. + */ +export function bindProvider(contract: C, accepts: AcceptsTable): Provider { + const { versions: accepted } = checkAccepts(contract, accepts); + const { versions } = contractData(contract); + const current = versions.at(-1)!; + // checkAccepts established this shape. + const mappings = accepts as CheckedAccepts; + const table: Dispatch = new Map([[current.version, dispatchEntries(current, undefined)]]); + for (const { version } of accepted) { + const data = versions.find((candidate) => candidate.version === version)!; + table.set(version, dispatchEntries(data, mappings[version]?.methods)); + } + const provider = { contract, accepted }; + dispatchTables.set(provider, table); + return provider; +} + +/** An accepts table after `checkAccepts`, as the binding reads it. */ +type CheckedAccepts = { + readonly [version: string]: { readonly methods?: { readonly [member: string]: CheckedMethodMapping } }; +}; + +type CheckedMethodMapping = + | "refuse" + | { readonly call: (args: Value, context: Context) => Promise } + | { readonly args?: (old: Value) => Value; readonly result?: (current: Value) => Value }; + +/** Accepted version -> method -> entry. */ +type Dispatch = Map>; + +type DispatchEntry = + | { readonly kind: "refuse" } + | { + readonly kind: "call"; + readonly args: Snapshot; + readonly call: (args: Value, context: Context) => Promise; + validator: Validator | undefined; + } + | { + readonly kind: "method"; + readonly args: Snapshot; + readonly mapArgs: ((old: Value) => Value) | undefined; + readonly mapResult: ((current: Value) => Value) | undefined; + validator: Validator | undefined; + }; + +const dispatchTables = new WeakMap, Dispatch>(); + +function dispatchEntries( + version: VersionData, + mappings: { readonly [member: string]: CheckedMethodMapping } | undefined, +): Map { + const entries = new Map(); + // A method has one signature in every variant that has it. + const methods = new Map([...version.variants.values()].flatMap(({ methods }) => [...methods])); + for (const [member, { args }] of methods) { + const mapping = mappings !== undefined && Object.hasOwn(mappings, member) ? mappings[member] : undefined; + entries.set(member, dispatchEntry(args, mapping)); + } + return entries; +} + +function dispatchEntry(args: Snapshot, mapping: CheckedMethodMapping | undefined): DispatchEntry { + if (mapping === "refuse") return { kind: "refuse" }; + if (mapping !== undefined && "call" in mapping) { + return { kind: "call", args, call: mapping.call, validator: undefined }; + } + return { kind: "method", args, mapArgs: mapping?.args, mapResult: mapping?.result, validator: undefined }; +} + +/** + * Checks a call's arguments against the provider's schema for the caller's bound version: three map lookups, one + * compiled check, at most one direct mapping old -> current (never for current-version calls). A method without + * arguments gets `null` (its internal and wire form). Unknown method: `invalid/member`; version not accepted or refused: `contract_mismatch`. + */ +export function checkArgs( + provider: Provider, + method: M, + raw: unknown, + boundVersion: number, + context: Context, +): CheckedArgs { + const name = provider.contract.name; + const methods = dispatchTables.get(provider)!.get(boundVersion); + if (methods === undefined) { + throw new ChordError({ code: "contract_mismatch", message: `${name} v${boundVersion} is not accepted` }); + } + const entry = methods.get(method); + if (entry === undefined) throw invalid("member", `${name} v${boundVersion} has no method "${method}"`); + if (entry.kind === "refuse") { + throw new ChordError({ code: "contract_mismatch", message: `${name} v${boundVersion} refuses "${method}"` }); + } + entry.validator ??= validatorFor(entry.args, context); + check(entry.validator, raw, "args", context); + // The check above establishes the argument type; mapping output is trusted. + const args = raw as Value; + if (entry.kind === "call") return { kind: "call", args, call: entry.call }; + return { + kind: "method", + args: (entry.mapArgs === undefined ? args : entry.mapArgs(args)) as ArgsOf, + result: entry.mapResult, + }; +} diff --git a/packages/chord/src/next/data/errors.ts b/packages/chord/src/next/data/errors.ts new file mode 100644 index 000000000..697f3b578 --- /dev/null +++ b/packages/chord/src/next/data/errors.ts @@ -0,0 +1,142 @@ +// ChordError: Chord's one error type, its codes and reasons (link protocol) and its wire form; and PluginError, what a +// plugin throws to give its callers a code of its own. + +import type { JsonValue } from "../../json.ts"; + +/** Version of the error table below on links; adding or moving a code or reason bumps it. */ +export const PROTOCOL_VERSION = 1; + +/** Every code with the reasons it may carry. `invalid` requires a reason; every other code may omit it. */ +export const CHORD_ERROR_REASONS = { + invalid: ["args", "params", "member", "placement", "reentrant", "read_after_bulk", "cycle"], + contract_mismatch: ["updating"], + forbidden: ["revoked"], + gone: [], + not_ready: [], + // No reason: released by its holder. `ended`: its object ended; `closed`: its host, store or native closed; + // `settled`: used after the call or run it belonged to settled; `version`: an old handle after a migration; + // `discarded`: a transaction cut at the seal. + released: ["ended", "closed", "settled", "version", "discarded"], + too_large: ["part"], + unavailable: ["locked", "park_timeout", "busy", "storage_failed"], + aborted: [], +} as const; + +export type ChordErrorCode = keyof typeof CHORD_ERROR_REASONS; + +export type ChordErrorReason = (typeof CHORD_ERROR_REASONS)[C][number]; + +/** The wire form of a ChordError and its constructor argument: one member per code, so wrong pairings fail to type. */ +type ChordErrorJson = { [C in ChordErrorCode]: ErrorFields }[ChordErrorCode]; + +type ErrorFields = { + readonly code: C; + readonly message: string; + readonly details?: JsonValue; +} & (C extends "invalid" + ? { readonly reason: ChordErrorReason } + : [ChordErrorReason] extends [never] + ? { readonly reason?: undefined } + : { readonly reason?: ChordErrorReason }); + +class ChordErrorClass extends Error { + readonly code: ChordErrorCode; + readonly reason: ChordErrorReason | undefined; + readonly details: JsonValue | undefined; + + constructor(init: ChordErrorJson, options?: ErrorOptions) { + super(init.message, options); + this.name = "ChordError"; + this.code = init.code; + this.reason = init.reason; + this.details = init.details; + } + + /** The wire form; stack and cause never cross a link. */ + toJSON(): ChordErrorJson { + const { code, reason, message, details } = this; + // The fields were copied from a ChordErrorJson, so the pairing holds. + return { + code, + ...(reason === undefined ? {} : { reason }), + message, + ...(details === undefined ? {} : { details }), + } as ChordErrorJson; + } +} + +/** A Chord failure. Narrowing on `code` narrows `reason`. */ +export type ChordError = ChordErrorClass & ChordErrorJson; + +/** + * Creates a ChordError: `new ChordError({ code: "invalid", reason: "args", message })`. A class cannot declare its + * instances as a tagged union; this constructor signature does. + */ +export const ChordError = ChordErrorClass as new (init: ChordErrorJson, options?: ErrorOptions) => ChordError; + +/** Whether `error` is a ChordError, optionally with the given code. */ +export function isChordError( + error: unknown, + code?: C, +): error is Extract { + return error instanceof ChordError && (code === undefined || error.code === code); +} + +/** + * An error a plugin throws for its callers: its own `code`, a message and JSON `details`. Only these cross a link + * (never a stack or a cause); a consumer rethrows it as a PluginError. + */ +export class PluginError extends Error { + readonly code: string; + readonly details: JsonValue | undefined; + + constructor(code: string, message: string, details?: JsonValue) { + super(message); + this.name = "PluginError"; + this.code = code; + this.details = details; + } +} + +/** An `invalid` error with the given reason. */ +export function invalid( + reason: ChordErrorReason<"invalid">, + message: string, + details?: JsonValue, +): Extract { + // The constructor's return type cannot narrow on its argument. + return new ChordError({ code: "invalid", reason, message, details }) as Extract; +} + +/** `error` itself when it is a ChordError, otherwise an `unavailable` error naming `label`, with `error` as cause. */ +export function asChordError(error: unknown, label: string): ChordError { + if (isChordError(error)) return error; + const message = error instanceof Error ? error.message : String(error); + return new ChordError({ code: "unavailable", message: `${label}: ${message}` }, { cause: error }); +} + +/** + * Rebuilds a ChordError from a link. Checks the envelope only: a known code, a reason allowed for it (required for + * `invalid`), a string message, no other keys. `details` is kept by reference. + */ +export function chordErrorFromJson(raw: unknown): ChordError { + // Trust boundary: `raw` is fresh JSON from the link decoder. + if (typeof raw !== "object" || raw === null || Array.isArray(raw)) throw envelopeError("not an object"); + const json = raw as Record; + for (const key of Object.keys(json)) if (!ENVELOPE_KEYS.has(key)) throw envelopeError(`unknown key "${key}"`); + const { code, reason, message } = json; + if (typeof code !== "string" || !Object.hasOwn(CHORD_ERROR_REASONS, code)) throw envelopeError("unknown code"); + const reasons: readonly string[] = CHORD_ERROR_REASONS[code as ChordErrorCode]; + if (reason === undefined ? code === "invalid" : !reasons.includes(reason as string)) { + throw envelopeError(`reason not allowed for ${code}`); + } + if (typeof message !== "string") throw envelopeError("message is not a string"); + // The checks above establish the pairing. + return new ChordError(json as ChordErrorJson); +} + +const ENVELOPE_KEYS = new Set(["code", "reason", "message", "details"]); + +function envelopeError(problem: string): Error { + return new Error(`Invalid ChordError JSON: ${problem}`); +} diff --git a/packages/chord/src/next/data/failure.ts b/packages/chord/src/next/data/failure.ts new file mode 100644 index 000000000..a76190118 --- /dev/null +++ b/packages/chord/src/next/data/failure.ts @@ -0,0 +1,26 @@ +// Failure causes of an open: transient (retry later) or broken (bad configuration). The failing stage decides. + +import { asChordError, type ChordError } from "./errors.ts"; + +/** Why an open failed; the kind comes from the failing stage, never from the error code. */ +export type FailureCause = + | { readonly kind: "transient"; readonly error: ChordError } + | { readonly kind: "broken"; readonly error: ChordError }; + +/** Marks an open's failure as broken (bad configuration): `throw broken(error)`. */ +export function broken(cause: unknown): Error { + return new BrokenOpen(cause); +} + +/** The cause of a value thrown by an open: broken when it came from `broken()`, transient otherwise. */ +export function openFailure(thrown: unknown, label: string): FailureCause { + if (thrown instanceof BrokenOpen) return { kind: "broken", error: asChordError(thrown.cause, label) }; + return { kind: "transient", error: asChordError(thrown, label) }; +} + +class BrokenOpen extends Error { + constructor(cause: unknown) { + super(cause instanceof Error ? cause.message : String(cause), { cause }); + this.name = "BrokenOpen"; + } +} diff --git a/packages/chord/src/next/data/index.ts b/packages/chord/src/next/data/index.ts new file mode 100644 index 000000000..3eb2015e8 --- /dev/null +++ b/packages/chord/src/next/data/index.ts @@ -0,0 +1,11 @@ +// Data for apps: Chord's errors and plugin errors, principals, and how a plugin's open reports a broken configuration. + +export { + ChordError, + type ChordErrorCode, + type ChordErrorReason, + isChordError, + PluginError, +} from "./errors.ts"; +export { broken, type FailureCause } from "./failure.ts"; +export type { Principal, PrincipalKeyFunction } from "./principal.ts"; diff --git a/packages/chord/src/next/data/internal.ts b/packages/chord/src/next/data/internal.ts new file mode 100644 index 000000000..d59881ebe --- /dev/null +++ b/packages/chord/src/next/data/internal.ts @@ -0,0 +1,14 @@ +// Data for Chord's layers: per-call checks of params and arguments, principal keys, and errors on the wire. + +export { bindProvider, type CheckedArgs, checkArgs, type Provider, type Value } from "./args.ts"; +export { + asChordError, + CHORD_ERROR_REASONS, + chordErrorFromJson, + invalid, + PROTOCOL_VERSION, +} from "./errors.ts"; +export { openFailure } from "./failure.ts"; +export { checkAddress, checkParams, identityKey } from "./params.ts"; +export { canonicalJson, principalKey } from "./principal.ts"; +export { check, validatorFor } from "./validator.ts"; diff --git a/packages/chord/src/next/data/params.ts b/packages/chord/src/next/data/params.ts new file mode 100644 index 000000000..17ce090e1 --- /dev/null +++ b/packages/chord/src/next/data/params.ts @@ -0,0 +1,43 @@ +// Params and identity: the checked params of a get, their key, and the in-process identity of an object. + +import type { Context } from "../../context/index.ts"; +import { type Contract, contractData, type ParamsOf, type ShapeOf } from "../contracts/contract.ts"; +import { type Address, addressKey } from "../contracts/host-kind.ts"; +import { joinKey, type ParamsData } from "../contracts/params.ts"; +import { check, validatorFor } from "./validator.ts"; + +/** + * Checks `raw` against the contract's params with one compiled closed-object check (`invalid/params` on failure) and + * returns the caller's own object with its key: the field values in declared order, each escaped, joined with `|`. + */ +export function checkParams( + contract: C, + raw: unknown, + context: Context, +): { readonly params: ParamsOf; readonly key: string } { + const { params } = contractData(contract); + check(validatorFor(params, context), raw, "params", context); + // The check above establishes the params type. + const checked = raw as ParamsOf & { readonly [field: string]: string | number | boolean }; + return { params: checked, key: joinKey(params.fields.map((field) => String(checked[field]))) }; +} + +/** The in-process identity of an object: `name|key|principalKey`, each part escaped; `local` has no principal. */ +export function identityKey( + contract: C, + key: string, + ...principalKey: ShapeOf["kind"] extends "local" ? [] : [principalKey: string] +): string { + return joinKey([contract.name, key, ...principalKey]); +} + +/** + * An address from a kind's name and params as a hold names it or a placement returned it: the params checked against + * the kind's with one compiled closed-object check (`invalid/placement` on failure), the key computed. + */ +export function checkAddress(kind: string, params: ParamsData, raw: unknown, context: Context): Address { + check(validatorFor(params, context), raw, "placement", context); + // The check above establishes a flat record of the kind's scalar fields. + const values = raw as { readonly [field: string]: string | number | boolean }; + return { kind, params: values, key: addressKey(kind, params, values) }; +} diff --git a/packages/chord/src/next/data/principal.ts b/packages/chord/src/next/data/principal.ts new file mode 100644 index 000000000..b9de9e5dc --- /dev/null +++ b/packages/chord/src/next/data/principal.ts @@ -0,0 +1,57 @@ +// Principals: the app's caller identities, and the keys Chord derives from them. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; + +/** An app's principal: plain JSON with a `kind`. Chord never persists it. */ +export type Principal = { readonly kind: string; readonly [field: string]: JsonValue }; + +/** An app's own principal key; it must be the same function on every host. */ +export type PrincipalKeyFunction

= (principal: P) => string; + +/** + * The key of a principal: its canonical JSON, or `customKey(principal)`. Cached per principal object; a custom key's + * cache is keyed by the function too, so a live principal never retains a dropped function. + */ +export function principalKey

( + principal: P, + context: Context, + customKey?: PrincipalKeyFunction

, +): string { + const cache = customKey === undefined ? defaultKeys : customKeyCache(customKey); + let key = cache.get(principal); + if (key === undefined) { + key = customKey === undefined ? canonicalJson(principal, context) : customKey(principal); + cache.set(principal, key); + } + return key; +} + +const defaultKeys = new WeakMap(); + +const customKeys = new WeakMap, WeakMap>(); + +function customKeyCache(customKey: PrincipalKeyFunction): WeakMap { + let cache = customKeys.get(customKey); + if (cache === undefined) { + cache = new WeakMap(); + customKeys.set(customKey, cache); + } + return cache; +} + +/** Canonical JSON: sorted object keys, `-0` written as `0`. Counts one walk. */ +export function canonicalJson(value: JsonValue, context: Context): string { + context.telemetry.metrics?.counter("chord.data.walks").add(1); + return canonical(value); +} + +function canonical(value: JsonValue): string { + // JSON.stringify writes -0 as 0. + if (value === null || typeof value !== "object") return JSON.stringify(value); + if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`; + const fields = Object.keys(value) + .sort() + .map((key) => `${JSON.stringify(key)}:${canonical(value[key]!)}`); + return `{${fields.join(",")}}`; +} diff --git a/packages/chord/src/next/data/validator.ts b/packages/chord/src/next/data/validator.ts new file mode 100644 index 000000000..6c3369a6b --- /dev/null +++ b/packages/chord/src/next/data/validator.ts @@ -0,0 +1,72 @@ +// Compiled checks: TypeBox validators of Chord-owned schemas, and the `invalid` error of a failed check. + +import { Compile, type Validator } from "typebox/compile"; +import type { TLocalizedValidationError } from "typebox/error"; +import { ResetExternal, ResetFunctions } from "typebox/schema"; +import { Errors } from "typebox/value"; +import type { Context } from "../../context/index.ts"; +import type { TSchema } from "../contracts/schema.ts"; +import { type ChordError, invalid } from "./errors.ts"; + +/** A Chord-owned record holding a schema: a snapshot, or a contract's params. */ +type SchemaOwner = { readonly schema: TSchema }; + +/** The validator of an owner's schema, compiled on first use (counted) and cached weakly by the owner. */ +export function validatorFor(owner: SchemaOwner, context: Context): Validator { + let validator = validators.get(owner); + if (validator === undefined) { + context.telemetry.metrics?.counter("chord.data.compilations").add(1); + try { + validator = Compile(owner.schema); + } finally { + // TypeBox keeps its last build in module-level accumulators; reset them so no build state is retained. + ResetFunctions(); + ResetExternal(); + } + validators.set(owner, validator); + } + return validator; +} + +const validators = new WeakMap(); + +/** Checks `value` (counted); a failure throws `invalid/` with the first error's JSON Pointer path. */ +export function check( + validator: Validator, + value: unknown, + reason: "args" | "params" | "placement", + context: Context, +): void { + context.telemetry.metrics?.counter("chord.data.validations").add(1); + if (!validator.Check(value)) throw failedCheck(validator.Type(), value, reason, context); +} + +/** + * The error of a failed check: TypeBox's first error from its diagnostic walk (counted; it builds the whole list). + * TypeBox does not escape `~` and `/` in its instance paths, so keys containing them make the path ambiguous. + */ +function failedCheck( + schema: TSchema, + value: unknown, + reason: "args" | "params" | "placement", + context: Context, +): ChordError { + context.telemetry.metrics?.counter("chord.data.walks").add(1); + const [error] = Errors(schema, value); + const { path, problem } = describe(error); + return invalid(reason, `${reason}${path === "/" ? "" : path}: ${problem}`, { path }); +} + +/** + * The path and problem of a TypeBox error: TypeBox reports a missing field at its parent, and an unknown one as a + * false schema (`additionalProperties: false`) at its own path. Anything else keeps TypeBox's message. + */ +function describe(error: TLocalizedValidationError): { path: string; problem: string } { + if (error.keyword === "required") { + return { path: `${error.instancePath}/${error.params.requiredProperties[0]}`, problem: "missing" }; + } + if (error.keyword === "boolean" && error.schemaPath.endsWith("/additionalProperties")) { + return { path: error.instancePath, problem: "unknown field" }; + } + return { path: error.instancePath || "/", problem: error.message }; +} diff --git a/packages/chord/src/next/documents/README.md b/packages/chord/src/next/documents/README.md new file mode 100644 index 000000000..0c2748771 --- /dev/null +++ b/packages/chord/src/next/documents/README.md @@ -0,0 +1,91 @@ +# documents + +Documents: definitions, the handles an object's attempts get, the object's documents (in memory, or persisted in the +host's store), keyed watches and `at(key)`, retirement, and transactions over documents, tables and job runs. + +## For apps (`index.ts`) + +```ts +import { defineState } from "./index.ts"; + +const Notes = defineState("notes", { schema: obj({ list: arr(Note) }), version: 1, initial: () => ({ list: [] }) }); +const Drafts = defineState("drafts", { schema: obj({ body: str() }), version: 1, initial: () => ({ body: "" }), keyed: true }); +const Prefs = defineState("prefs", { + schema: obj({ theme: str() }), version: 2, initial: () => ({ theme: "light" }), + migrate: (old, from) => ({ theme: "light" }), // a stored or in-memory value of an older version + persist: true, // in the host's store (a local type with empty params) + checkpointWhen: (value, ops, { deltasSinceBase }) => deltasSinceBase >= 50, // store a new base, not a delta +}); + +const notes = self.state(Notes); // a singleton: a Watchable, `initial()` until written +const drafts = self.state(Drafts); // keyed: get(key, ctx), keys(ctx), watch(key, ctx), at(key) in the factory +const pinned = drafts.at("c1"); // a Watchable of one key (a state member, a derive or `on` source) +await self.tx(async (tx) => { + (await tx.draft(notes)).list.push({ text }); + (await tx.draft(drafts, "c1")).body = text; + tx.put(edits, { id, path }); // tables (`store/`) in the same commit +}, ctx); +await drafts.change("c1", (draft) => { draft.body = ""; }, ctx); // one-document shorthand +await notes.retire(ctx); // back to initial(), watches get `retired: true` +``` + +- `defineState(name, { schema, version, initial, keyed?, migrate?, persist?, checkpointWhen? })`: the root is a JSON + object. `initial()` is checked against the schema once, when the declaring definition is first used (retaken only + when the declaring version or fingerprint changes). `persist: true` requires `checkpointWhen` (types and + declaration) and a `local` type with empty params with a store on its host (P3). +- Singleton handle: `value`, `watch(ctx)`, `change(fn, ctx)`, `retire(ctx)`. A retirement reaches watches as + `initial()` with ops `[["r", initial]]` and `retired: true`; a retirement and a recreation in one transaction are one + root replacement, flagged. +- Keyed handle: `get(key, ctx)` (`undefined` when absent), `keys(ctx)` (code point order), `watch(key, ctx)` + (`undefined` when absent; it follows the address through retirements and recreations), `at(key)` (in the factory + only: `initial()` while absent, loaded before the object goes live, pinned until the next attempt's factory + returned), `change(key, fn, ctx)`, `retire(key, ctx)`. A persisted key is loaded on its first read on the object's + line (one load per key, shared by its readers), cached while read, pinned or watched, and evicted after ten + minutes idle; an absent key is not cached. +- `Tx`: `await tx.draft(handle, key?)` (the same draft for the same document within one transaction; a cold persisted + key loads first), `tx.retire(handle, key?)` (a no-op when absent), `tx.start(job, args, { key? })` (`jobs/`), and + the table writes and reads of `store/` (`put`, `update`, `delete`, `updateWhere`, `deleteWhere`, `get`, `page`, + `rows`). Every `tx.*` that returns a promise must be awaited before the callback returns. + +Rules: + +- Documents belong to their object, not to an attempt: a retry or a native restart keeps them, a new attempt's + `self.state` gets handles to the same documents (an old handle below the slot's version is refused + `released/version`). Persisted documents load once per object. An ending attempt's handles are sealed: they read + synchronously, anything taking the line is refused `released` (a cold read queued before the seal too); its open + transaction is discarded (drafts aborted, its callback's result rejected `released/discarded`). +- An object's transactions run one at a time on its line. A transaction covers only its own object. An in-memory-only + transaction commits synchronously in Chord's batch; one with a persisted document or a table write enters the + store's commit step, and once admitted it is adopted whatever happens to its attempt. +- When the callback resolves, an aborted `ctx` or an ending attempt discards the transaction; a throw discards it too. + Unchanged drafts commit nothing. Drafts are revoked once the transaction settles: return plain values, read data + from the handle after the commit. +- Inside the callback: only `await tx.*`; no second `self.tx`, no calls to other objects (not detected: it hangs). +- The shorthands `handle.change` and `handle.retire` run on the object's line from anyone holding the handle. That + only the object writes them is a convention, like the rule for `tx.draft`: keep handles inside the owner. +- A stored document of a newer version, a `migrate` that throws or returns a value outside the schema, or the same + version with another fingerprint fails the object `broken` (at activation, or while live for a keyed load); the data + stays. + +## For Chord's layers (`internal.ts`) + +- `Documents`: one object's documents and line: slots by name, `hydrate(declared, pins, ctx)` after each factory + (drops undeclared in-memory slots, takes the pins), `load(ctx)` (the persisted singletons and pinned keys, a line + job; `undefined` when nothing is persisted), `tx(fn, token, ctx)`, `fence(token)` (cuts the attempt's transaction at + its seal), `seal()` and `end()` at the object's end. +- `SealToken`: one per attempt; sealed when the attempt ends. Transactions and handles check it. +- `Singleton`, `Keyed`: the handles (`slot`, `token`, `version`); `KeyDeclarer` is what `at(key)` pins through. +- `KeyedSlot`, `KeySource`: a keyed slot's residents, per-key sources (the targets of `at(key)` and keyed members, + with their pins) and shared cold loads. +- `DocumentSlot`, `AnyDefinition`: a slot as transactions draft and adopt it. +- `Transaction`, `Staged`: Chord's own changes staged in a transaction (a job's run records), adopted with the + documents in the same commit, then told how the transaction ended once the line is free. + +Inside the directory: `definition.ts` (`defineState`, versions, fingerprints, `migrate`), `slot.ts` (singleton slots +and the slot contract), `keyed-slot.ts`, `address-source.ts` (the frames of one document address: changes, root +replacements, retirements), `persistence.ts` (a slot's stored side: load, find, keys, the writes of a commit), +`residency.ts` (keyed residents and their eviction: Durable's idle structure, one timer per object), `handles.ts`, +`fence.ts`, `transaction.ts`, `line.ts` (one holder at a time, in order) and `documents.ts`. + +Tests: `test/next/documents/` (one file per source). Examples: `02-documents.ts`, `21-persisted-documents.ts`, +`23-keyed-watches.ts`. diff --git a/packages/chord/src/next/documents/address-source.ts b/packages/chord/src/next/documents/address-source.ts new file mode 100644 index 000000000..dc0a2d416 --- /dev/null +++ b/packages/chord/src/next/documents/address-source.ts @@ -0,0 +1,56 @@ +// Watchable document addresses (slice 6a §11; decision "One retirement mechanism"): the frames of a source that +// follows one address, a singleton's slot or one key of a keyed slot. A change arrives with its ops, a write over +// another base as a root replacement, a retirement as `[["r", initial()]]` with the `retired` flag, and a document +// created after a retirement in the same batch as one root replacement with the flag. + +import type { Context } from "../../context/index.ts"; +import type { Op, Prepared } from "../../delta/index.ts"; +import { Source } from "../state/internal.ts"; +import type { JsonObject } from "./definition.ts"; +import type { Documents } from "./documents.ts"; + +const NO_OPS: readonly Op[] = []; + +/** A source following one document address; `value` is the document's, or `initial()` while it is absent. */ +export abstract class AddressSource extends Source { + /** The documents of the object whose document this is (a member source asks whose source it follows). */ + abstract readonly owner: Documents; + /** The ops of the current batch's frame, and whether the document was retired in it. */ + #ops = NO_OPS; + #retired = false; + + abstract override get value(): JsonObject; + + deliver(context: Context): void { + this.publish(this.value, this.#ops, this.#retired, context); + this.#ops = NO_OPS; + this.#retired = false; + } + + /** + * A commit wrote the document, which read `before`: its ops, or a root replacement when the draft started from + * another base or the document was retired earlier in the batch. + */ + written(before: JsonObject, prepared: Prepared): void { + // The ops are kept for a frame, which only a watched source gets: an unwatched one keeps none. + if (this.watched) this.#ops = prepared.base === before && !this.#retired ? prepared.ops : [["r", prepared.value]]; + this.changed(); + } + + /** The document was retired: it reads `initial` again, and the frame says so. */ + retired(initial: JsonObject): void { + // Reset by the frame, which only a watched source gets. + if (this.watched) { + this.#ops = [["r", initial]]; + this.#retired = true; + } + this.changed(); + this.retiredInBatch(); + } + + /** The value changed outside a commit (a migration, a load): a root replacement. */ + replaced(): void { + if (this.watched) this.#ops = [["r", this.value]]; + this.changed(); + } +} diff --git a/packages/chord/src/next/documents/definition.ts b/packages/chord/src/next/documents/definition.ts new file mode 100644 index 000000000..77906479b --- /dev/null +++ b/packages/chord/src/next/documents/definition.ts @@ -0,0 +1,157 @@ +// Document definitions: `defineState(name, ...)`, its fingerprint, the one check of a definition's `initial()` (R8), +// the checked `migrate` of an older value and, for a persisted document, its checkpoint rule (slice 6a §10). + +import type { Context } from "../../context/index.ts"; +import type { Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Static, TSchema } from "../contracts/index.ts"; +import { checkName, fingerprintOf } from "../contracts/internal.ts"; +import { snapshotSchema } from "../contracts/snapshot.ts"; +import { validatorFor } from "../data/internal.ts"; + +/** A document root: a JSON object. */ +export type JsonObject = { [key: string]: JsonValue }; + +/** What a persisted document's `checkpointWhen` learns of its stored revisions. */ +export type CheckpointInfo = { + /** The deltas stored after its newest base, without the change being decided. */ + readonly deltasSinceBase: number; +}; + +/** Whether a commit's change of a persisted document is stored as a whole new base (`true`) or as a delta. */ +export type CheckpointWhen = (value: T, ops: readonly Op[], info: CheckpointInfo) => boolean; + +/** What `defineState` takes for any document. */ +type StateOptions = { + readonly schema: S; + readonly version: number; + readonly initial: () => DocumentOf; + /** Maps a value of an older version `from` to this version's shape; pure. Required to read older values. */ + readonly migrate?: (old: JsonObject, from: number) => DocumentOf; + readonly keyed?: K; +}; + +/** What a persisted document adds (slice 6a §10): `checkpointWhen` is required. */ +type PersistOptions = { + /** Stored in the host's store; only in `local` types with empty params (P3). */ + readonly persist: true; + /** Called once per commit per changed document; creations and version changes always store a base. */ + readonly checkpointWhen: CheckpointWhen>; +}; + +/** + * Declares a document: a singleton per object, or with `keyed: true` many per object, each by a string key; with + * `persist: true` stored in the host's store. Its name is a restricted name (lowercase `[a-z0-9@/._~-]`). + */ +export function defineState( + name: string, + options: StateOptions & { readonly persist?: false; readonly checkpointWhen?: never }, +): StateDefinition, K, false>; +export function defineState( + name: string, + options: StateOptions & PersistOptions, +): StateDefinition, K, true>; +export function defineState( + name: string, + options: StateOptions & { + readonly persist?: boolean; + readonly checkpointWhen?: CheckpointWhen>; + }, +): StateDefinition, K, boolean> { + checkName(name, "A state name"); + const { schema, version, initial, migrate, keyed, persist, checkpointWhen } = options; + return new StateDefinition( + name, + schema, + version, + initial, + migrate, + keyed ?? false, + persist ?? false, + checkpointWhen, + ); +} + +/** The value of a document whose schema is `S`: its static type, which must be a JSON object. */ +type DocumentOf = Static extends JsonObject ? Static : never; + +const NEVER: CheckpointWhen = () => false; + +/** A declared document; `T` is its value, `K` whether it is keyed, `P` whether it is persisted. */ +export class StateDefinition { + readonly name: string; + readonly schema: TSchema; + readonly version: number; + readonly keyed: boolean; + readonly persist: boolean; + /** + * `hash64` of the schema's normal form (descriptions and titles ignored), computed once: the same version with + * another fingerprint is a schema changed without a version bump. + */ + readonly fingerprint: string; + /** Type only: whether the definition is keyed. */ + declare readonly "~keyed"?: K; + /** Type only: whether the definition is persisted. */ + declare readonly "~persist"?: P; + readonly #initial: () => T; + readonly #migrate: ((old: JsonObject, from: number) => T) | undefined; + /** Method-typed so that a definition of a narrower `T` still is a `StateDefinition`. */ + readonly #checkpointWhen: { rule(value: T, ops: readonly Op[], info: CheckpointInfo): boolean }["rule"]; + #checked = false; + + constructor( + name: string, + schema: TSchema, + version: number, + initial: () => T, + migrate: ((old: JsonObject, from: number) => T) | undefined, + keyed: boolean, + persist: boolean, + checkpointWhen: CheckpointWhen | undefined, + ) { + this.name = name; + this.schema = schema; + this.version = version; + this.#initial = initial; + this.#migrate = migrate; + this.keyed = keyed; + this.persist = persist; + this.#checkpointWhen = checkpointWhen ?? NEVER; + this.fingerprint = fingerprintOf(snapshotSchema(schema, name).normal); + } + + /** Whether a persisted document's change is stored as a new base (its `checkpointWhen`); never asked in memory. */ + checkpointWhen(value: T, ops: readonly Op[], info: CheckpointInfo): boolean { + return this.#checkpointWhen(value, ops, info); + } + + /** A fresh `initial()` value; the first is checked against the schema (counted), later ones are not (R8). */ + initial(context: Context): T { + const value = this.#initial(); + if (this.#checked) return value; + this.#check(value, "initial()", context); + this.#checked = true; + return value; + } + + /** + * `old`, a value of version `from`, migrated to this version and checked against the schema once, as it is + * produced (R8, counted). Throws without a `migrate` (the access is refused, the data kept) and when the result does + * not match. + */ + migrate(old: JsonObject, from: number, context: Context): T { + if (this.#migrate === undefined) { + throw new Error(`${this.name}: no migrate from version ${from} to ${this.version}`); + } + const value = this.#migrate(old, from); + this.#check(value, `migrate() from version ${from}`, context); + return value; + } + + #check(value: T, what: string, context: Context): void { + context.telemetry.metrics?.counter("chord.data.validations").add(1); + if (!validatorFor(this, context).Check(value)) { + throw new Error(`${this.name}: ${what} does not match the schema`); + } + } +} diff --git a/packages/chord/src/next/documents/documents.ts b/packages/chord/src/next/documents/documents.ts new file mode 100644 index 000000000..6ca0f5e82 --- /dev/null +++ b/packages/chord/src/next/documents/documents.ts @@ -0,0 +1,237 @@ +// An object's documents (§9, P2, slice 6a §1.2, §10): its slots across attempts, its line (its transactions and line +// jobs run one at a time), the seal fence over the transaction holding the line, the factory's pins between attempts, +// the activation's loads and the residency of keyed documents. + +import type { Context } from "../../context/index.ts"; +import { ChordError } from "../data/index.ts"; +import type { Batch } from "../state/internal.ts"; +import type { ChordStore } from "../store/store.ts"; +import type { JsonObject, StateDefinition } from "./definition.ts"; +import { SealToken } from "./fence.ts"; +import { type HandleOf, type KeyDeclarer, Keyed, Singleton } from "./handles.ts"; +import { KeyedSlot } from "./keyed-slot.ts"; +import { Line } from "./line.ts"; +import { Persistence } from "./persistence.ts"; +import { Residency } from "./residency.ts"; +import { type AnyDefinition, SingletonSlot } from "./slot.ts"; +import { Transaction } from "./transaction.ts"; + +/** Who owns an object's documents: the object's name, batch and context, its store address, its live failure. */ +export type DocumentsOwner = { + /** The owner's name in errors. */ + readonly label: string; + readonly batch: Batch; + /** The object's context: shared cold loads run under it (rule 11). */ + readonly context: Context; + /** The host's store and the object's owner string in it (`:`, W7), when it has one. */ + readonly store: { readonly store: ChordStore; readonly owner: string } | undefined; + /** A document the code cannot read fails the live object `broken` (slice 6a §1.4); ignored when it is not live. */ + fail(error: ChordError): void; +}; + +/** + * The documents of one object, from its first activation to its end: one slot per declared name, kept across + * attempts, and the line. Each attempt declares its handles under its own seal token; the object's own transactions + * (job outcomes) run under `own`, sealed only by the object's end. + */ +export class Documents { + /** The owner's name in errors. */ + readonly label: string; + readonly batch: Batch; + /** The object's context. */ + readonly context: Context; + readonly line = new Line(); + /** + * The idle persisted keyed documents and their eviction. A sweep's only rejection is `onLine`'s `released` once + * the object ended (a sweep itself drops trackers in memory and does not throw): nothing to report. + */ + readonly residency = new Residency((sweep) => void this.onLine(async () => sweep()).catch(() => {})); + readonly #owner: DocumentsOwner; + /** The token of the object's own transactions: sealed with the object (`seal`). */ + readonly own = new SealToken(); + readonly #slots = new Map(); + /** The transaction holding the line, until it settles or its token is sealed. */ + #holder: Transaction | undefined; + /** `sealed`: the object is ending; its documents stay readable, its transactions throw `released`. */ + #state: "open" | "sealed" | "ended" = "open"; + /** The pins the last factory that returned declared (`at(key)`): held until the next one returns (rule 6). */ + #pins: readonly (() => void)[] = []; + + constructor(owner: DocumentsOwner) { + this.label = owner.label; + this.batch = owner.batch; + this.context = owner.context; + this.#owner = owner; + } + + /** + * A handle to `definition`'s slot for an attempt sealed by `token`, whose `at(key)` pins `declarer` takes: the slot + * made by the first declaration of its name, checked against the values for later ones (`SlotDefinition.check`). + * "Declared twice" is the attempt's check. + */ + declare>( + definition: D, + token: SealToken, + declarer: KeyDeclarer, + ): HandleOf { + const slot = this.#slots.get(definition.name); + if (slot !== undefined) slot.definition.check(definition); + const persistence = slot === undefined && definition.persist ? this.#persistence(definition.name) : undefined; + // The definition's `keyed` decides the slot's class and the handle's; its documents are the definition's `T`. + if (definition.keyed) { + const keyed = slot ?? this.#add(new KeyedSlot(this, definition, persistence)); + return new Keyed(keyed as KeyedSlot, token, definition.version, declarer) as unknown as HandleOf; + } + const singleton = slot ?? this.#add(new SingletonSlot(this, definition, persistence)); + return new Singleton(singleton as SingletonSlot, token, definition.version) as unknown as HandleOf; + } + + /** + * Takes an attempt's declarations once its factory returned: its `at(key)` pins replace the previous factory's, the + * slots it no longer declares are dropped (their watches end; persisted ones load again when declared again), every + * declared one hydrated (`initial()`, migrations; a throw fails the activation). + */ + hydrate(declared: ReadonlyMap, pins: readonly (() => void)[], context: Context): void { + const previous = this.#pins; + this.#pins = pins; + for (const release of previous) release(); + // A dropped persisted slot goes whole, its loaded documents too: observably the same as keeping them, since + // every committed write is in the store and a later declaration reloads from there. + for (const [name, slot] of this.#slots) { + if (declared.has(name)) continue; + this.#slots.delete(name); + slot.drop(); + } + for (const [name, definition] of declared) this.#slots.get(name)!.hydrate(definition, context); + } + + /** + * Loads, in one line job, every persisted singleton not loaded yet and every pinned key not loaded since it was + * pinned (an attempt's activation, before it goes live, rule 6); `undefined` when nothing is persisted. A failure + * fails the activation. + */ + load(context: Context): Promise | undefined { + const persisted = [...this.#slots.values()].filter((slot) => slot.persisted !== undefined); + // In memory only: nothing to wait for (an activation without persisted documents takes no extra turn). + if (persisted.length === 0) return undefined; + return this.onLine(async () => { + for (const slot of persisted) await slot.loadDeclared(context); + }); + } + + /** + * Runs `job` on the line (a cold load, a page of keys, an eviction sweep); refused `released` once the object is + * ending or `token` (the handle's that asks, when the job is one handle's) is sealed, a job queued before either + * included (anything taking the line after the seal is refused, rule 4). + */ + async onLine(job: () => Promise, token?: SealToken): Promise { + if (this.#state !== "open" || token?.sealed) throw this.released(); + const release = await this.line.acquire(); + try { + if (this.#state !== "open" || token?.sealed) throw this.released(); + return await job(); + } finally { + release(); + } + } + + /** + * Runs `fn` as one transaction on the line under `token` and commits what it drafted (§9): a throw, an abort of + * `context`, the token's seal or the object's end discards everything. + */ + async tx(fn: (tx: Transaction) => R | Promise, token: SealToken, context: Context): Promise { + this.batch.assertOutside("tx"); + this.assertAdmitted(token); + const release = await this.line.acquire(); + const tx = new Transaction(this, token, release, context); + try { + this.assertAdmitted(token); + this.#holder = tx; + const result = await fn(tx); + // In-memory-only commits are synchronous (W12): no await unless the store's step is taken. + const step = tx.commit(); + if (step !== undefined) await step; + return result; + } catch (error) { + tx.discard(); + throw error; + } finally { + if (this.#holder === tx) this.#holder = undefined; + release(); + tx.settle(); + } + } + + /** + * Seals `token` (the seal fence): its transactions are refused from now on, and the one holding the line is + * discarded and the line released. + */ + fence(token: SealToken): void { + token.seal(); + if (this.#holder?.token === token) this.#cut(); + } + + /** Throws `released` once the object ended. */ + assertLive(): void { + if (this.#state === "ended") throw this.released(); + } + + /** Throws `released` once `token` is sealed or the object is ending: no new work takes the line. */ + assertAdmitted(token: SealToken): void { + if (token.sealed || this.#state !== "open") throw this.released(); + } + + /** The live object fails `broken` with `error` (a document the code cannot read); ignored when it is not live. */ + fail(error: ChordError): void { + this.#owner.fail(error); + } + + /** The object is ending: every transaction is refused, the one holding the line discarded; documents stay readable. */ + seal(): void { + if (this.#state !== "open") return; + this.#state = "sealed"; + this.own.seal(); + this.#cut(); + } + + /** + * Resolves once the line is free. After `seal` only an admitted step can hold it (every other holder was cut): + * the object's teardown waits here, so the step adopts into documents that still exist (slice 6a §1.3). + */ + async settled(): Promise { + const release = await this.line.acquire(); + release(); + } + + /** Drops every document and pin; watches end `ended`, and every later use throws `released`. */ + end(): void { + this.#state = "ended"; + this.residency.end(); + for (const slot of this.#slots.values()) slot.drop(); + this.#slots.clear(); + this.#pins = []; + } + + /** The `released` error of this object's documents. */ + released(): ChordError { + return new ChordError({ code: "released", reason: "ended", message: `${this.label}: ended` }); + } + + /** Discards the transaction holding the line and releases the line. */ + #cut(): void { + const holder = this.#holder; + this.#holder = undefined; + holder?.cut(); + } + + #add(slot: S): S { + this.#slots.set(slot.definition.current.name, slot); + return slot; + } + + /** The store side of persisted documents named `name`; the attempt checked the store first (P3). */ + #persistence(name: string): Persistence { + const { store, owner } = this.#owner.store!; + return new Persistence(store, owner, name); + } +} diff --git a/packages/chord/src/next/documents/fence.ts b/packages/chord/src/next/documents/fence.ts new file mode 100644 index 000000000..5f02888f4 --- /dev/null +++ b/packages/chord/src/next/documents/fence.ts @@ -0,0 +1,17 @@ +// The seal fence (slice 6a §1.3): the token an attempt, a job run's invocation or the object itself hands its +// transactions. Once sealed, new transactions under it are refused `released` and the open one is discarded. + +/** One seal token; sealed once, never unsealed. */ +export class SealToken { + #sealed = false; + + /** Whether work under this token is refused. */ + get sealed(): boolean { + return this.#sealed; + } + + /** Seals the token; `Documents.fence` also discards the transaction it holds open. */ + seal(): void { + this.#sealed = true; + } +} diff --git a/packages/chord/src/next/documents/handles.ts b/packages/chord/src/next/documents/handles.ts new file mode 100644 index 000000000..686d78e9a --- /dev/null +++ b/packages/chord/src/next/documents/handles.ts @@ -0,0 +1,133 @@ +// Document handles (slice 6a §1.2): what `self.state` returns, one per attempt and definition, a thin view of its +// object's slot bound to the attempt's seal token and the definition's version. + +import type { Context } from "../../context/index.ts"; +import type { Draft } from "../../delta/index.ts"; +import type { Watch, Watchable, WatchHandle } from "../../watch.ts"; +import type { JsonObject, StateDefinition } from "./definition.ts"; +import type { SealToken } from "./fence.ts"; +import type { KeyedSlot, KeySource } from "./keyed-slot.ts"; +import type { SingletonSlot } from "./slot.ts"; + +/** A singleton document: a watchable that always has a value (`initial()` until written, and after a retire). */ +export interface SingletonHandle extends Watchable { + /** One transaction that drafts this document and passes the draft to `fn`. */ + change(fn: (draft: Draft) => R, context: Context): Promise; + /** One transaction that retires this document: back to `initial()`. */ + retire(context: Context): Promise; +} + +/** + * Keyed documents: many per object, by string key. Not a watchable itself: each key is, through `watch(key)` and + * `at(key)`; a live collection is a singleton holding it. + */ +export interface KeyedHandle { + /** The document's value, `undefined` when absent (persisted and not in memory: loaded on the object's line). */ + get(key: string, context: Context): Promise; + /** The keys of the present documents, in code point order. */ + keys(context: Context): AsyncIterable; + /** + * A watch of the document at `key`, `undefined` when absent. It follows the address: a retirement is one frame with + * `retired: true` and `initial()`, a later creation a root replacement; it ends with the object. + */ + watch(key: string, context: Context): Promise | undefined>; + /** + * In the factory only: the document at `key` as a watchable (a state member, a derive source, an `on` source), + * `initial()` while absent, loaded before the object goes live and pinned while this attempt lives. + */ + at(key: string): Watchable; + /** One transaction that drafts the document at `key` and passes the draft to `fn`. */ + change(key: string, fn: (draft: Draft) => R, context: Context): Promise; + /** One transaction that retires the document at `key` (a no-op when absent). */ + retire(key: string, context: Context): Promise; +} + +/** The handle `self.state(definition)` returns. */ +export type HandleOf = D extends StateDefinition + ? K extends true + ? KeyedHandle + : SingletonHandle + : never; + +/** + * A singleton's handle. Reads go to the slot (allowed after the seal); `change` and `retire` take the line under the + * attempt's token. Exposed as a member or read by a derive, it resolves to the slot itself. + */ +export class Singleton implements SingletonHandle { + readonly slot: SingletonSlot; + readonly token: SealToken; + /** The version of the definition it was declared with: the slot refuses it once its values are newer (R15). */ + readonly version: number; + + constructor(slot: SingletonSlot, token: SealToken, version: number) { + this.slot = slot; + this.token = token; + this.version = version; + } + + get value(): JsonObject { + this.slot.definition.assertVersion(this.version); + return this.slot.value; + } + + async watch(context: Context): Promise> { + this.slot.definition.assertVersion(this.version); + // The slot's values are this handle's documents. + return this.slot.openWatch(context) as Watch; + } + + change(fn: (draft: Draft) => R, context: Context): Promise { + return this.slot.owner.tx(async (tx) => fn(await tx.draft(this)), this.token, context); + } + + retire(context: Context): Promise { + return this.slot.owner.tx((tx) => tx.retire(this), this.token, context); + } +} + +/** Who declares an attempt's `at(key)` pins: its factory, which alone may (the attempt). */ +export interface KeyDeclarer { + /** Pins `key` of `slot` for the attempt and returns its source; throws outside the factory. */ + at(slot: KeyedSlot, key: string): KeySource; +} + +/** A keyed handle: reads go to the slot; `change`, `retire` and cold reads take the line under the attempt's token. */ +export class Keyed implements KeyedHandle { + readonly slot: KeyedSlot; + readonly token: SealToken; + /** As `Singleton.version`. */ + readonly version: number; + readonly #declarer: KeyDeclarer; + + constructor(slot: KeyedSlot, token: SealToken, version: number, declarer: KeyDeclarer) { + this.slot = slot; + this.token = token; + this.version = version; + this.#declarer = declarer; + } + + get(key: string, context: Context): Promise { + return this.slot.get(key, this, context); + } + + keys(context: Context): AsyncIterable { + return this.slot.keys(this, context); + } + + watch(key: string, context: Context): Promise | undefined> { + return this.slot.watch(key, this, context); + } + + at(key: string): Watchable { + // The source's values are this slot's documents. + return this.#declarer.at(this.slot, key) as unknown as Watchable; + } + + change(key: string, fn: (draft: Draft) => R, context: Context): Promise { + return this.slot.owner.tx(async (tx) => fn(await tx.draft(this, key)), this.token, context); + } + + retire(key: string, context: Context): Promise { + return this.slot.owner.tx((tx) => tx.retire(this, key), this.token, context); + } +} diff --git a/packages/chord/src/next/documents/index.ts b/packages/chord/src/next/documents/index.ts new file mode 100644 index 000000000..e7568d8f0 --- /dev/null +++ b/packages/chord/src/next/documents/index.ts @@ -0,0 +1,5 @@ +// Documents for apps: definitions, handles and transactions. + +export { defineState, type JsonObject, type StateDefinition } from "./definition.ts"; +export type { HandleOf, KeyedHandle, SingletonHandle } from "./handles.ts"; +export type { Tx } from "./transaction.ts"; diff --git a/packages/chord/src/next/documents/internal.ts b/packages/chord/src/next/documents/internal.ts new file mode 100644 index 000000000..b2eee6b2b --- /dev/null +++ b/packages/chord/src/next/documents/internal.ts @@ -0,0 +1,9 @@ +// Documents for Chord's layers: an object's documents and slots, the handles' classes, the seal token, transactions. + +export { AddressSource } from "./address-source.ts"; +export { Documents, type DocumentsOwner } from "./documents.ts"; +export { SealToken } from "./fence.ts"; +export { type KeyDeclarer, Keyed, Singleton } from "./handles.ts"; +export { type KeyedSlot, KeySource } from "./keyed-slot.ts"; +export type { AnyDefinition, DocumentSlot } from "./slot.ts"; +export type { Staged, Transaction } from "./transaction.ts"; diff --git a/packages/chord/src/next/documents/keyed-slot.ts b/packages/chord/src/next/documents/keyed-slot.ts new file mode 100644 index 000000000..6575fd8ec --- /dev/null +++ b/packages/chord/src/next/documents/keyed-slot.ts @@ -0,0 +1,414 @@ +// Keyed slots (slice 6a §1.2, §4, §10, §11): the documents of one keyed name on one object, by key. In memory, every +// present document stays; persisted, a document is loaded on demand (a cold read is a job on the object's line, one +// load in flight per key, shared) and leaves memory when unpinned and idle for the retention (`residency.ts`). A key +// read through `at(key)` or a watch has its own source, which exists while the key is pinned. + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import { type Prepared, type Tracker, track } from "../../delta/index.ts"; +import type { Watch } from "../../watch.ts"; +import { ChordError } from "../data/index.ts"; +import { asChordError } from "../data/internal.ts"; +import { compareText, storedText } from "../store/text.ts"; +import { AddressSource } from "./address-source.ts"; +import type { JsonObject } from "./definition.ts"; +import type { Documents } from "./documents.ts"; +import type { SealToken } from "./fence.ts"; +import type { Loaded, Persistence, Stored } from "./persistence.ts"; +import { Resident } from "./residency.ts"; +import { type AnyDefinition, type DocumentSlot, type PersistedSlot, SlotDefinition } from "./slot.ts"; + +/** Whom a read is for: a handle's seal token and the version it was declared with (R15). */ +export type Access = { readonly token: SealToken; readonly version: number }; + +/** + * A cold load in flight, shared by its waiters: their handles' tokens, and those already sealed when the load took + * the line (`undefined` until then), whose waiters it refuses (rule 4). + */ +type SharedLoad = { + readonly done: Promise; + readonly tokens: Set; + refused: ReadonlySet | undefined; +}; + +/** The keys `keys(ctx)` reads per page from the store. */ +const KEYS_PAGE = 100; + +/** A keyed slot: one resident per document in memory, one source per pinned key. */ +export class KeyedSlot implements DocumentSlot, PersistedSlot { + readonly owner: Documents; + readonly definition: SlotDefinition; + readonly persisted: PersistedSlot | undefined; + readonly #persistence: Persistence | undefined; + readonly #residents = new Map(); + /** The sources of pinned keys: a key is pinned while it has one. */ + readonly #sources = new Map(); + /** Cold loads in flight, by key: shared by every reader of the key. */ + readonly #loads = new Map(); + /** Names a refused key (the slot's name is the same at every version): made once, not per key check. */ + readonly #keyWhat: string; + + constructor(owner: Documents, definition: AnyDefinition, persistence: Persistence | undefined) { + this.owner = owner; + this.definition = new SlotDefinition(definition); + this.#keyWhat = `${definition.name}: key`; + this.#persistence = persistence; + this.persisted = persistence === undefined ? undefined : this; + } + + get persistence(): Persistence { + // Only reached through `persisted`, which a persisted slot alone has. + return this.#persistence!; + } + + /** + * The value at `key`, `undefined` when absent. A document in memory is read at once (also through a sealed + * handle); a cold persisted one is loaded on the line (refused `released` once the access's token is sealed), and + * `context` ends only this caller's wait. An absent key is not cached (rule 20). + */ + get(raw: string, access: Access, context: Context): Promise { + // Not `async`: `#get` is the one async layer; a refused key still reaches the caller as a rejection. + try { + return this.#get(this.key(raw), access, context); + } catch (error) { + return Promise.reject(error); + } + } + + /** `get` of a key that went through `key` already. */ + async #get(key: string, access: Access, context: Context): Promise { + this.owner.assertLive(); + this.definition.assertVersion(access.version); + const resident = this.#resident(key); + if (resident !== undefined) { + this.#touch(resident); + return resident.tracker.value; + } + if (this.#persistence === undefined) return undefined; + await this.#loadShared(key, access.token, context); + return this.#residents.get(key)?.tracker.value; + } + + /** The present keys in code point order: in memory, the residents; persisted, the store's, a page per line job. */ + async *keys(access: Access, context: Context): AsyncIterable { + this.owner.assertLive(); + this.definition.assertVersion(access.version); + const persistence = this.#persistence; + if (persistence === undefined) { + yield* [...this.#residents.keys()].sort(compareText); + return; + } + let after: string | undefined; + for (;;) { + const page = await this.owner.onLine(() => persistence.keys(after, KEYS_PAGE, context), access.token); + yield* page; + if (page.length < KEYS_PAGE) return; + after = page.at(-1); + } + } + + /** + * A watch of the document at `key`, `undefined` when absent (after a cold load); it follows the address (a + * retirement is a frame with the `retired` flag) and pins the key until it ends. + */ + async watch(raw: string, access: Access, context: Context): Promise | undefined> { + const key = this.key(raw); + if ((await this.#get(key, access, context)) === undefined) return undefined; + const release = this.pin(key, context); + // Retired between the load and now: absent. + if (!this.#residents.has(key)) { + release(); + return undefined; + } + let watch: Watch; + try { + // The source's values are this slot's documents. + watch = this.#sources.get(key)!.openWatch(context) as Watch; + } catch (error) { + release(); + throw error; + } + void watch.closed.then(release); + return watch; + } + + /** + * Pins `key`: its source (made by the first pin) and its document stay in memory until the returned release + * (idempotent) runs; a persisted key not in memory is loaded by the next activation load (`loadDeclared`). + */ + pin(raw: string, context: Context): () => void { + const key = this.key(raw); + let source = this.#sources.get(key); + if (source === undefined) { + const ready = this.#persistence === undefined || this.#residents.has(key); + source = new KeySource(this, key, this.definition.current.initial(context), ready); + this.#sources.set(key, source); + } + source.pins++; + const resident = this.#residents.get(key); + if (resident !== undefined) this.owner.residency.forget(resident); + const pinned = source; + let released = false; + return () => { + if (released) return; + released = true; + if (--pinned.pins > 0 || this.#sources.get(key) !== pinned) return; + this.#sources.delete(key); + const idle = this.#residents.get(key); + if (idle !== undefined) this.#touch(idle); + }; + } + + /** Loads pinned `key` on the line unless it was loaded since it was pinned (a consumer's `at(key)`, slice 6a §12). */ + async ensure(raw: string, context: Context): Promise { + const key = this.key(raw); + if (this.#sources.get(key)?.ready !== false) return; + await this.owner.onLine(() => this.load(key, context)); + } + + /** The source of pinned `key`. */ + source(key: string): KeySource { + return this.#sources.get(this.key(key))!; + } + + /** + * `key` as the slot and the store keep it: the `str()` text rules (slice 6a §8), a lone surrogate replaced by + * U+FFFD, NUL refused `invalid/args`. Plugin keys are a trust boundary; every entry point takes them through here, + * so memory and the store agree on one key. + */ + key(key: string): string { + return storedText(key, this.#keyWhat); + } + + /** Loads every pinned key not loaded since it was pinned (activation, holding the line). */ + async loadDeclared(context: Context): Promise { + for (const [key, source] of this.#sources) if (!source.ready) await this.load(key, context); + } + + /** + * Loads the document at `key` (the caller holds the line) and installs it while it is still absent from memory (a + * commit's `adopt` always wins). A stored version newer than the code's, another fingerprint or a failing `migrate` + * fails the owner (slice 6a §1.4) and rejects with the cause. + */ + async load(key: string, context: Context, token?: SealToken): Promise { + if (this.#residents.has(key)) return; + // A store error rejects as the store's (`storage_failed`, `aborted`). + const document = await this.persistence.read(key, context); + // The transaction was cut meanwhile: what it read may be stale already. + if (token?.sealed === true) return; + let loaded: Loaded | undefined; + try { + loaded = + document === undefined ? undefined : this.persistence.decode(document, this.definition.current, context); + } catch (error) { + // What the code cannot read (a failing `migrate`, whatever it throws) fails the owner. + const failure = asChordError(error, this.owner.label); + this.owner.fail(failure); + throw failure; + } + if (loaded !== undefined && !this.#residents.has(key)) { + const resident = new Resident(this, key, track(loaded.value), loaded.stored, this.definition.current.version); + this.#residents.set(key, resident); + this.#touch(resident); + } + const source = this.#sources.get(key); + if (source !== undefined) source.ready = true; + } + + cold(key: string): boolean { + return this.#persistence !== undefined && !this.#residents.has(key); + } + + stored(key: string): Stored | undefined { + return this.#residents.get(key)?.stored; + } + + /** + * Takes the attempt's declaration: a newer version migrates each document in memory on its next access (persisted + * ones store a base with their next change); the pinned keys are watched, so they migrate now, all or none, and + * take `initial()` again. + */ + hydrate(definition: AnyDefinition, context: Context): void { + const from = this.definition.hydrated; + this.definition.hydrate(definition); + if (from === undefined || from.version === definition.version) return; + const pinned = [...this.#sources.keys()].flatMap((key) => this.#residents.get(key) ?? []); + const migrated = pinned.map((resident) => [resident, this.#migrated(resident, context)] as const); + for (const [resident, tracker] of migrated) { + resident.tracker = tracker; + resident.version = definition.version; + } + this.owner.batch.commit(() => { + for (const source of this.#sources.values()) source.retake(definition.initial(context)); + }, context); + } + + current(key: string): Tracker | undefined { + return this.#resident(key)?.tracker; + } + + /** The value a draft of an absent document starts from: a fresh `initial()`. */ + initial(context: Context): JsonObject { + return this.definition.current.initial(context); + } + + adopt(key: string, tracker: Tracker, prepared: Prepared, stored: Stored | undefined): void { + const source = this.#sources.get(key); + const before = source?.peek(); + tracker.adopt(prepared); + let resident = this.#residents.get(key); + const { version } = this.definition.current; + if (resident === undefined) { + resident = new Resident(this, key, tracker, stored, version); + this.#residents.set(key, resident); + } else { + resident.tracker = tracker; + resident.stored = stored; + resident.version = version; + } + if (source === undefined) this.#touch(resident); + else source.written(before!, prepared); + } + + /** Retires the document at `key` (inside the batch): out of memory, its source reads `initial()` with the flag. */ + retire(key: string): void { + const resident = this.#residents.get(key); + if (resident !== undefined) { + this.#residents.delete(key); + this.owner.residency.forget(resident); + } + this.#sources.get(key)?.retire(); + } + + /** The sweep found `resident` idle past the retention: it leaves memory. */ + evict(resident: Resident): void { + if (this.#residents.get(resident.key) === resident) this.#residents.delete(resident.key); + } + + /** Its object ended or no attempt declares it any more: the documents go and the keys' watches end. */ + drop(): void { + for (const resident of this.#residents.values()) this.owner.residency.forget(resident); + this.#residents.clear(); + for (const source of this.#sources.values()) source.end(); + this.#sources.clear(); + } + + /** + * A cold persisted key's load, one in flight per key, on the line under the owner's context (rule 11). The load is + * every waiter's: when it takes the line, a waiter whose handle was sealed meanwhile (its attempt failed live, the + * object still open) is refused `released`, and with no other waiter the load does not run (rule 4). + */ + async #loadShared(key: string, token: SealToken, context: Context): Promise { + // Also when a load is in flight: a sealed handle's cold read is refused (rule 4). + this.owner.assertAdmitted(token); + let load = this.#loads.get(key); + if (load === undefined) load = this.#startLoad(key); + load.tokens.add(token); + await awaitWithContext(load.done, context); + if (load.refused?.has(token)) throw this.owner.released(); + } + + /** Starts the shared load of `key`; it leaves `#loads` when it settles. */ + #startLoad(key: string): SharedLoad { + const tokens = new Set(); + const load: SharedLoad = { + tokens, + refused: undefined, + // The job runs once the line is taken, after `load` is assigned. + done: this.owner.onLine(async () => { + const refused = new Set([...tokens].filter(({ sealed }) => sealed)); + load.refused = refused; + if (refused.size === tokens.size) throw this.owner.released(); + await this.load(key, this.owner.context); + }), + }; + this.#loads.set(key, load); + const settled = (): void => { + this.#loads.delete(key); + }; + load.done.then(settled, settled); + return load; + } + + /** The resident at `key`, migrated to the slot's version first when it is older (migration on access). */ + #resident(key: string): Resident | undefined { + const resident = this.#residents.get(key); + if (resident === undefined || resident.version === this.definition.current.version) return resident; + const tracker = this.#migrated(resident, this.owner.context); + resident.tracker = tracker; + resident.version = this.definition.current.version; + return resident; + } + + /** + * `resident`'s value migrated from its version to the slot's (checked, R8). A throwing `migrate` fails the live + * owner `broken` (slice 6a §1.4) and throws the failure (an activation fails with it). + */ + #migrated(resident: Resident, context: Context): Tracker { + try { + return track(this.definition.current.migrate(resident.tracker.value, resident.version, context)); + } catch (error) { + const failure = asChordError(error, this.owner.label); + this.owner.fail(failure); + throw failure; + } + } + + /** A use of `resident` while unpinned: persisted, it is idle from now (Durable's granule rule). */ + #touch(resident: Resident): void { + if (this.#persistence !== undefined && !this.#sources.has(resident.key)) this.owner.residency.touch(resident); + } +} + +/** + * One pinned key of a keyed slot as a source (`handle.at(key)`, a keyed member's `at(key)`, `watch(key)`): the + * document's value, `initial()` while it is absent; retirements arrive with the `retired` flag. + */ +export class KeySource extends AddressSource { + readonly owner: Documents; + readonly #slot: KeyedSlot; + readonly key: string; + /** Its pins: at 0 the slot forgets it. */ + pins = 0; + /** Whether the key was loaded since it was pinned (always, in memory): its value is known. */ + ready: boolean; + #initial: JsonObject; + + constructor(slot: KeyedSlot, key: string, initial: JsonObject, ready: boolean) { + super(slot.owner.batch); + this.owner = slot.owner; + this.#slot = slot; + this.key = key; + this.#initial = initial; + this.ready = ready; + } + + get value(): JsonObject { + this.#slot.owner.assertLive(); + if (!this.ready) { + const message = `${this.#slot.definition.current.name}(${this.key}): read before it was loaded`; + throw new ChordError({ code: "not_ready", message }); + } + return this.peek(); + } + + /** The value without the checks (inside its slot's commits). */ + peek(): JsonObject { + return this.#slot.current(this.key)?.value ?? this.#initial; + } + + /** The document was retired: `initial()` from now, with the flag. */ + retire(): void { + this.retired(this.#initial); + } + + /** A new definition took over (inside a batch): `initial()` again, the value as a root replacement. */ + retake(initial: JsonObject): void { + this.#initial = initial; + if (this.ready) this.replaced(); + } + + /** Its slot was dropped: the watches end. */ + end(): void { + this.endWatches(); + } +} diff --git a/packages/chord/src/next/documents/line.ts b/packages/chord/src/next/documents/line.ts new file mode 100644 index 000000000..3f5b62a89 --- /dev/null +++ b/packages/chord/src/next/documents/line.ts @@ -0,0 +1,15 @@ +// An object's line (§9): its transactions, cold loads, key pages and eviction sweeps run on it one at a time, in the +// order they asked. + +/** An object's line: `acquire()` resolves with the release once every earlier holder released; releasing twice is harmless. */ +export class Line { + #tail: Promise = Promise.resolve(); + + /** Resolves with this holder's release once every earlier holder released. */ + acquire(): Promise<() => void> { + const { promise, resolve } = Promise.withResolvers(); + const previous = this.#tail; + this.#tail = promise; + return previous.then(() => resolve); + } +} diff --git a/packages/chord/src/next/documents/persistence.ts b/packages/chord/src/next/documents/persistence.ts new file mode 100644 index 000000000..7b6082286 --- /dev/null +++ b/packages/chord/src/next/documents/persistence.ts @@ -0,0 +1,156 @@ +// Persisted documents' store side (slice 6a §8, §10): a slot's address in the documents engine, what is stored of each +// loaded document (Durable's `LoadedDocument` fields), its loads with the version and fingerprint checks and the +// checked `migrate`, and the writes one commit stages for a changed or retired document. + +import type { Context } from "../../context/index.ts"; +import type { Prepared } from "../../delta/index.ts"; +import { + changeDocument, + createDocument, + type DocumentAddress, + type DocumentContent, + findDocument, + listDocumentKeys, + loadDocument, + retireDocument, + type StoredDocument, +} from "../store/engine.ts"; +import type { StagedWrite } from "../store/step.ts"; +import type { ChordStore } from "../store/store.ts"; +import type { JsonObject } from "./definition.ts"; +import type { AnyDefinition } from "./slot.ts"; + +/** What is stored of a loaded document: its record's id and revision, its base's version, the deltas after it. */ +export type Stored = { + readonly id: number; + readonly revision: number; + /** The version of the stored base: below the definition's, the next change stores a base (rule 1). */ + readonly version: number; + readonly deltasSinceBase: number; +}; + +/** A loaded document: its value at the definition's version, and what is stored of it. */ +export type Loaded = { readonly value: JsonObject; readonly stored: Stored }; + +/** + * What one commit stores of a document, in the store writes' words: a retirement (nothing stored after it), a + * creation whose id the step assigns (`create`, its position among the commit's creations), or a change of the record + * `stored` was prepared on. + */ +export type StoredChange = + | { readonly kind: "retire" } + | { readonly kind: "create"; readonly version: number; readonly index: number } + | { readonly kind: "change"; readonly stored: Stored }; + +/** The documents of one persisted definition name on one object: `(chord, :, name, key)`. */ +export class Persistence { + readonly store: ChordStore; + readonly #address: Omit; + + /** `owner` is `:` (W7); `kind` the definition's name. */ + constructor(store: ChordStore, owner: string, kind: string) { + this.store = store; + this.#address = { namespace: "chord", owner, kind }; + } + + /** + * The stored document at `key`, `undefined` when absent: found and loaded with plain reads (a store error rejects + * as the store's). Counted (`chord.store.loads`). + */ + async read(key: string, context: Context): Promise { + const address = this.#at(key); + const document = await this.store.read(async (executor) => { + const record = await findDocument(executor, address); + return record === undefined ? undefined : loadDocument(executor, record); + }, context); + if (document !== undefined) context.telemetry.metrics?.counter("chord.store.loads").add(1); + return document; + } + + /** + * `document` as `definition` reads it: a stored version newer than the code's, or the same version with another + * fingerprint (D9), throws; an older one is migrated and the result checked once (R8). What throws here is the + * code's (a failing `migrate` too, whatever it throws), never the store's. + */ + decode(document: StoredDocument, definition: AnyDefinition, context: Context): Loaded { + const { record, deltasSinceBase } = document; + const { name, version, fingerprint } = definition; + if (record.version > version) { + throw new Error(`${name}: stored at version ${record.version}, newer than the code's ${version}`); + } + if (record.version === version && record.fingerprint !== fingerprint) { + throw new Error(`${name}: stored with another schema at version ${version} (a schema changed without a bump)`); + } + // The engine stores what commits gave it: JSON objects of the record's version. + const stored = document.value as JsonObject; + const value = record.version === version ? stored : definition.migrate(stored, record.version, context); + return { + value, + stored: { id: record.id, revision: record.revision, version: record.version, deltasSinceBase }, + }; + } + + /** What is stored of the live document at `key` without loading it (a retirement's find), `undefined` when absent. */ + async find(key: string, context: Context): Promise { + const record = await this.store.read((executor) => findDocument(executor, this.#at(key)), context); + if (record === undefined) return undefined; + // The change counter is all a retirement needs; `deltasSinceBase` is not read for one. + return { id: record.id, revision: record.revision, version: record.version, deltasSinceBase: 0 }; + } + + /** Up to `limit` keys of live documents after `after`, in code point order. */ + keys(after: string | undefined, limit: number, context: Context): Promise { + return this.store.read((executor) => listDocumentKeys(executor, this.#address, after, limit), context); + } + + /** + * Stages one commit's writes of the document at `key` into `writes` (rule 8: serialized here, before the queue): + * its retirement when `retired` is stored, then its creation, or the change of `stored` (a base when created, when + * the stored version is older, or when `checkpointWhen` asks; a delta otherwise). `creations` counts the commit's + * creations so far (the step's ids follow their order). Returns what the commit stores. + */ + stage( + key: string, + definition: AnyDefinition, + stored: Stored | undefined, + change: { readonly prepared: Prepared | undefined; readonly retires: boolean }, + writes: StagedWrite[], + creations: number, + ): StoredChange { + const { prepared, retires } = change; + if (retires && stored !== undefined) writes.push(retireDocument(stored.id, stored.revision)); + if (prepared === undefined) return { kind: "retire" }; + const { version, fingerprint } = definition; + if (retires || stored === undefined) { + const base = { version, fingerprint, value: prepared.value }; + writes.push(createDocument(this.#at(key), "latest", base, JSON.stringify(prepared.value))); + return { kind: "create", version, index: creations }; + } + const info = { deltasSinceBase: stored.deltasSinceBase }; + const base = stored.version !== version || definition.checkpointWhen(prepared.value, prepared.ops, info); + const content: DocumentContent = base + ? { kind: "base", version, fingerprint, value: prepared.value } + : { kind: "delta", version, ops: prepared.ops }; + const text = JSON.stringify(base ? prepared.value : prepared.ops); + writes.push(changeDocument(stored.id, stored.revision, content, text)); + const deltasSinceBase = base ? 0 : stored.deltasSinceBase + 1; + return { kind: "change", stored: { id: stored.id, revision: stored.revision + 1, version, deltasSinceBase } }; + } + + /** The address of `key`: a singleton's (`""`) has no key. */ + #at(key: string): DocumentAddress { + return key === "" ? this.#address : { ...this.#address, key }; + } +} + +/** What is stored after a commit's `change` was adopted with the commit's created ids. */ +export function storedAfter(change: StoredChange, ids: readonly number[]): Stored | undefined { + switch (change.kind) { + case "retire": + return undefined; + case "create": + return { id: ids[change.index]!, revision: 1, version: change.version, deltasSinceBase: 0 }; + case "change": + return change.stored; + } +} diff --git a/packages/chord/src/next/documents/residency.ts b/packages/chord/src/next/documents/residency.ts new file mode 100644 index 000000000..b8aad848c --- /dev/null +++ b/packages/chord/src/next/documents/residency.ts @@ -0,0 +1,110 @@ +// Residency of keyed documents (slice 6a §10, rule 23): which keyed documents are in memory, and the eviction of +// persisted ones nobody pinned and nobody read or wrote for the retention. One idle structure and one timer per +// object, Durable's (`Session.#touchDocument`): an insertion-ordered map whose oldest entry the timer waits for, a +// touch moving an entry to the back at most once per granule (O(1) per touch, no timer per key). + +import type { Tracker } from "../../delta/index.ts"; +import type { JsonObject } from "./definition.ts"; +import type { Stored } from "./persistence.ts"; + +/** One keyed document in memory: its slot, its tracker and, persisted, what is stored of it. */ +export class Resident { + readonly slot: { evict(resident: Resident): void }; + readonly key: string; + tracker: Tracker; + stored: Stored | undefined; + /** The definition version the tracker's value is at: below the slot's, it migrates on its next access. */ + version: number; + + constructor( + slot: { evict(resident: Resident): void }, + key: string, + tracker: Tracker, + stored: Stored | undefined, + version: number, + ) { + this.slot = slot; + this.key = key; + this.tracker = tracker; + this.stored = stored; + this.version = version; + } +} + +/** How long an unpinned persisted keyed document stays in memory after its last read or write (slice 6a §10). */ +export const RETENTION_MS = 10 * 60_000; + +/** A touch moves an entry at most once per granule: a tenth of the retention, up to 1 s (Durable's rule). */ +const GRANULE_MS = Math.min(1_000, RETENTION_MS / 10); + +/** The idle persisted keyed documents of one object, oldest first, and the one timer evicting them. */ +export class Residency { + /** Runs a sweep on the object's line. */ + readonly #onLine: (sweep: () => void) => void; + readonly #idle = new Map(); + /** The pending sweep: when it is due and its timer; stays set until its job runs on the line. */ + #sweep: { readonly at: number; timer: ReturnType | undefined } | undefined; + #ended = false; + + constructor(onLine: (sweep: () => void) => void) { + this.#onLine = onLine; + } + + /** A use of an unpinned `resident`: it becomes the newest idle one (once per granule), and a sweep is scheduled. */ + touch(resident: Resident): void { + const now = Date.now(); + const since = this.#idle.get(resident); + // A hot document moves at most once per granule: it may then leave up to a granule early (a reload). + if (since !== undefined && now - since < GRANULE_MS) return; + this.#idle.delete(resident); + this.#idle.set(resident, now); + if (this.#sweep === undefined) this.#schedule(); + } + + /** `resident` is pinned or left memory: it is no longer idle. */ + forget(resident: Resident): void { + this.#idle.delete(resident); + } + + /** The object ended: nothing is evicted any more, the timer is cleared. */ + end(): void { + this.#ended = true; + this.#idle.clear(); + clearTimeout(this.#sweep?.timer); + this.#sweep = undefined; + } + + /** Schedules the eviction of the oldest idle document when its retention expires. */ + #schedule(): void { + const oldest = this.#idle.values().next(); + if (this.#ended || oldest.done === true) return; + const at = oldest.value + RETENTION_MS; + const sweep: { readonly at: number; timer: ReturnType | undefined } = { at, timer: undefined }; + this.#sweep = sweep; + // A timer firing early sweeps nothing and schedules the next one. + sweep.timer = setTimeout( + () => + this.#onLine(() => { + if (this.#sweep === sweep) this.#sweep = undefined; + this.#evictExpired(); + }), + Math.max(0, at - Date.now()), + ); + } + + /** + * On the line: evicts the idle documents whose retention passed, oldest first, then schedules the next sweep. No + * load in flight needs a pin against it: cold loads run on the line too, so a sweep never runs during one, and a + * load installing an unpinned key touches it (it is then the newest idle one). + */ + #evictExpired(): void { + if (this.#ended) return; + const expired = Date.now() - RETENTION_MS; + for (const [resident, since] of this.#idle) { + if (since > expired) break; + this.#idle.delete(resident); + resident.slot.evict(resident); + } + if (this.#sweep === undefined) this.#schedule(); + } +} diff --git a/packages/chord/src/next/documents/slot.ts b/packages/chord/src/next/documents/slot.ts new file mode 100644 index 000000000..657e8d90d --- /dev/null +++ b/packages/chord/src/next/documents/slot.ts @@ -0,0 +1,239 @@ +// Document slots (slice 6a §1.2): one per declared document name on its object, across the object's attempts. A slot +// holds the documents' trackers, the version and definition its values are of, what is stored of persisted ones, and +// the watches; each attempt's handles are thin views of it. This file: what every slot is, and the singleton's slot +// (the keyed one is `keyed-slot.ts`). + +import type { Context } from "../../context/index.ts"; +import { type Prepared, type Tracker, track } from "../../delta/index.ts"; +import { ChordError } from "../data/index.ts"; +import { AddressSource } from "./address-source.ts"; +import type { JsonObject, StateDefinition } from "./definition.ts"; +import type { Documents } from "./documents.ts"; +import type { SealToken } from "./fence.ts"; +import type { Persistence, Stored } from "./persistence.ts"; + +/** A document definition, whatever its value and keyedness. */ +export type AnyDefinition = StateDefinition; + +/** + * What a transaction drafts and adopts: documents of one object by key (a singleton ignores the key). The slots of + * `self.state`, and Chord's own (a job's run state documents). + */ +export interface DocumentSlot { + readonly owner: Documents; + /** The tracker of the document at `key`, `undefined` while absent. */ + current(key: string): Tracker | undefined; + /** The value a draft of an absent document starts from. */ + initial(context: Context): JsonObject; + /** The persisted slot's store side; `undefined` for documents in memory only. */ + readonly persisted: PersistedSlot | undefined; + /** Adopts a commit of the document at `key` (inside the batch), with what is stored of it after the commit. */ + adopt(key: string, tracker: Tracker, prepared: Prepared, stored: Stored | undefined): void; + /** Retires the document at `key` (inside the batch). */ + retire(key: string): void; +} + +/** What a transaction needs of a persisted slot (slice 6a §4, §10). */ +export interface PersistedSlot { + readonly persistence: Persistence; + readonly definition: SlotDefinition; + /** Whether the document at `key` may be stored without being in memory (a persisted keyed document not loaded). */ + cold(key: string): boolean; + /** What is stored of the document at `key` as memory knows it; `undefined` when absent or cold. */ + stored(key: string): Stored | undefined; + /** + * Loads the document at `key` (its caller holds the line): installed when present and still absent from memory, + * and, for a transaction's load (`token`), while the token is not sealed (a cut transaction released the line, so + * a successor may have changed or retired the key meanwhile). + */ + load(key: string, context: Context, token?: SealToken): Promise; +} + +/** + * The definition a slot's values are of (the one that hydrated it last; the first declaration until then): the checks + * of a new attempt's declaration and the old handles' version fence (R15). + */ +export class SlotDefinition { + #current: AnyDefinition; + #hydrated = false; + + constructor(definition: AnyDefinition) { + this.#current = definition; + } + + /** The definition the values are of. */ + get current(): AnyDefinition { + return this.#current; + } + + /** The definition the values were hydrated with, `undefined` before the first hydration. */ + get hydrated(): AnyDefinition | undefined { + return this.#hydrated ? this.#current : undefined; + } + + /** + * Checks a declaration of this name by a new attempt: a flipped `keyed` or `persist`, a version older than the values (old code) + * or the same version with another fingerprint (a schema changed without a bump) throw (the attempt fails broken). + */ + check(definition: AnyDefinition): void { + const { name, version, keyed, persist, fingerprint } = this.#current; + if (definition.keyed !== keyed) { + throw new Error(`${name}: declared ${definition.keyed ? "keyed" : "singleton"}, its documents are not`); + } + if (definition.persist !== persist) { + throw new Error(`${name}: declared ${definition.persist ? "persisted" : "in memory"}, its documents are not`); + } + if (definition.version < version) { + throw new Error(`${name}: version ${definition.version} is older than its documents' (${version})`); + } + if (definition.version === version && definition.fingerprint !== fingerprint) { + throw new Error(`${name}: the schema changed without a version bump (version ${version})`); + } + } + + /** Records the hydration with `definition`. */ + hydrate(definition: AnyDefinition): void { + this.#current = definition; + this.#hydrated = true; + } + + /** + * Throws `released` for a handle of `version` once the values migrated past it (old code reads no new shape), and + * `not_ready` for a handle of a newer version until its activation migrated them (new code reads no old shape: a + * retry's factory). + */ + assertVersion(version: number): void { + const current = this.#current; + if (version === current.version) return; + const message = `${current.name}: a handle of version ${version}; the documents are at version ${current.version}`; + if (version < current.version) throw new ChordError({ code: "released", reason: "version", message }); + throw new ChordError({ code: "not_ready", message: `${message} until the activation migrates them` }); + } +} + +/** + * A singleton's slot: one tracker while written, `initial()` while absent; the source its handles and members watch. + * Persisted, it loads once per object (again after a failed load) and stays in memory while the object lives. + */ +export class SingletonSlot extends AddressSource implements DocumentSlot, PersistedSlot { + readonly owner: Documents; + readonly definition: SlotDefinition; + readonly persisted: PersistedSlot | undefined; + readonly #persistence: Persistence | undefined; + /** `initial()`, taken at hydration; `undefined` before the first. */ + #initial: JsonObject | undefined; + #tracker: Tracker | undefined; + /** What is stored of it (persisted); `undefined` while absent. */ + #stored: Stored | undefined; + /** In memory: always; persisted: once loaded. */ + #loaded: boolean; + + constructor(owner: Documents, definition: AnyDefinition, persistence: Persistence | undefined) { + super(owner.batch); + this.owner = owner; + this.definition = new SlotDefinition(definition); + this.#persistence = persistence; + this.persisted = persistence === undefined ? undefined : this; + this.#loaded = persistence === undefined; + } + + get persistence(): Persistence { + // Only reached through `persisted`, which a persisted slot alone has. + return this.#persistence!; + } + + get value(): JsonObject { + this.owner.assertLive(); + if (this.#initial === undefined || !this.#loaded) { + const message = `${this.definition.current.name}: read before activation`; + throw new ChordError({ code: "not_ready", message }); + } + return this.#tracker?.value ?? this.#initial; + } + + /** + * Takes the attempt's declaration `definition` (`check` accepted it): the first takes `initial()`; a newer version + * migrates the value in memory (a persisted one stores a base with its next change) and takes `initial()` again, + * and a value that changed reaches watches as a root replacement; the same version keeps everything. + */ + hydrate(definition: AnyDefinition, context: Context): void { + const from = this.definition.hydrated; + if (from !== undefined && from.version === definition.version) { + this.definition.hydrate(definition); + return; + } + const before = from === undefined || !this.#loaded ? undefined : this.value; + const tracker = + this.#tracker === undefined || from === undefined + ? this.#tracker + : track(definition.migrate(this.#tracker.value, from.version, context)); + this.#initial = definition.initial(context); + this.#tracker = tracker; + this.definition.hydrate(definition); + if (before !== undefined) this.batch.commit(() => this.replaced(), context); + } + + /** Loads the persisted document unless it is loaded already (the caller holds the line). */ + async load(_key: string, context: Context): Promise { + if (this.#loaded) return; + const document = await this.persistence.read("", context); + // Singletons load at activation, on the line: what the code cannot read fails the activation. + const loaded = + document === undefined ? undefined : this.persistence.decode(document, this.definition.current, context); + if (loaded !== undefined) { + this.#tracker = track(loaded.value); + this.#stored = loaded.stored; + } + this.#loaded = true; + } + + /** Loads the persisted document unless it is loaded already (activation, holding the line). */ + loadDeclared(context: Context): Promise { + return this.load("", context); + } + + /** A singleton has one document: its key is always `""`. */ + key(_key: string): string { + return ""; + } + + cold(_key: string): boolean { + return false; + } + + stored(_key: string): Stored | undefined { + return this.#stored; + } + + current(_key: string): Tracker | undefined { + return this.#tracker; + } + + /** The value a draft of the absent document starts from: the `initial()` taken at hydration. */ + initial(_context: Context): JsonObject { + return this.#initial!; + } + + adopt(_key: string, tracker: Tracker, prepared: Prepared, stored: Stored | undefined): void { + // Unchecked: a step admitted before the object ended adopts after its seal (slice 6a §1.3). + const before = this.#tracker?.value ?? this.#initial!; + tracker.adopt(prepared); + this.#tracker = tracker; + this.#stored = stored; + this.written(before, prepared); + } + + /** Retires the document: it reads `initial()` again, and the frame says it was retired. */ + retire(_key: string): void { + this.#tracker = undefined; + this.#stored = undefined; + this.retired(this.#initial!); + } + + /** Its object ended or no attempt declares it any more: the document goes and its watches end. */ + drop(): void { + this.#tracker = undefined; + this.#stored = undefined; + this.endWatches(); + } +} diff --git a/packages/chord/src/next/documents/transaction.ts b/packages/chord/src/next/documents/transaction.ts new file mode 100644 index 000000000..0dfb6c1bb --- /dev/null +++ b/packages/chord/src/next/documents/transaction.ts @@ -0,0 +1,510 @@ +// Transactions (§9, R12, R13): drafts of an object's documents, prepared and adopted in one batch, with Chord's own +// staged changes (job runs, §15), fenced by the token they run under; they run on the object's line (`line.ts`). + +import { type Context, withoutAbortSignal } from "../../context/index.ts"; +import { type Change, type Draft, type Prepared, type Tracker, track } from "../../delta/index.ts"; +import { ChordError } from "../data/index.ts"; +import type { JobHandle, Outcome, Run, StartOptions } from "../jobs/index.ts"; +import type { Job } from "../jobs/internal.ts"; +import type { Columns, PutRowOf, RowOf } from "../store/columns.ts"; +import type { Condition, SetOf } from "../store/expression.ts"; +import type { ColumnRefs, Query } from "../store/query.ts"; +import type { StagedWrite } from "../store/step.ts"; +import type { ChordStore } from "../store/store.ts"; +import type { KeyOf } from "../store/table.ts"; +import { TableChanges } from "../store/table-changes.ts"; +import { type Page, type PageOptions, ROWS_PAGE, type Table, TableHandle } from "../store/table-handle.ts"; +import type { JsonObject } from "./definition.ts"; +import type { Documents } from "./documents.ts"; +import type { SealToken } from "./fence.ts"; +import type { Keyed, KeyedHandle, Singleton, SingletonHandle } from "./handles.ts"; +import type { KeyedSlot } from "./keyed-slot.ts"; +import { type Stored, type StoredChange, storedAfter } from "./persistence.ts"; +import type { DocumentSlot, SingletonSlot } from "./slot.ts"; + +/** What `self.tx` gives its callback. Inside it: only `await tx.*`, no other `self.tx`, no calls to other objects. */ +export interface Tx { + /** The document's draft: the same one for the same document within one transaction. */ + draft(handle: SingletonHandle): Promise>; + draft(handle: KeyedHandle, key: string): Promise>; + /** Retires the document when the transaction commits (its address reads `initial()` again); a no-op when it is absent. */ + retire(handle: SingletonHandle): Promise; + retire(handle: KeyedHandle, key: string): Promise; + /** + * Starts a run of `job`, one of this object's jobs, with this transaction's other writes (§15): both commit or + * neither. Returns its `Run` at once; a discarded transaction's `Run` rejects `released` from `outcome`. A key of an + * unsettled run (one started earlier in this transaction too) returns that run. + */ + start(job: JobHandle, args: A, options?: StartOptions): Run>; + /** The row with `key` as this transaction leaves it (its own `put`, `update`, `delete` overlaid; W10). */ + get( + table: Table, + key: KeyOf, + ): Promise | undefined>; + /** A page of committed rows, stable for the transaction (without its own writes). */ + page( + table: Table, + q: Query, + limit: number, + options?: PageOptions, + ): Promise>>; + /** Every committed row of `q`, as `page`. */ + rows(table: Table, q: Query): AsyncIterable>; + /** Writes `row` when the transaction commits (an upsert: the whole row replaced). */ + put(table: Table, row: PutRowOf): void; + /** Changes the row with `key` when the transaction commits; an absent key changes nothing. */ + update( + table: Table, + key: KeyOf, + set: SetOf | ((row: ColumnRefs) => SetOf), + ): void; + /** Deletes the row with `key` when the transaction commits. */ + delete(table: Table, key: KeyOf): void; + /** Changes every row matching `where` when the transaction commits (one statement, its count recorded). */ + updateWhere( + table: Table, + where: (row: ColumnRefs) => Condition, + set: SetOf | ((row: ColumnRefs) => SetOf), + ): void; + /** Deletes every row matching `where` when the transaction commits. */ + deleteWhere( + table: Table, + where: (row: ColumnRefs) => Condition, + ): void; +} + +/** + * Chord's own changes staged in a transaction (a job's run records): adopted in its commit step with the documents, + * then told how the transaction ended, once the owner's line is free. + */ +export interface Staged { + /** Adopts the changes, inside the commit's batch. */ + adopt(): void; + /** The transaction committed: the actions after the commit (a run starts, waiters settle). */ + committed(): void; + /** The transaction was discarded: nothing was adopted. */ + discarded(): void; +} + +/** + * What one transaction does with one document: a draft, a retirement, or both (retired, then drafted anew); `found` + * is what is stored of a cold persisted document it retires without loading (R13). + */ +type Entry = { + readonly slot: DocumentSlot; + readonly key: string; + draft: { readonly tracker: Tracker; readonly change: Change } | undefined; + retire: boolean; + found: Stored | undefined; +}; + +/** + * One document change a commit adopts: a removal, a draft's prepared change, or both; and for a persisted document + * what the commit stores of it. + */ +type Adoption = { + readonly entry: Entry; + readonly prepared: Prepared | undefined; + readonly retires: boolean; + readonly stored: StoredChange | undefined; +}; + +/** The created documents' ids of a commit without store writes: none. */ +const NO_IDS: readonly number[] = []; + +/** + * One transaction of an object, under the seal token of whoever runs it (an attempt, a job run's invocation, the + * object). Commit prepares every draft (Delta records ops and checks only placed values) and adopts the changed + * documents and the staged changes in one batch; drafts are revoked once it settles. Cut at its token's seal, it + * discards everything, releases the line and refuses every later use. + */ +export class Transaction implements Tx { + readonly #documents: Documents; + readonly token: SealToken; + readonly #release: () => void; + readonly #context: Context; + readonly #entries = new Map>(); + /** Chord's own staged changes, by their owner (a job); made by the first. */ + #staged: Map | undefined; + /** + * The store writes, in staging order, and the store they commit to; made by the first (the flag that sends the + * commit through the store's step, W12). + */ + #writes: { readonly store: ChordStore; readonly staged: StagedWrite[] } | undefined; + /** The tables written, each with its writes and overlay. */ + #tables: Map | undefined; + /** `tx.*` reads underway: a callback settling with one is refused (rule 3, as Durable's). */ + #pending = 0; + #state: "open" | "committing" | "stepping" | "committed" | "discarded" | "cut" = "open"; + /** Rejected by `cut` while the commit waits in the store's queue: its caller is told at once (rule 3). */ + #queued: PromiseWithResolvers | undefined; + + constructor(documents: Documents, token: SealToken, release: () => void, context: Context) { + this.#documents = documents; + this.token = token; + this.#release = release; + this.#context = context; + } + + /** The object whose transaction this is. */ + get documents(): Documents { + return this.#documents; + } + + draft(handle: SingletonHandle): Promise>; + draft(handle: KeyedHandle, key: string): Promise>; + draft(handle: SingletonHandle | KeyedHandle, key = ""): Promise> { + // Not `async`: the draft's promise is returned as it is, without another promise around it (the in-memory path + // stays as cheap as before the store); a throw still reaches the caller as a rejection. + let slot: SingletonSlot | KeyedSlot; + let at: string; + // Keyed keys follow the text rules (`KeyedSlot.key`); a singleton's key is always "". + try { + slot = this.#slotOf(handle); + at = key === "" ? key : slot.key(key); + } catch (error) { + return Promise.reject(error); + } + // The entry is made for this handle's slot, whose documents are `T`. + return this.draftSlot(slot, at) as Promise>; + } + + /** + * The draft of the document at `key` of `slot` (Chord's own slots, a job's run states). A cold persisted document + * is loaded first, inside the transaction (it holds the line); the draft counts as pending until it settles (rule + * 3). A warm draft settles at once. + */ + async draftSlot(slot: DocumentSlot, key: string): Promise> { + const entry = this.#entry(slot, key); + const persisted = slot.persisted; + if (entry.draft !== undefined || entry.retire || persisted?.cold(key) !== true) return this.#begin(entry); + this.#pending++; + try { + await persisted.load(key, this.#context, this.token); + this.#assertOpen(); + return this.#begin(entry); + } finally { + this.#pending--; + } + } + + retire(handle: SingletonHandle): Promise; + retire(handle: KeyedHandle, key: string): Promise; + async retire(handle: SingletonHandle | KeyedHandle, key = ""): Promise { + const slot = this.#slotOf(handle); + const at = key === "" ? key : slot.key(key); + const entry = this.retireSlot(slot, at); + const persisted = slot.persisted; + // A cold persisted document is found, not loaded (one read; absent: a no-op, R13). + if (entry.found === undefined && persisted?.cold(at)) { + entry.found = await this.#track(persisted.persistence.find(at, this.#context)); + } + } + + /** Retires the document at `key` of `slot` when the transaction commits. */ + retireSlot(slot: DocumentSlot, key: string): Entry { + const entry = this.#entry(slot, key); + entry.draft?.change.abort(); + entry.draft = undefined; + entry.retire = true; + return entry; + } + + start(job: JobHandle, args: A, options: StartOptions = {}): Run> { + // Job handles are Chord's own: `self.job` made them. + return (job as unknown as Job).stage(this, args, this.#context, options) as Run>; + } + + get( + table: Table, + key: KeyOf, + ): Promise | undefined> { + const handle = this.#handle(table); + const changes = this.#tables?.get(handle); + const read = changes === undefined ? handle.get(key, this.#context) : changes.get(key, this.#context); + // The handle's rows are the table's, typed by `C`. + return this.#track(read) as Promise | undefined>; + } + + page( + table: Table, + q: Query, + limit: number, + options: PageOptions = {}, + ): Promise>> { + // As in `get`; the query's table is checked against the handle's. + const read = this.#handle(table).page(q as unknown as Query, limit, this.#context, options); + return this.#track(read) as Promise>>; + } + + async *rows( + table: Table, + q: Query, + ): AsyncIterable> { + let after: string | undefined; + do { + const page = await this.page(table, q, ROWS_PAGE, after === undefined ? {} : { after }); + yield* page.items; + after = page.next; + } while (after !== undefined); + } + + put(table: Table, row: PutRowOf): void { + this.#changes(table).put(row); + } + + update( + table: Table, + key: KeyOf, + set: SetOf | ((row: ColumnRefs) => SetOf), + ): void { + this.#changes(table).update(key, set); + } + + delete(table: Table, key: KeyOf): void { + this.#changes(table).delete(key); + } + + updateWhere( + table: Table, + where: (row: ColumnRefs) => Condition, + set: SetOf | ((row: ColumnRefs) => SetOf), + ): void { + this.#changes(table).updateWhere(where, set); + } + + deleteWhere( + table: Table, + where: (row: ColumnRefs) => Condition, + ): void { + this.#changes(table).deleteWhere(where); + } + + /** The changes `owner` staged in this transaction, made by `make` on its first use. */ + stage(owner: object, make: () => S): S { + this.#assertOpen(); + this.#staged ??= new Map(); + let staged = this.#staged.get(owner); + if (staged === undefined) { + staged = make(); + this.#staged.set(owner, staged); + } + // Each owner stages one kind of change, the one its `make` returns. + return staged as S; + } + + /** The changes `owner` staged in this transaction so far, if any. */ + staged(owner: object): S | undefined { + // As in `stage`: the owner's own kind. + return this.#staged?.get(owner) as S | undefined; + } + + /** + * Prepares every draft and adopts what changed, and the staged changes, in one batch; nothing when nothing changed. + * A document retired and drafted anew is removed, then adopted: its frame carries the retirement. Persisted + * documents stage their writes here (rule 8), which sends the commit through the store's step. + */ + commit(): void | Promise { + this.#assertOpen(); + if (this.#pending > 0) + throw new Error(`${this.#documents.label}: the callback settled before its pending tx operations`); + this.#state = "committing"; + this.#context.abortSignal?.throwIfAborted(); + const adoptions: Adoption[] = []; + let creations = 0; + for (const entries of this.#entries.values()) { + for (const entry of entries.values()) { + let prepared = entry.draft?.change.prepare(); + if (prepared !== undefined && prepared.ops.length === 0) { + prepared.abort(); + prepared = undefined; + } + const { slot, key } = entry; + const retires = entry.retire && (slot.current(key) !== undefined || entry.found !== undefined); + if (prepared === undefined && !retires) continue; + const persisted = slot.persisted; + let stored: StoredChange | undefined; + if (persisted !== undefined) { + const { persistence, definition } = persisted; + this.#writes ??= { store: persistence.store, staged: [] }; + const change = { prepared, retires }; + const was = persisted.stored(key) ?? entry.found; + stored = persistence.stage(key, definition.current, was, change, this.#writes.staged, creations); + if (stored.kind === "create") creations++; + } + adoptions.push({ entry, prepared, retires, stored }); + } + } + const writes = this.#writes; + if (writes === undefined) { + this.#adopt(adoptions, NO_IDS); + this.#state = "committed"; + return; + } + return this.#step(writes.store, writes.staged, adoptions); + } + + /** + * Commits through the store's step (§3): admitted while neither cut nor sealed (else `released`, nothing written), + * then the documents and staged changes adopted inside the step. Once queued, the caller's abort no longer counts; + * a cut while queued rejects `released` at once (the queue drops the entry at its admission, writing nothing). + */ + async #step(store: ChordStore, staged: readonly StagedWrite[], adoptions: readonly Adoption[]): Promise { + const fence = { + admit: () => { + if (this.#state !== "committing" || this.token.sealed) return false; + this.#state = "stepping"; + return true; + }, + adopt: (_seq: number, ids: readonly number[]) => this.#adopt(adoptions, ids), + }; + this.#queued = Promise.withResolvers(); + try { + // The race handles the commit's own later rejection too. + await Promise.race([store.commit(staged, fence, withoutAbortSignal(this.#context)), this.#queued.promise]); + } catch (error) { + if (this.#state === "cut") throw this.#cutError(); + throw error; + } finally { + this.#queued = undefined; + } + this.#state = "committed"; + } + + /** + * Adopts the changed documents (persisted ones with what is stored of them, `ids` being the commit's created + * documents' ids) and the staged changes in one batch; nothing when nothing changed. + */ + #adopt(adoptions: readonly Adoption[], ids: readonly number[]): void { + const staged = this.#staged; + if (adoptions.length === 0 && staged === undefined) return; + this.#documents.batch.commit(() => { + for (const { entry, prepared, retires, stored } of adoptions) { + if (retires) entry.slot.retire(entry.key); + if (prepared === undefined) continue; + const after = stored === undefined ? undefined : storedAfter(stored, ids); + entry.slot.adopt(entry.key, entry.draft!.tracker, prepared, after); + } + for (const each of staged?.values() ?? []) each.adopt(); + }, this.#context); + } + + /** Aborts every draft and prepared change. */ + discard(): void { + if (this.#state === "cut") return; + this.#state = "discarded"; + this.#abortDrafts(); + } + + /** + * Its token was sealed while it held the line: every draft is aborted, the line released, and every later use, + * its commit included, throws `released`. + */ + cut(): void { + // A commit admitted to the step runs to its end; one still queued is refused at admission. + if (this.#state !== "open" && this.#state !== "committing") return; + this.#state = "cut"; + this.#abortDrafts(); + this.#release(); + this.#queued?.reject(this.#cutError()); + } + + /** Tells the staged changes how the transaction ended; runs once the object's line is free. */ + settle(): void { + if (this.#staged === undefined) return; + const committed = this.#state === "committed"; + for (const staged of this.#staged.values()) { + if (committed) staged.committed(); + else staged.discarded(); + } + } + + #abortDrafts(): void { + for (const entries of this.#entries.values()) for (const entry of entries.values()) entry.draft?.change.abort(); + } + + /** The slot behind a handle of this object, after the handle's version fence (R15). */ + #slotOf(handle: SingletonHandle | KeyedHandle): SingletonSlot | KeyedSlot { + // Handles are Chord's own: `self.state` made them. + const { slot, version } = handle as Singleton | Keyed; + slot.definition.assertVersion(version); + return slot; + } + + #entry(slot: DocumentSlot, key: string): Entry { + this.#assertOpen(); + if (slot.owner !== this.#documents) { + throw new Error(`${this.#documents.label}: a transaction drafts only its own object's documents`); + } + let entries = this.#entries.get(slot); + if (entries === undefined) { + entries = new Map(); + this.#entries.set(slot, entries); + } + let entry = entries.get(key); + if (entry === undefined) { + entry = { slot, key, draft: undefined, retire: false, found: undefined }; + entries.set(key, entry); + } + return entry; + } + + #assertOpen(): void { + if (this.#state === "open") return; + if (this.#state === "cut") throw this.#cutError(); + throw new Error(`${this.#documents.label}: the transaction has settled`); + } + + #cutError(): ChordError { + return new ChordError({ + code: "released", + reason: "discarded", + message: `${this.#documents.label}: the transaction was discarded at the seal`, + }); + } + + /** The handle behind `table`, one of this object's tables (another object's is refused). */ + #handle(table: object): TableHandle { + this.#assertOpen(); + // Plugin code may pass any table: only this object's handles are taken. + const handle = table; + if (!(handle instanceof TableHandle) || handle.owner.scope !== this.#documents) { + throw new Error(`${this.#documents.label}: a transaction writes only its own object's tables`); + } + return handle; + } + + /** The writes to `table` in this transaction, made by the first. */ + #changes(table: object): TableChanges { + const handle = this.#handle(table); + this.#tables ??= new Map(); + let changes = this.#tables.get(handle); + if (changes === undefined) { + this.#writes ??= { store: handle.store, staged: [] }; + changes = new TableChanges(handle, this.#writes.staged); + this.#tables.set(handle, changes); + } + return changes; + } + + /** Counts `read` as pending until the operation's promise settles (rule 3). */ + async #track(read: Promise): Promise { + this.#pending++; + try { + return await read; + } finally { + this.#pending--; + } + } + + /** The entry's draft, begun on the current document (or a fresh `initial()`) by the first use. */ + #begin(entry: Entry): Draft { + if (entry.draft === undefined) { + const { slot, key } = entry; + const current = entry.retire ? undefined : slot.current(key); + const tracker = current ?? track(slot.initial(this.#context)); + entry.draft = { tracker, change: tracker.beginChange() }; + } + return entry.draft.change.state; + } +} diff --git a/packages/chord/src/next/host/README.md b/packages/chord/src/next/host/README.md new file mode 100644 index 000000000..2474ca16a --- /dev/null +++ b/packages/chord/src/next/host/README.md @@ -0,0 +1,479 @@ +# host + +Hosts, facets, objects, natives and references: hosts serving `local` and `remote` types, with failures and retries, +principals and variants, links to children, to a parent and over routes to addressable hosts (with relays between +them), the heartbeat, the `ctx` of method handlers, (slice 4b) revocation, cycles across hosts, presence, +keep-alive counts, size limits per part and compression at the origin, (slice 5) routines and the objects' jobs +(declared here, built in `jobs/`), and (slice 6a) the host's store: tables, persisted documents and eager table +migrations at activation (built in `store/` and `documents/`), and live objects that fail. + +## For apps (`index.ts`) + +```ts +import { broken } from "../data/index.ts"; +import { createHost, defineFacet, sleep } from "./index.ts"; + +const facet = defineFacet((host) => { // a facet declares only: types to serve + host.serve(Notes, (_, self) => { // empty params: activated at load, lives until close + const notes = self.state(NotesDoc); + const info = self.get(SessionInfo, {}); + const watcher = self.open( // a native: opened during activation, closed at the end + async (ctx) => watchDir(info.dataDir.value, ctx.abortSignal), + async (w, reason) => w.close(), + ); + self.on(info.dataDir, async ({ value: dir }, ctx) => watcher.value.follow(dir, ctx)); // first the current value; never a failure + self.onLive(async (ctx) => { // once live; its ctx is aborted when the object ends + for (;;) { await sleep(HOUR, ctx); await compact(ctx); } + }); + return { + notes, + count: self.derive([notes], ([n]) => n.list.length), + add: async ({ text }, ctx) => { + await self.tx(async (tx) => { (await tx.draft(notes)).list.push({ text }); }, ctx); + return null; + }, + rescan: async (ctx) => watcher.value.rescan(ctx), + }; + }); + host.serve(Remote, (_, self) => { + self.open(async (ctx) => { + const config = readConfig(); + if (!config) throw broken(new Error("no remote configured")); // retrying is pointless + return connect(config, ctx.abortSignal); // a throw here is retried with backoff + }, (c) => c.close()); + return {}; + }); + host.serve(Draft, ({ conversation }, self) => { // params: lives while referenced + const store = self.get(Notes, {}); // live before this object is + return { summary: self.derive([store.count], ([c]) => `${conversation}: ${c}`) }; + }); + host.serve(NotesView, (_, principal, self) => { // remote: one object per principal key, one variant + const store = self.get(Notes, {}); + if (principal.kind === "guest") return { variant: "reader", members: { count: store.count } }; + return { variant: "editor", members: { count: store.count, add: (args, ctx) => store.add(args, ctx) } }; + }); +}); + +const host = createHost({ + principal: { kind: "server" }, // what this host's own requests act as + onError: (error) => log(error), + closeTimeout: 30_000, + readyTimeout: 60_000, +}); +await host.load([{ name: "@app/notes", facet }], ctx); // every host-lifetime object live or failed (reported) +const notes = await host.get(Notes, {}, ctx); // live and hydrated; waits through transient retries +await notes.add({ text: "milk" }, ctx); +notes.release(); +const draft = host.hold(Draft, { conversation: "c1" }, ctx); // a reference at once: loading, then live +(await draft.state.watch(ctx)).start(async (call) => call.kind === "value" && render(call.value)); // loading, live, failed, ended +if (draft.state.value.kind === "failed") draft.retry(); // a Retry button +host.accept(link, { principal: { kind: "guest", session: "S1" } }, ctx); // a child's link, authenticated by its entry point +await host.close(ctx); // closes links, then natives with reason "close" +``` + +Addressable hosts and routes (slice 4a): a server routes holds of worker types to the worker of their address. + +```ts +export const Server = hostKind("server"); +export const Session = hostKind("session", { id: str() }); +export const Notes = remote("notes", { version: 1, params: { session: str() }, + placement: ({ session }) => Session({ id: session }), variants: { … } }); + +const server = createHost

({ address: Server(), principal: { kind: "server" }, onError }); +server.route(Session, { // before load, one route per kind + authorize: (principal, address) => principal.kind === "owner" || principal.session === address.params.id, + resolve: async (address, context) => { // once at a time per address; context aborted at close + // No `signal: context.abortSignal`: it aborts at host close and would kill the worker (an `error` on the child); + // the worker exits when its link closes. + const child = fork("worker.ts", [], { serialization: "advanced" }); + child.send({ id: address.params.id }); // the app's own start message + return { link: ipcLink(child), principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, +}, ctx); // ctx: the parent of the route's work +await server.load(serverFacets, ctx); +wss.on("connection", (ws, req) => server.accept(webSocketLink(ws), { principal: authenticate(req) }, ctx)); + +// worker.ts: connect first (W6), then load +const worker = createHost

({ address: Session({ id }), principal: { kind: "worker", session: id }, onError }); +worker.connect(ipcLink(process), { principal: { kind: "server" }, trusted: true }, ctx); +await worker.load(workerFacets, ctx); + +// a tab: tab → server (relays, parses nothing) → route(Session) → worker +const notes = await tab.get(Notes, { session: "S1" }, ctx); +``` + +Authority and housekeeping (slice 4b): + +```ts +type Revocation = { grant: string } | { session: string }; +const server = createHost({ + address: Server(), principal: { kind: "server" }, onError, + revokedBy: (p) => (p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] : []), // the same on every host + compression: nodeCompression, // Node only: `src/next-node/compression.ts` +}); +server.revoke({ grant: "g1" }, ctx); // ends g1's objects here and on every worker, closes g1's links +host.serve(Notes, ({ session }, principal, self) => { // in a facet: `host.references` is the facet's host's too + const viewers = self.derive([host.references(Notes)], ([refs]) => refs.filter((r) => r.params.session === session).length); + … +}); +(await worker.activity.watch(ctx)).start(async (call) => call.kind === "value" && supervise(call.value)); // { references, calls, jobs, lingering } +server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded: 64 * MiB } }, ctx); +``` + +- `createHost({ address?, principal?, principalKey?, telemetry?, onError, closeTimeout?, readyTimeout?, + heartbeat?, revokedBy?, compression? })`; `R` is the app's revocation type (JSON). `address` (`Session({ id: "S1" })`, see `contracts/`'s `hostKind`) makes the host addressable: it serves only + objects placed there; without one it is a client. A host with a `principal` acts as it for its own requests; a host + without one (a client) has none and serves only local types (`serve` of a remote type throws at `load`). + `principalKey` maps a principal + to its part of an identity (L1's canonical JSON by default; the same function on every host). `closeTimeout` (30 s) + bounds each native's close and the drain of an ending object's calls; `readyTimeout` (60 s) bounds every activation + attempt (its wait for the previous instance's teardown included) and is the default deadline of a `get`. `heartbeat: { ping?, close? }`: a link silent for `ping` ms (15 s) + sends a `ping`, one silent for `close` ms (30 s), its handshake included, is closed (its references go `loading`); + one interval timer per host (`min(ping, close) / 10`), running only while it has links. A handler that blocks the + event loop longer than `close` gets its links closed. +- `revokedBy(principal) => R[]`: what revokes a principal; its revocation keys are the canonical JSON of each value, + computed once per principal object (cached weakly). `host.revoke(r, ctx)`, synchronously: every remote object whose + principal has `r` among its keys ends at once, whatever references remain (R11): each holder gets `forbidden` with + reason `revoked` (a UI can say "access revoked"; a hold over a link sends `gone` with it), running calls are aborted + with it (callers rejected), an activation in progress is abandoned; teardown follows as for any end (drain, closes + with reason `end`). Holds of the principal that arrived before the revoke but still wait (for `load`, or relayed for + their route's link) end too (`gone`). + Then every route link is sent `revoke { revocation }`, which the child applies with its own `revokedBy` (and passes + down its own route links), and every child link whose principal it revokes closes (a relay sends + `closed { origin }` upstream for it, after the `revoke`). Never this host's own principal, never the parent link; a + `revoke` arriving on any link but the parent is ignored; nothing is replayed to links made later. Revocation is an + end, not a blocklist: a later hold for the principal (a new authentication) binds a new object. Local objects have + no principal and are never revoked. +- `host.references(Type)` (also on a facet's `host`): a JSON watchable `{ params, principal?, incarnation }[]`, one entry per + reference to an object of `Type` here: a binding of this host (`host.get`, `self.get`, `ctx.get`) or a hold over a + link served here. `principal` is the reference's (absent for a local type); `incarnation` its caller's (a + tab's own through a relay, this host's for its own references; a `ctx.get` reference the call's). Entries are + added when a reference binds and removed when it ends, in the batch where that happens (a link drop is a departure + derives see at once); lingering objects have none. The list is made for the first `references(Type)` (with the + references existing then) and kept; while watched, a batch's changes are folded into the published list once (one + splice op per removal, one for the additions: a revoke of many holds is one list and one frame), otherwise it is + built on its next read. A departure discovered while a commit's links send (a transport failing on send) lands in + the next batch, so one commit stays one message per link and origin. +- `host.activity`: a JSON watchable `{ references, calls, jobs, lingering }`: holds arriving over links and served + here, method calls running here (local and remote), job runs running or finishing here, objects lingering after their + last reference. Counted where the work happens; the changes of one microtask turn are one publication + (`[["r", counts]]`), a slow watcher gets the newest counts; nothing is scheduled while nobody watches or derives from + it. Chord counts; the app's supervisor decides. +- `compression`: a synchronous `Compression` (`{ deflate(bytes), inflate(bytes, limit) }`, see `link/`): this host + deflates each JSON part above 16 KiB (UTF-8) it originates, whatever the link (IPC included), only if it shrinks + (values, ops, results, arguments; never params, never arguments sent up its parent link) and inflates deflated parts + it owns in place. Child links refuse compressed parts (a protocol error: clients never send them, and a small part + could inflate to 64 MiB). Without one (a browser) it sends uncompressed and inflates asynchronously + (`DecompressionStream`), only parts it owns once their id and placement are known and their wire size passed, + within the smaller of its link's decoded limit and a hold's stamped one, holding later messages of the link in + order; acks go out after a message was applied, so a slow inflate is backpressure like a slow consumer. +- **Cycles across hosts (Q5):** every object instance gets an instance id (uuidv7) at its first attempt; a new object + for an identity is a new instance. A hold made during an activation carries the chain of instance ids waiting for it + (the activating object's chain, which it got from the hold that created it or from the object that declared it + here, plus its own id; the last 32); honoured from trusted links only, relays forward it as it is. A provider refuses + a hold whose chain names the instance it would bind while that activates: broken `invalid/cycle` naming the cycle by + label where known ("app.a(1) → … → app.a(1)"); a local `self.get` of an activation in the chain is refused the same + way. A dependency failing with `invalid/cycle` fails its dependent broken with that error (owner, an exception to + R18's transient dependency failures: a retry would meet the same cycle). Known limitation: a chain is fixed when its + object is created, so a cycle entering an object that is already activating (two activations started at the same + moment) is not detected: both fail `transient` at `readyTimeout` and retry with backoff. +- `onError(error: HostError)` gets everything that fails in the background, by `kind` (`failure` where the value is a + `FailureCause`, `error` otherwise; a position is `{ kind: "on" | "onLive" | "native", position }`, from 1): + `activation_failed` (`object`, `failure`: every failed attempt), `object_failed` (`object`, `failure`: a live object + failed, slice 6a), `storage_failed` (`error`: the host's store stopped; the host closes itself), `derive_failed` + (`object`, `member?`, `error`: a derive's compute threw, once per failure episode, watched or not; `member` when it is + an exposed state member), `close_failed` (`object`, `native` position, `error`: a close that threw while nobody + waited), `close_timeout` (`object`, `native`: a close still running at its `closeTimeout`, a possible leak), `drain_timeout` (`object`: calls still running at `closeTimeout`; their natives close anyway), + `listener_error` (a throwing watch listener), `delivery_failed` (a frame delivery or a link's flush that threw) and + `protocol_error` (`error`: what closed a link, see `link/`) and `resolve_failed` (`address`: its key, `error`: a + route's `resolve` rejected; the address waits its backoff), `routine_failed` (`object`, `routine`: its position, + `error`: a routine call threw, or its watch could not be taken; a source's failure calls never reach a routine), `job_failed` (`object`, `job`, + `id`, `error`: a run threw, not after an abort), `job_finish_failed` (the same: a `finish` threw; it is retried) and + `job_dropped` (`object`, `job`, `id`: a blocked run dropped without `finish`, by `abort` or its object's end). A + throwing `onError` is ignored. +- `host.load([{ name, facet }], ctx)`: once per host; runs every setup, then activates every type with empty params + and resolves when each has had its first outcome, live or failed (the start barrier; later retries never hold it). + A setup that throws rejects it with that error and closes the host as `close` does, its links included. Gets wait + for it. +- `host.hold(Type, params, ctx, { readyTimeout? })`: a reference at once (`Held`: its `variant` is `undefined` until + its first binding), which follows its object until released and never rejects: `loading` while `load` runs, while + the object activates or retries, while there is no parent link or it is down; `failed` while the object is failed + and no attempt runs; `live`. Throws at once for invalid params (`invalid/params`) and a local type not served here + (`unavailable`); before `load` resolved + these are decided once it did, and the reference fails `broken` with that error. A placement or `authorize` that + throws or refuses, or a placement whose params fail its kind's check (`invalid/placement`), fails the reference + `broken`. A local type never crosses a link. Where a reference made here (`host.get`/`hold`, `self.get`, `ctx.get`) + or a hold arriving on a link goes, first match (`Placement`; the address is the contract's `placement` of the params, + computed by the caller; a hold carries it to be routed, and a host serving the type computes it again): + + | Case | Goes | + | --- | --- | + | the type is served here and its address is this host's, or the host or the type has none | here | + | a reference made here to a served type whose address is another host's | on, as if not served here (rows below): worker S1 reaches `Notes { session: "S2" }` through the parent | + | an arriving hold for a served type whose params place it on another host | broken `invalid/placement` (misrouted; never forwarded) | + | the address is of a kind this host has a route for | over that route's link (the params checked against the kind, `authorize` first; `forbidden` when it refuses) | + | otherwise (a remote type) | over the parent link, open yet or not; a hold that arrived *on* the parent link is never sent back: `unavailable` | + | an arriving hold and no parent link | `unavailable` | + + A hold that arrives and goes on is relayed (`remote/`): this host holds it upstream as the arriving hold's caller + and forwards between the two without parsing params, arguments, results or state. +- `host.get(Type, params, ctx, { readyTimeout? })`: a hold, awaited until live. Each getter has one deadline for its + whole wait, `load` and reconnects included. A broken failure rejects it at once with its error; through transient + failures and drops it waits, and at its deadline rejects with the last failure's error (`unavailable` "did not + become live within …" when none failed yet). A getter whose deadline passes or whose context is cancelled releases + only its own reference. +- `host.serve(Type, factory, { linger? })` in a facet's setup: a live object with params whose last reference is + released stays live for `linger` ms (0 by default), with its documents and dependencies; a reference within that + time reuses it, otherwise it ends. An object waiting, activating or failed when its last reference goes ends at once; + a failed state member does not end a lingering object; `host.close` ends lingering objects at once. +- `limits: { part?, decoded? }` on `accept`, `connect` and a route's `resolve` result: per part of what arrives on the + link, its size on the wire and after inflating (none on trusted links, 16 MiB and 64 MiB otherwise, unless set). + Checked before a part is parsed or inflated; past a limit only its hold or call fails (`too_large/part`: an argument + its call, a result its call, a snapshot or a state part its hold, which the consumer releases at the provider; that + failure is final: the reference leaves its path, a reconnect holds nothing again, and only `ref.retry()` places it + anew and holds it with a new id), the link stays. A relay + checks `part` on the admitting link and stamps that link's decoded limit on the upstream hold (`limit`, honoured + only on a parent link); calls on the hold inherit it, and the host that inflates enforces the smaller of its own and + the stamped one while inflating. A relay does not check answers from upstream: the end that parses a part checks it + against its own link's limits. Limits count UTF-8 bytes; byte segments are no parts (the transport's cap bounds + them). +- `host.accept(link, { principal, trusted?, codec? }, ctx)`: a link from a child (`codec`: `binaryCodec` by default, + the same at both ends; `trusted` skips the message schema check, see `link/`). Every hold and call arriving on it + acts as `principal`, stamped by this host (principal fields on its holds are ignored); its calls get the child's + incarnation as `ctx.incarnation`. `ctx` is the parent of the work arriving on it (its cancellation is dropped: a + link lives until closed). Every link carries both directions: this host may hold types over it too. +- `host.connect(link, { principal, trusted?, codec? }, ctx)`: the parent link. It throws while the previous one is + open (also while it is still closing); once that closed (a drop), `connect` takes the next. Every remote reference + of the parent path that is not released is held on it again, with this link's own ids (see `remote/`). `principal` + is who the parent is; holds arriving on it may name whom they act for (`principal`, `incarnation`), which this host + honours. Connecting before `load` lets factories that reference remote types go live without waiting (W6). +- `host.route(Kind, { authorize, resolve }, ctx)`, before `load`, once per kind (`ctx`: the parent of its `resolve`s + and of the work arriving on its links, with their telemetry; aborted for them at host close): how this host reaches + hosts of `Kind`. + `authorize(principal, address)` runs for every hold routed over it with the hold's principal, before anything + starts (`false`: broken `forbidden`; a throw: broken with that error). `resolve(address, context)` returns `{ link, + principal, trusted?, codec? }`, the address's link (role `route`); it runs at most once at a time per address, later + holds wait for it, and `context` is aborted only when the host closes (a link it returns later is closed at once). A + `resolve` whose holds were all released meanwhile finishes and its link is kept. A rejection (`resolve_failed`), a + link closed already, or the address's link closing while holds over it exist starts the address's backoff (1 s + doubling to 60 s, separate from objects' identities); the holds stay `loading` and resolve again after it while they + exist. Only a link that completed its handshake resets the backoff. A `resolve` returns the link before the worker + loaded (the worker connects first): waiting for its `load` could deadlock. Chord never closes a route link except + at `host.close`. When this host holds over a route link it sends whom each hold acts for (the table in `remote/`): + a guest's handler that `ctx.get`s a worker type reaches the worker as the guest. +- `host.close(ctx)`: aborts every route's `resolve`, closes its links first, children, routes, then the parent (no + `gone` goes out), ends every remote reference (`released/closed`, parked calls rejected with it), then ends every + object at once (calls fail `released/ended`, `self.tx` throws; documents stay readable until it is done), aborts pending opens, waits for endings already running, then drains and + closes object by object, the last to go live first (a dependent before its dependencies), with reason `close`. + Every call returns the same closing. It rejects with an `AggregateError` of the closes that threw. It does not wait + for an open that ignores its abort: that value is closed when it arrives (a throw is `close_failed`). +- `defineFacet

(setup)`: `setup(host)` declares only, synchronously: `host.serve(Type, factory, options?)` and + `host.references(Type)`. Facets have no documents, routines, natives or jobs (slice 6a): state and work belong to + objects. A host for `P` takes only facets typed for exactly `P`. +- `createHost({ store?, ... })` (slice 6a): the host's store (`openStore` in `store/`), which the app opens, gives to + one open host at a time (`invalid/args` otherwise) and closes after it. A backend error stops the store: the host + reports `storage_failed` once and closes itself. A `local` type with empty params may then declare + `self.table(definition)` (once per table name) and persisted documents (`defineState(..., { persist: true, + checkpointWhen })`); the types refuse both elsewhere (P3), and so does the factory at runtime. Its activation + loads the persisted singletons and pinned keys, then creates or migrates its tables (the `readyTimeout` deadline + paused meanwhile, resumed with its remaining budget), then binds its references. A table or document the code + cannot read (a newer version, a history edited in place, a failed `rows` step or `migrate`) fails the object + `broken`; a live object whose read fails so fails while live (`object_failed`), keeping its documents for the + next attempt. +- A factory `(params, self) => implementation` declares with `self.state`, `self.derive`, `self.get`, `self.open`, + `self.on`, `self.onLive` and `self.job` (`jobs/`), and writes with `self.tx`. It runs once per attempt. The implementation has exactly the contract's members: state + members are JSON watchables (document handles, derives, reference members, constants), methods `(args, ctx) => + Promise`, or `(ctx) => Promise` for a method without arguments (the reference omits them too: + `notes.clear(ctx)`). A remote factory `(params, principal, self) => ({ variant, members })` gets the principal its + object is for and returns one variant with exactly that variant's members. +- **Routines** (§14): `self.on(source, fn)` and `self.onLive(fn)` start right after the object went live, in the same + turn (a method call admitted in that turn may run before the first `on` call), and stop when it ends: their ctx is + aborted with the end's error and the drain waits for their running calls (each admitted before its handler runs: a + handler may end its own object before its first await). Once the object ended no further call starts, and a routine + that ends its object as it starts (an `onLive` releasing the last reference) starts none after it. Each call gets its + own ctx (the object's principal for a remote type, the host's for a local one; the host's incarnation; `ctx.get` + released when the call settles). They are not counted in `host.activity`. + - `on` over a JSON watchable (one whose watch is Chord's shared `Watch`: a document, a derive, a reference member, a + constant, a Durable view) calls `fn(call, ctx)` with the value call `{ kind: "value", value, ops, retired }`; over a + native watchable (whose watch is the shared `NativeWatch`: `ref.state`, a registry value) `fn({ kind: "value", + value }, ctx)`, as the overloads type it. Failure calls never reach `fn`: the routine waits for the next value (the + failure is reported where it happens, a derive's as `derive_failed`). The first call + gets the current value as `[["r", value]]`, then one call per change; calls never overlap, every change made during + a call (the first one and retries included) folds into one next call (the newest value; for JSON its folded ops, + and `retired` when any change was a retirement), made one macrotask after the busy call so every value its watch + still buffers is taken first (a native watch replays them one per microtask), and none aborts it. A call that + throws is reported (`routine_failed`) and the backoff starts, also when the call changed the source (a handler + writing its own source and throwing never feeds itself): after 1 s, doubling to 60 s, it calls with the newest + value, or the last call's, as a whole value, `retired` again if the failed call had it; a change during the backoff + calls at once. Any call cancels a pending retry, and a call that succeeds resets the backoff. The source ending (a + document of an object that ended, a released reference) ends the routine without a report; a watch that cannot be + taken ends it with one report (`routine_failed`). A retirement (a Durable document's; + Chord's own documents retire this way from slice 6a) is a call with `retired: true`, and the routine goes on. + - `onLive(fn)` is called once: returning ends it, a throw is reported and restarts it after the same backoff (reset + after a run of at least a minute); an abort (the object ends) is no error. + - `sleep(ms, ctx)` resolves after `ms` and rejects with the abort reason once `ctx` is aborted, clearing its timer and + listener (delays above 2^31-1 ms are chained timers); `untilAborted(ctx)` resolves once `ctx` is aborted. Routines + and handlers wait only through calls that take their ctx. +- `ctx` (`Ctx

`) in a handler: `principal` (the caller as its hold's link stamped it, or as a parent link's hold + named it; on a local object's handler the host's), `incarnation` (the calling host's over a link, the origin's + through a relay, this host's otherwise, R20), `get(Type, params, { readyTimeout? })` (a reference as + `ctx.principal` sees it, with `ctx.incarnation`, also over a route; released when the handler settles; afterwards + `get` throws `released`), the abort signal (the caller cancelled or released, or the object ends) and telemetry. +- `self.open(open, close, { closeTimeout? })` returns a `Native`: its `value` is readable from the moment its open + resolved until its close starts; before, it throws `not_ready`, after, `released`. `open(ctx)` gets a context + aborted when the attempt is abandoned; `close(value, reason, ctx)` gets `{ kind: "end" | "close" | "reload" }` (end: + last reference gone or a failed attempt; close: `host.close`; reload: slice 7) and a context aborted at its + `closeTimeout`. Throw `broken(error)` from an open when retrying is pointless. +- A reference has the contract's members, `state`, `retry()` and `release()`. A remote reference has `variant`, its + discriminant (`RefIn` narrows the type), `undefined` before its first binding and the last bound variant while + loading or failed; at runtime it carries every member of every variant, and one outside the bound variant fails + `invalid/member`. +- `ref.state` (a `NativeWatchable`, made on first access): `{ kind: "loading" } | { kind: "live" } | { kind: + "failed"; cause; retryAt }` (`retryAt`: when the object's backoff window ends, `undefined` for a broken failure) `| + { kind: "ended"; error }` (for good: `released` without a reason when its holder released it, `released/ended` when + its object ended, `released/closed` at host close, `forbidden/revoked`). A watch gets the current state, then every + change; it ends `ended` when the reference ends. What a + reference does in each state: + + | State | Calls | State members | + | --- | --- | --- | + | `loading` | park: sent in the order made once live, each until the reference's `readyTimeout` (`unavailable/park_timeout`) | the last value; before the first binding a read throws `not_ready` and a `watch` waits for the first value | + | `live` | run | readable | + | `failed` | reject at once with the cause's error | a local object that went live once (it failed live, or a retry failed): the last values, a new `watch` opens at them, no frames until a new attempt is live; otherwise reads throw the cause's error and existing watches stay open | + | `ended` | throw its `error` | reads throw `released`; watches end `ended` | + + Calls made after the reference went live never overtake queued ones (a call a handler makes while they are sent + queues behind them). A parked call is checked against its deadline when it is about to be sent, and against the + variant it is sent on; it also leaves the queue when its caller cancels (rejected with the reason), the reference + fails (the cause's error) or is released (`released`). A `host.hold` released before `load` resolved makes nothing. +- A local reference follows its object's state members through member sources (`member-source.ts`, one per object and + member): when a new attempt goes live every member re-binds by name to it (watches get one frame from the last + value, other objects' derives over the member recompute, a failed one included), as a remote reference's re-hold + does. The keys a reference holds on a keyed member (`at(key)`, `watch(key)`) follow the object the same way + (`keyed-members.ts`, one source per member and key; the new attempt's member loads them before it goes live). A key + whose first load is still in flight when its object fails or ends is given up: the activation waiting on it fails + `transient` (R18) and retries on the next attempt; the late pin is released once. +- `ref.retry()`: a failed object (broken too) gets an attempt now, its backoff window skipped; every reference to it + goes `loading`. Ignored while an attempt runs, once live and once released; over a link it is a `retry` message. + +Rules: + +- One object per (type, params), per (type, params, principal key) for a remote type. A local type with empty params + lives from `load` to `close`, failed or not; every other type from its first reference (a get, a hold over a link) + to its last release, failed or not. `load` never activates remote types. +- **Principals of same-host references:** `host.get` acts as the host's principal; `ctx.get` as `ctx.principal`; + `self.get` in a remote object's factory as that object's principal, in a local object's factory as the host's. Over + a route link they travel with the hold; over the parent link the parent acts as this host (upward is a boundary: a + worker that acts for a user towards the server passes the user as data and the server object decides). +- **Activation**, one attempt: the factory; singleton documents hydrated; references bound and live; natives opened one + after another in declaration order (an opener may read documents, references and earlier natives, not derives); + members checked; derives computed; live. An attempt not live within `readyTimeout` is abandoned: its openers' + contexts are aborted, what it opened is closed (`end`), an open resolving later is closed at once. +- **Failures** become a `FailureCause` with a ChordError; the kind comes from where it failed: the factory, a + singleton's `initial()`, the member check (a native watchable as state included), a first derive compute or a + dependency cycle are `broken`; a throwing open is `transient` unless it threw `broken(error)`; the `readyTimeout` is + `transient`; a dependency that failed (R18) is `transient` with `unavailable` "A: dependency B failed" and B's error + as `cause`. A keyed document's `initial()` runs in a later transaction and fails only that transaction. A cycle + (`a(1) → b(1) → a(1)`) fails every object on it. +- **Retries:** one attempt per object at a time. A transient failure is retried after a backoff per identity: 1 s, + doubling to 60 s, reset by a success; a new object for an identity inside its window waits for the window's end. + A broken failure is retried only by `ref.retry()` (and reload, slice 7, `retryOn`, slice 8); a new object after its + end tries at once. Only a dependency failure, or a `readyTimeout` while waiting for dependencies, keeps the object's + references; when that dependency goes live (a remote one: bound again after a drop) the object is retried at once. A dependency failure is transient whatever the dependency's kind: a dependent of a broken dependency is + retried with its backoff for as long as it is held. A retry binds its references before the previous attempt's are + released, so a dependency held only by the retrying object never ends in between. +- **Ending:** transactions refused and the callers of running calls rejected `released` at once, their handlers' + contexts aborted, routines' and job runs' too (their waiters rejected); then the drain (the handlers', routine + calls' and runs' own promises, a run's `finish` included, up to `closeTimeout`), the natives closed in reverse + (each up to its `closeTimeout`; a throw never stops the others), the documents (readable until then) dropped, the + derives detached, the references released, every watch on its members ended (`ended`). A new instance or attempt + for the identity starts only after that; gets wait. +- Calls check the method against the bound variant first (`invalid/member`), then the arguments against the provider's + current version (`invalid/member`, `invalid/args`); on one host arguments and results pass by reference. +- **Release** rejects the calls made through the reference with `released` and aborts their handlers' contexts, on one + host as over a link. A get needs the provider's version and fingerprint (`contract_mismatch`); an unserved + type is `unavailable`, a closed host `released`. +- Watches taken through a reference end when it is released, even while others keep the object alive. +- A failure's error keeps its stack frames' receivers alive until its `stack` is read (V8), its failed attempt among + them (natives closed, documents dropped). An object keeps its last failure's error until it goes live or ends; an + app logs the errors `onError` gets and drops them. + +Patterns for natives (documentation, nothing to build). Restarting a native with work in flight (R16): a restart +(reload, slice 7) closes the old native, then opens the new one, one after the other. Two patterns keep in-flight work +alive: + +- **Drain in the background:** `close` detaches the old resource and returns at once; the resource disposes of itself + when its work has finished, or after the native's `closeTimeout` at most (code mode's sandbox). Chord then opens the + new one immediately; callers look the current resource up when they start (`watcher.value` at each use). +- **Swap through a registry** (slice 8): the resource is a registry contribution; readers always find the current one, + and a reader that must wait for one waits, cancellably, until one is present. + +A native that follows a relocation (design §6) is not restarted at all: the native is a stable manager opened once; a +routine on the session location (slice 5) switches the resource it manages (the watched directory) and bumps a +generation, so late events from the old location are dropped. + +Runnable examples, one topic each, in `test/next/examples/` (run from `packages/chord` with +`node --conditions=source --experimental-strip-types test/next/examples/`): `01-objects.ts` (contracts, facets, +references, calls, lifetimes), `02-documents.ts` (singleton and keyed documents, transactions, shorthands), +`03-derives.ts` (derives, `mapShared`, derives over other objects), `04-watches.ts` (frames, folding for a busy +listener, retire, cancellation, release), `05-errors.ts` (invalid arguments and params, unknown members, contract mismatch, released references, +cycles, throwing factories), `06-natives.ts` (opening in order, `not_ready`, close reasons, throwing and slow closes), +`07-failures.ts` (transient and broken failures, backoff, dependency recovery, `readyTimeout`, the start barrier), +`08-remote.ts` (two hosts, variants per principal over a shared store, replicas), `09-calls.ts` (calls over a link, +bytes, errors, `ctx.principal`, cancellation, release), `10-drops.ts` (a hold before any link, `ref.state`, parked +calls, a drop and the re-hold), `11-linger-and-retry.ts` (`linger`, failed references, `ref.retry()`), +`12-routes.ts` (addresses, placement, a route to an in-process worker, a guest's `ctx.get` through it), +`13-relay.ts` (tab, server and a forked worker over WebSocket and IPC, the worker's start data through its core +facet, §18), `14-flow-control.ts` (a slow tab: folding at the provider, and at the worker per origin through a +relay while a fast tab keeps receiving), `15-revocation.ts` (a grant revoked: its tabs' links closed, its objects on +the worker ended), `16-presence-and-activity.ts` (a viewer count from `host.references`, departures, the activity a +supervisor watches), `17-limits-and-compression.ts` (a 500 KB snapshot compressed or not, a part past a limit, a +compression bomb), `18-routines.ts` (a reconciling `on` with a retry, a periodic `onLive` with `sleep`) and +`19-jobs.ts` (a shell job with a throttled state document, per-principal views of `running`, an abort and a chained +job), and (slice 6a) `20-tables.ts` (tables, writes, pages, counts), `21-persisted-documents.ts` (documents across two +hosts on one store file), `22-migrations.ts` (`evolveTable` and an eager migration at activation), +`23-keyed-watches.ts` (`watch(key)`, `at(key)`, the `retired` flag), `24-keyed-members.ts` (a keyed member pinned per +key by its consumers) and `25-export-import.ts` (an archive in JSONL and an import). + +## For Chord's layers (`internal.ts`) + +For `remote/`: `ChordHost` (`place` decides where an arriving hold goes, here or relayed over a path; `serving` +finds the object a hold placed here binds to; `relays` and `relayOrigin`, the origins this host relays holds for by +their number; `pendingLoad`, `closing`, `linkOpened`, `unlink`), `AnyHost` (a host whatever its principal type: `ChordHost

` is invariant in `P`), +`Target` (what a binding is bound to, implemented by `ChordObject` and by `remote/`'s `RemoteObject`), +`ChordObject` (`state`, `retry`) and `Holder` (what holds an object: a binding, or a hold over a link: `moved()` when +the object's `state` changed, which a new holder also reads itself; `end(error)`), `Binding` (`moved`, `end`), +`CheckedParams`, `Caller` (principal and incarnation of a reference's maker or a hold's origin), `RefState` with `LOADING`, `LIVE`, `RELEASED`, and +`settlement`. Also on `ChordHost` for `remote/`: `revoke` (a parent's `revoke` message), `compression`, `count(kind, +delta)` (the keep-alive counts), `cycle(chain, id, closing)` (the `invalid/cycle` error naming a chain by labels), +`unplaced` (the path of remote references failed for a part past the limits) and `pathOf(Type, params, principal)` +(placing one anew on `retry`). + +Inside the directory: `ChordHost` (types, objects by identity, facets, the batch, the backoff map, the endings new +instances wait for, its links and the heartbeat timer, its routes and the parent path; `reference` checks a +reference's route, `bind` binds it to an object here or a remote object over a path, `acquire` gets one for a +caller), `Placement` (where a reference or an arriving hold goes: here, a route's path or the parent's; whether a hold +served here is placed here), `Route` (one per routed kind: per address its `Path`, at most one running +`resolve`, the backoff per address; an address is kept while it has holders, a link or a running `resolve`), +`ChordObject` (one per identity: its state across attempts, holders, retries, cycle checks with its instance id and +the chain it was created for, lingering, ending and revocation, and the count of its running and finishing job runs, +which keeps it like a reference: `countRuns`), `Activity` (`host.activity`: counters, one +publication per microtask turn), `References` (one type's presence list), `RevocationKeys` (each principal's keys, +in a `WeakMap`), +`Attempt` (one activation: it is the +factory's `self`, runs the steps with their failure kinds, takes the calls, starts its routines and jobs once live, +drains and closes), `Routines` (an owner's routines: declared, then started once for a `RoutineOwner`; each `on` +follows its source with a watch whose listener never throws and never waits: it keeps the newest frame, JSON ops +folded by Delta's `PendingOps`, and the routine makes one call at a time from it, the first call and retries +included; its state is tagged: starting, idle, calling, backoff with its timer, ended), +`sleep`, `NativeHandle` (one +native: opened once, closed once, a late value closed at once; `within` bounds a close or a drain), `Backoff` (the +windows per identity and when each ends, holding no object), `Binding` (a reference's link to its object or remote +object: its state, the object that declared it, its per-reference watches, calls, parked calls and the waits for a +state; released, it keeps only its name), `ParkedCalls` (a reference's queue of calls made while loading, a timer +each), `ReferenceState` (`ref.state`: its native watches, told of each change), `CallContext` (a handler's +`ctx`), `withLinkedAbortSignal` (a call's or a get's context aborted by the caller and by a long-lived signal, its +reference's or its attempt's, linked only until it settles: `AbortSignal.any` keeps cleanup entries per call while its +sources live before Node 24.16.0), `checkImplementation` (the variant and member check), `setUp` (runs one facet's setup) and +`resolveWatchable` (the member check, Q14), and `nextDelay` (the 1 s to 60 s doubling every backoff shares). Cycle checks follow the waits of activating objects with a visited set, +linear in the waits. diff --git a/packages/chord/src/next/host/activity.ts b/packages/chord/src/next/host/activity.ts new file mode 100644 index 000000000..badc5a609 --- /dev/null +++ b/packages/chord/src/next/host/activity.ts @@ -0,0 +1,56 @@ +// Keep-alive counts (§3, R26): `host.activity`, a JSON watchable of the work keeping a host busy, for the app's +// supervisor. Counters are kept where the work happens; their changes in one microtask turn are one publication. + +import { BACKGROUND_CONTEXT, type Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { Source } from "../state/internal.ts"; + +/** What `host.activity` counts. */ +export type ActivityCounts = { + /** Holds arriving over links and served here. */ + readonly references: number; + /** Method calls running on this host, local and remote. */ + readonly calls: number; + /** Running jobs (slice 5). */ + readonly jobs: number; + /** Objects kept live by their type's `linger` after their last reference. */ + readonly lingering: number; +}; + +/** + * `host.activity`. A count marks it changed; while it is watched or derived from, the first change of a microtask turn + * schedules one publication, a batch that gives its watches the counts as they are then (`[["r", counts]]`: a slow + * watch gets the newest counts) and marks derives over it. Its value is built from the counters when read or + * published. + */ +export class Activity extends Source { + readonly #counts = { references: 0, calls: 0, jobs: 0, lingering: 0 }; + /** The counts as last read, until the next change. */ + #value: ActivityCounts | undefined; + #scheduled = false; + + get value(): JsonValue { + this.#value ??= { ...this.#counts }; + return this.#value; + } + + /** + * Adds `delta` to the count of `kind`; published with the turn's other changes. Nothing is scheduled while nobody + * watches or derives from it: a later watch or derive reads the counts as they are then. + */ + count(kind: keyof ActivityCounts, delta: number): void { + this.#counts[kind] += delta; + this.#value = undefined; + if (this.#scheduled || (!this.watched && this.dependents.size === 0)) return; + this.#scheduled = true; + queueMicrotask(() => { + this.#scheduled = false; + this.batch.within(() => this.changed(), BACKGROUND_CONTEXT); + }); + } + + deliver(context: Context): void { + const value = this.value; + this.publish(value, [["r", value]], false, context); + } +} diff --git a/packages/chord/src/next/host/attempt.ts b/packages/chord/src/next/host/attempt.ts new file mode 100644 index 000000000..aa9071dbd --- /dev/null +++ b/packages/chord/src/next/host/attempt.ts @@ -0,0 +1,620 @@ +// Attempts (§6, §13, §14, §15): one activation of an object, a fresh instance each time. It is the `self` its factory +// declares with (handles to the object's documents and jobs, under its seal token); it runs the activation steps, +// each failing with its own kind; it takes the live object's calls and starts its routines and jobs; and it ends with +// its seal (slice 6a §1.3), the drain and the closes of its natives. + +import { awaitWithContext, type Context, withAbortSignal } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { NativeWatchable, Watchable } from "../../watch.ts"; +import type { Contract, Implementation, ParamsOf } from "../contracts/index.ts"; +import { contractData, type ShapeOf } from "../contracts/internal.ts"; +import { ChordError, type FailureCause, type Principal } from "../data/index.ts"; +import { asChordError, checkArgs, checkParams, invalid, openFailure } from "../data/internal.ts"; +import type { HandleOf, JsonObject, StateDefinition, Tx } from "../documents/index.ts"; +import { + AddressSource, + type AnyDefinition, + type KeyDeclarer, + type KeyedSlot, + type KeySource, + SealToken, +} from "../documents/internal.ts"; +import type { JobDefinition, JobHandle, JobImplementation } from "../jobs/index.ts"; +import { type AnyJobImplementation, Job } from "../jobs/internal.ts"; +import { Derive, type DeriveSource } from "../state/internal.ts"; +import type { Columns } from "../store/columns.ts"; +import { tableSqlName } from "../store/sql.ts"; +import type { ChordStore } from "../store/store.ts"; +import type { TableDefinition } from "../store/table.ts"; +import { type Table, TableHandle } from "../store/table-handle.ts"; +import { prepareTables } from "../store/table-migration.ts"; +import { CallContext, type Caller, type Ctx, withLinkedAbortSignal } from "./ctx.ts"; +import { Deadline } from "./deadline.ts"; +import type { HostError } from "./host.ts"; +import { attachDerives, checkImplementation, type KeyedBinding, type Members } from "./members.ts"; +import { + type CloseFailed, + drained, + type Native, + type NativeCloseReason, + NativeHandle, + type NativeOptions, + type NativeSlot, +} from "./native.ts"; +import type { ChordObject } from "./object.ts"; +import type { Binding, CheckedParams, Reference, Target } from "./reference.ts"; +import { + type OnHandler, + type OnLiveHandler, + type OnNativeHandler, + type RoutineDeclarations, + Routines, +} from "./routines.ts"; + +/** + * What a factory declares with. Declarations (`state`, `derive`, `get`, `open`, `on`, `onLive`, `job`) are valid only + * while it runs. `P` is the app's principal type, which the ctx of routine calls and job runs carry. + */ +export interface Self

extends RoutineDeclarations

{ + /** This object's documents of `definition`, in memory for its life (across its attempts). */ + state>(definition: D): HandleOf; + /** A value computed from `sources` by the pure, synchronous `compute`. */ + derive[], T extends JsonValue>( + sources: S, + compute: (values: { readonly [I in keyof S]: S[I] extends Watchable ? V : never }) => T, + ): Watchable; + /** A reference to another object, live once this object is, released when it ends. */ + get(Type: C, params: ParamsOf): Reference; + /** + * A native, opened during activation after the references are live and the earlier natives are open, and closed + * in reverse when the attempt ends. `open` gets a context aborted when the attempt is abandoned; `close` one aborted + * at the close timeout. Throw `broken(error)` from `open` when retrying is pointless. + */ + open( + open: (context: Context) => Promise, + close: (value: T, reason: NativeCloseReason, context: Context) => void | Promise, + options?: NativeOptions, + ): Native; + /** Runs `fn` as one transaction of this object (§9). */ + tx(fn: (tx: Tx) => R | Promise, context: Context): Promise; + /** + * This object's job `definition` (§15), once per job name: `run` runs each started run outside transactions, + * `finish` records its outcome in one. Starts are taken once the object is live. + */ + job(definition: JobDefinition, implementation: JobImplementation): JobHandle; +} + +/** + * What the factory of a `local` type with empty params declares with: one object per host and type, so its data may + * persist (P3). `P` as for `Self`. + */ +export interface StoreSelf

extends Self

{ + /** This object's documents of `definition`: in memory for its life, or persisted in the host's store. */ + state>(definition: D): HandleOf; + /** This object's table `definition` in the host's store, once per table name. */ + table(definition: TableDefinition): Table; +} + +/** + * A factory: synchronous and declarative, run once per attempt. A remote one also gets the principal in its object's + * identity and returns one variant, `{ variant, members }`. Methods get a `Ctx` for the app's principal type `P`. + */ +export type Factory = ShapeOf["kind"] extends "local" + ? (params: ParamsOf, self: keyof ParamsOf extends never ? StoreSelf

: Self

) => Implementation> + : (params: ParamsOf, principal: P, self: Self

) => Implementation>; + +/** How an attempt's activation ended. `dependency_failed` (R18) keeps the attempt's references. */ +type Activation = + | { readonly kind: "live" } + | { readonly kind: "failed"; readonly cause: FailureCause } + | { readonly kind: "dependency_failed"; readonly cause: FailureCause } + | { readonly kind: "ended"; readonly error: ChordError }; + +/** The error of an object, or a getter, that waited `ms` without the object going live. */ +export function notLive(label: string, ms: number): ChordError { + return new ChordError({ code: "unavailable", message: `${label}: did not become live within ${ms} ms` }); +} + +/** + * One attempt of an object. Activation: factory, documents hydrated (the object's slots), references bound (before the + * previous attempt's are released) and live, natives opened one after another, members checked, derives computed; + * bounded by the host's `readyTimeout`. Ending: sealed at once (calls aborted, its transactions refused and the one + * holding the line discarded, its job invocations cut off), then the drain, the closes in reverse, derives detached; + * its object releases its references. Documents and job runs stay on the object. + */ +export class Attempt implements StoreSelf, KeyDeclarer { + readonly #object: ChordObject; + /** The object's context: the natives' closes run under it. */ + readonly #context: Context; + /** Aborted when the attempt is abandoned or ends: openers, activation's waits and calls run under its signal. */ + readonly #abort = new AbortController(); + /** The object's context under this attempt's signal. */ + readonly #opening: Context; + /** The seal token its handles and transactions carry (the seal fence). */ + readonly #token = new SealToken(); + /** The documents it declared, by name: "declared twice" is per attempt. */ + readonly #declared = new Map(); + /** Its factory's `at(key)` pins: the object holds them from its factory's return to the next one's (rule 6). */ + readonly #pins: (() => void)[] = []; + #declaring = false; + /** Set once it went live: its derives' failures are reported from then on (`derive_failed`). */ + #isLive = false; + /** The checked implementation and, for a remote object, its variant, once activation checked them. */ + #members: Members | undefined; + #variant: string | undefined; + readonly #derives: Derive[] = []; + readonly #routines = new Routines(); + readonly #jobs = new Map(); + /** The tables it declared, by name. */ + readonly #tables = new Map(); + /** References this attempt declared with `self.get` (params checked), bound during activation. */ + readonly #references: { readonly binding: Binding; readonly contract: Contract; readonly params: CheckedParams }[] = + []; + readonly #natives: NativeSlot[] = []; + /** + * Running calls' handlers, routine calls and job runs, each with its bookkeeping (call counts, `ctx.get` + * references): an ending attempt waits for them (the drain); the bookkeeping runs once, when its work settles or at + * teardown for work stuck past the drain (decision "The seal fence" (4)). One map and one reaction per piece of work: + * the call path costs no more than before the cleanups. + */ + readonly #running = new Map, (() => void) | undefined>(); + /** The previous attempt while it still holds its references (R18): released once this one has bound its own. */ + #previous: Attempt | undefined; + /** What activation returns once the attempt was abandoned: a timeout, a cycle through its object, or its end. */ + #abandoned: Exclude | undefined; + + constructor(object: ChordObject, previous: Attempt | undefined, context: Context) { + this.#object = object; + this.#previous = previous; + this.#context = context; + this.#opening = withAbortSignal(this.#abort.signal, context); + } + + state>(definition: D): HandleOf { + this.#assertDeclaring("state"); + const what = `state "${definition.name}"`; + if (definition.persist) this.#assertPersistable(what); + if (this.#declared.has(definition.name)) throw new Error(`${this.#object.label}: ${what} is declared twice`); + this.#declared.set(definition.name, definition); + return this.#object.documents.declare(definition, this.#token, this); + } + + /** `handle.at(key)`: pins `key` of `slot` while this attempt's factory's pins are held, and returns its source. */ + at(slot: KeyedSlot, key: string): KeySource { + this.#assertDeclaring("state(...).at"); + this.#pins.push(slot.pin(key, this.#opening)); + return slot.source(key); + } + + table(definition: TableDefinition): Table { + this.#assertDeclaring("table"); + const object = this.#object; + const { host, served, label } = object; + const store = this.#assertPersistable(`table "${definition.name}"`); + if (this.#tables.has(definition.name)) throw new Error(`${label}: table "${definition.name}" is declared twice`); + const sqlName = tableSqlName(host.address?.kind ?? "client", served.contract.name, definition.name); + const owner = { + label, + scope: object.documents, + assertLive: () => object.documents.assertLive(), + fail: (error: ChordError) => object.failLive(this, error), + }; + // The handle's rows are the definition's, typed by `C` and `K`. + const handle = new TableHandle(definition as unknown as TableDefinition, sqlName, store, owner); + this.#tables.set(definition.name, handle); + return handle as unknown as Table; + } + + derive[], T extends JsonValue>( + sources: S, + compute: (values: { readonly [I in keyof S]: S[I] extends Watchable ? V : never }) => T, + ): Watchable { + this.#assertDeclaring("derive"); + // The values are the sources' values, in order; the result is `T`. + const derive = new Derive(sources, compute as (values: readonly JsonValue[]) => T, this.#object.host.batch); + this.#derives.push(derive); + return derive as unknown as Watchable; + } + + get(Type: C, params: ParamsOf): Reference { + this.#assertDeclaring("get"); + const { host } = this.#object; + const checked = checkParams(Type, params, this.#opening); + const binding = host.reference(Type, this.#object, this.#caller(), {}); + this.#references.push({ binding, contract: Type, params: checked }); + // The binding was built from `Type`'s members. + return binding.reference as Reference; + } + + open( + open: (context: Context) => Promise, + close: (value: T, reason: NativeCloseReason, context: Context) => void | Promise, + options: NativeOptions = {}, + ): Native { + this.#assertDeclaring("open"); + const { host, label } = this.#object; + const closeTimeout = options.closeTimeout ?? host.timeouts.close; + const owner = { + object: label, + position: this.#natives.length + 1, + context: this.#context, + report: (error: HostError) => host.report(error), + }; + const native = new NativeHandle(open, close, closeTimeout, owner); + this.#natives.push(native); + return native; + } + + tx(fn: (tx: Tx) => R | Promise, context: Context): Promise { + return this.#object.documents.tx(fn, this.#token, context); + } + + on(source: Watchable, fn: OnHandler): void; + on(source: NativeWatchable, fn: OnNativeHandler): void; + on(source: Watchable | NativeWatchable, fn: (...args: never[]) => void | Promise): void { + this.#assertDeclaring("on"); + this.#routines.on(source, fn); + } + + onLive(fn: OnLiveHandler): void { + this.#assertDeclaring("onLive"); + this.#routines.onLive(fn); + } + + job(definition: JobDefinition, implementation: JobImplementation): JobHandle { + this.#assertDeclaring("job"); + const object = this.#object; + if (this.#jobs.has(definition.name)) { + throw new Error(`${object.label}: job "${definition.name}" is declared twice`); + } + const owner = { + host: object.host, + caller: this.#caller(), + context: this.#context, + signal: this.#abort.signal, + token: this.#token, + documents: object.documents, + admit: (running: Promise, cleanup?: () => void) => this.#admit(running, cleanup), + }; + // The job keeps its runs as JSON and calls the implementation with them; the handle types them again. + const records = object.jobRecords(definition as JobDefinition); + const job = new Job(records, implementation as unknown as AnyJobImplementation, owner); + this.#jobs.set(definition.name, job); + return job as unknown as JobHandle; + } + + /** + * Activates once `ending` (the previous instance's) has finished; `readyTimeout` bounds both (an attempt abandoned + * while it waits has opened nothing). + */ + async activate(ending: Promise | undefined): Promise { + const { ready } = this.#object.host.timeouts; + const deadline = new Deadline(ready, () => { + this.abandon({ kind: "failed", cause: { kind: "transient", error: notLive(this.#object.label, ready) } }); + }); + try { + if (ending !== undefined) { + try { + await awaitWithContext(ending, this.#opening); + } catch { + // An ending never rejects: only an abandonment (the deadline) ends this wait early. + return this.#abandoned!; + } + } + return await this.#activate(deadline); + } finally { + deadline.clear(); + } + } + + /** Abandons the activation with `outcome` (once): its waits stop and the openers' contexts are aborted. */ + abandon(outcome: Exclude): void { + if (this.#abandoned !== undefined) return; + this.#abandoned = outcome; + this.#abort.abort(outcome.kind === "ended" ? outcome.error : outcome.cause.error); + } + + /** The object went live: its jobs take starts and its routines start, in this turn; both stop when it is sealed. */ + live(): void { + this.#isLive = true; + for (const job of this.#jobs.values()) job.live(); + const { host, label } = this.#object; + this.#routines.start({ + label, + host, + caller: this.#caller(), + context: this.#opening, + admit: (running, cleanup) => this.#admit(running, cleanup), + }); + } + + /** The variant a remote object's factory chose; `undefined` for a local object. */ + get variant(): string | undefined { + return this.#variant; + } + + /** The state members, as Chord follows them. */ + get stateMembers(): ReadonlyMap { + return this.#members!.state; + } + + /** + * Calls a method for `caller`: a method outside the variant fails `invalid/member` first, then the arguments are + * checked against the current version (L1); the handler runs under a ctx aborted by the caller and by this attempt's + * end. Either rejects the caller at once; the handler runs on, its transactions refused, and the drain waits for it. + * Its `ctx.get` references are released when it settles. + */ + async call(method: string, args: unknown, context: Context, caller: Caller): Promise { + context.abortSignal?.throwIfAborted(); + const { host, served, label } = this.#object; + const handler = this.#members!.methods.get(method); + if (handler === undefined) throw invalid("member", `${label}${this.#inVariant()} has no method "${method}"`); + const checked = checkArgs(served.provider, method, args, served.contract.version, context); + const linked = withLinkedAbortSignal(this.#abort.signal, context); + const callContext = linked.context; + // A local object's handlers act as the host; a remote one's as their caller. + const principal = served.kind === "remote" ? caller.principal : host.principal; + const ctx = new CallContext(callContext, { principal, incarnation: caller.incarnation }, host); + // Admitted to the drain before the handler runs: the handler may end its own object before it returns. + const running = Promise.withResolvers(); + host.count("calls", 1); + this.#admit(running.promise, () => { + host.count("calls", -1); + linked.release(); + ctx.settle(); + }); + // Plugin code, not type-checked: a synchronous throw or a plain value becomes the promise. + (async () => handler(checked.args, ctx))().then(running.resolve, running.reject); + return awaitWithContext(running.promise, callContext); + } + + /** + * The keyed member `member`'s ultimate slot (slice 6a §12, rule 22: through re-exposures); an unknown one fails + * `invalid/member`. + */ + keyedMember(member: string): KeyedSlot { + const keyed = this.#members!.keyed.get(member); + if (keyed === undefined) throw invalid("member", `${this.#object.label} has no keyed member "${member}"`); + return keyed.kind === "own" ? keyed.slot : keyed.binding.keyedSlot(keyed.member); + } + + /** The keyed members as checked: what each is (`KeyedBinding`), for the keys its object holds across attempts. */ + get keyedMembers(): ReadonlyMap { + return this.#members!.keyed; + } + + /** + * Whether `source` (a state member) changes only through this object's own commits: one of this attempt's derives, + * or one of the object's documents. Another object's member source or document is foreign. + */ + owns(source: DeriveSource): boolean { + if (source instanceof Derive) return this.#derives.includes(source); + return source instanceof AddressSource && source.owner === this.#object.documents; + } + + /** The state member `member`, as Chord follows it; one outside the variant fails `invalid/member`. */ + member(member: string): DeriveSource { + const source = this.#members!.state.get(member); + if (source === undefined) { + throw invalid("member", `${this.#object.label}${this.#inVariant()} has no state member "${member}"`); + } + return source; + } + + /** + * Ends the attempt at once (the seal fence): calls, openers and routines aborted with `error` (their callers + * rejected with it), its transactions refused and the one holding the line discarded, its job invocations cut off; + * documents stay readable. + */ + seal(error: ChordError): void { + this.abandon({ kind: "ended", error }); + this.#object.documents.fence(this.#token); + for (const job of this.#jobs.values()) job.seal(error); + } + + /** + * Tears a sealed attempt down: waits for its running handlers, routine calls and runs up to the host's + * `closeTimeout` (the drain), then closes its natives in reverse with `reason`. Resolves with the closes that threw. + */ + async close(reason: NativeCloseReason): Promise { + const { host, label } = this.#object; + if ((await drained(this.#running.keys(), this.#running.size, host.timeouts.close)) === "timeout") { + host.report({ kind: "drain_timeout", object: label }); + } + // Work stuck past the drain keeps running, but no longer counts as a call or holds its `ctx.get` references. + for (const [running, cleanup] of this.#running) { + this.#running.delete(running); + cleanup?.(); + } + const failures: CloseFailed[] = []; + for (const native of [...this.#natives].reverse()) { + const failure = await native.close(reason); + if (failure !== undefined) failures.push(failure); + } + return failures; + } + + /** Detaches its derives (once their object no longer needs them): reads throw `released`, their watches end. */ + detach(): void { + const ended = this.#object.ended(); + for (const derive of this.#derives) derive.detach(ended); + } + + /** Releases its references, and the previous attempt's if this one never bound its own. */ + releaseReferences(): void { + this.#releasePrevious(); + for (const { binding } of this.#references) binding.release(); + } + + /** The activation steps; each catches its own failures with their kind (the table in slice 2's spec). */ + async #activate(deadline: Deadline): Promise { + let implementation: unknown; + let dependencies: Target[]; + try { + implementation = this.#declare(); + const { documents } = this.#object; + // The object holds the pins from now on. + documents.hydrate(this.#declared, this.#pins.splice(0), this.#opening); + const loading = documents.load(this.#opening); + if (loading !== undefined) await loading; + if (this.#tables.size > 0) await this.#prepareTables(deadline); + const { host } = this.#object; + dependencies = this.#references.map(({ binding, contract, params }) => + host.bind(binding, contract, params, this.#opening), + ); + for (const dependency of dependencies) this.#object.assertNoCycle(dependency, this.#opening); + } catch (error) { + return this.#broken(error); + } finally { + // Bound first: a dependency held only by the previous attempt never ends in between. + this.#releasePrevious(); + } + try { + await Promise.all(dependencies.map((dependency) => this.#object.waitFor(dependency, this.#opening))); + } catch (error) { + const abandoned = this.#abandoned; + // A timeout while waiting keeps the references (R18): a dependency that goes live later retries this object + // at once (a remote one whose link dropped and is held again). + if (abandoned?.kind === "failed" && abandoned.cause.kind === "transient") { + return { kind: "dependency_failed", cause: abandoned.cause }; + } + if (abandoned !== undefined) return abandoned; + // `waitFor` rejects with a ChordError naming the dependency, its error as the cause; or with a cycle's + // `invalid/cycle`, which fails this object broken too (proposal, slice 4b: an exception to R18, since every + // retry would meet the same cycle), keeping no references. + const failed = error as ChordError; + if (failed.code === "invalid" && failed.reason === "cycle") { + return { kind: "failed", cause: { kind: "broken", error: failed } }; + } + return { kind: "dependency_failed", cause: { kind: "transient", error: failed } }; + } + try { + // The keys the factory pinned on its dependencies' keyed members, loaded before this object goes live. + for (const { binding } of this.#references) { + const pinning = binding.pinKeys(this.#opening); + if (pinning !== undefined) await pinning; + } + } catch (error) { + // A pin load that failed its provider is a dependency failure (rule 6, R18). + const cause: FailureCause = { kind: "transient", error: asChordError(error, this.#object.label) }; + return this.#abandoned ?? { kind: "dependency_failed", cause }; + } + for (const native of this.#natives) { + // An attempt abandoned before this open (ended while it waited) closed its natives already: none opens late. + if (this.#abandoned !== undefined) return this.#abandoned; + try { + await awaitWithContext(native.open(this.#opening), this.#opening); + } catch (error) { + return this.#abandoned ?? { kind: "failed", cause: openFailure(error, this.#object.label) }; + } + } + try { + const checked = checkImplementation(this.#object.served.contract, implementation); + this.#members = checked.members; + this.#variant = checked.variant; + const { host, label } = this.#object; + attachDerives(this.#derives, this.#members.state, (member) => (error) => { + // During activation a throwing derive fails the attempt, which `activation_failed` reports alone. + if (!this.#isLive) return; + host.report({ kind: "derive_failed", object: label, ...(member === undefined ? {} : { member }), error }); + }); + for (const derive of this.#derives) void derive.value; + } catch (error) { + return this.#broken(error); + } + try { + // The keys references pin on the object's keyed members, loaded on this attempt's before it goes live. + const preparing = this.#object.prepareKeys(this.#members.keyed, this.#opening); + if (preparing !== undefined) await preparing; + } catch (error) { + const cause: FailureCause = { kind: "transient", error: asChordError(error, this.#object.label) }; + return this.#abandoned ?? { kind: "dependency_failed", cause }; + } + return this.#abandoned ?? { kind: "live" }; + } + + /** + * Brings the declared tables to their definitions before the references are bound (slice 6a §9): created fresh or + * migrated eagerly, with the `readyTimeout` deadline paused meanwhile (resumed with the budget it had left). + */ + async #prepareTables(deadline: Deadline): Promise { + const handles = [...this.#tables.values()]; + const tables = handles.map(({ definition, sql }) => ({ definition, table: sql.table })); + const fence = { admit: () => !this.#token.sealed && this.#abandoned === undefined, adopt: () => {} }; + deadline.pause(); + try { + await prepareTables(tables, handles[0]!.store, fence, this.#object.label, this.#opening); + } finally { + deadline.resume(); + } + } + + /** + * Runs the factory with declarations open; the member check establishes what it returned. A throwing factory + * releases its pins. + */ + #declare(): unknown { + this.#declaring = true; + try { + const { served, params, principal } = this.#object; + if (served.kind === "local") return served.factory(params, this); + // A remote object's identity has its principal. + return served.factory(params, principal!, this); + } catch (error) { + for (const release of this.#pins) release(); + throw error; + } finally { + this.#declaring = false; + } + } + + /** + * Persisted data needs a stable identity (P3: a `local` type with empty params, also checked by the types) and the + * host's store; returns the store. + */ + #assertPersistable(what: string): ChordStore { + const { host, served, label } = this.#object; + if (served.kind !== "local" || contractData(served.contract).params.fields.length > 0) { + throw new Error(`${label}: ${what}: persist needs a stable identity (a local type with empty params)`); + } + if (host.store === undefined) throw new Error(`${label}: ${what}: persist needs a store`); + return host.store; + } + + /** Adds `running` to the drain, with `cleanup` run once: when it settles, or at teardown if it never does. */ + #admit(running: Promise, cleanup?: () => void): void { + this.#running.set(running, cleanup); + // Out of the map already: the teardown ran the cleanup. + const settled = (): void => { + if (this.#running.delete(running)) cleanup?.(); + }; + running.then(settled, settled); + } + + /** Whom this object's own work acts for: a remote object as its principal, a local one as the host; from this host. */ + #caller(): Caller { + const { host, served, principal } = this.#object; + return { principal: served.kind === "remote" ? principal : host.principal, incarnation: host.incarnation }; + } + + /** ` (variant reader)` for a remote object, for errors. */ + #inVariant(): string { + return this.#variant === undefined ? "" : ` (variant ${this.#variant})`; + } + + #broken(error: unknown): Activation { + const cause: FailureCause = { kind: "broken", error: asChordError(error, this.#object.label) }; + return this.#abandoned ?? { kind: "failed", cause }; + } + + #releasePrevious(): void { + this.#previous?.releaseReferences(); + this.#previous = undefined; + } + + #assertDeclaring(declaration: string): void { + if (!this.#declaring) { + throw new Error(`${this.#object.label}: self.${declaration} is valid only in the factory`); + } + } +} diff --git a/packages/chord/src/next/host/backoff.ts b/packages/chord/src/next/host/backoff.ts new file mode 100644 index 000000000..bae0a91df --- /dev/null +++ b/packages/chord/src/next/host/backoff.ts @@ -0,0 +1,83 @@ +// Retry backoff per identity (§6): after a transient failure an identity waits 1 s, doubling up to 60 s. The entries +// hold no object, so a failed object is collected after its last reference while its window still runs. + +/** The delay after the first transient failure. */ +const FIRST_DELAY = 1_000; + +/** The longest delay. */ +const MAX_DELAY = 60_000; + +/** + * The delay after a failure: 1 s after the first (no `previous` delay), then the previous one doubled, up to 60 s. + * Identities, routines and job finishes back off alike. + */ +export function nextDelay(previous: number | undefined): number { + return previous === undefined ? FIRST_DELAY : Math.min(previous * 2, MAX_DELAY); +} + +/** + * One identity's backoff: the delay of its last window, when that window ends (`Date.now()` time), and the timer ending + * a window still running. + */ +type Entry = { readonly delay: number; readonly until: number; timer: ReturnType | undefined }; + +/** + * The host's backoff map. When a window ends, `due(identity)` runs: the host retries the identity's object, or + * deletes the entry when there is none. An entry is deleted on success, on a broken failure, and when its window has + * passed with nothing left to retry. + */ +export class Backoff { + readonly #entries = new Map(); + readonly #due: (identity: string) => void; + + constructor(due: (identity: string) => void) { + this.#due = due; + } + + /** Records a transient failure: a window of the previous delay doubled (1 s at first), then `due`. */ + failed(identity: string): void { + const previous = this.#entries.get(identity); + clearTimeout(previous?.timer); + const delay = nextDelay(previous?.delay); + const entry: Entry = { delay, until: Date.now() + delay, timer: undefined }; + entry.timer = setTimeout(() => { + entry.timer = undefined; + this.#due(identity); + }, delay); + this.#entries.set(identity, entry); + } + + /** Whether `identity` is inside a window: a new object for it waits for `due` before its first attempt. */ + waiting(identity: string): boolean { + return this.#entries.get(identity)?.timer !== undefined; + } + + /** When the window of `identity` ends, while one runs: a failed reference's `retryAt`. */ + until(identity: string): number | undefined { + const entry = this.#entries.get(identity); + return entry?.timer === undefined ? undefined : entry.until; + } + + /** Ends the window at once (its object is retried now, R18); the delay stays for the next failure. */ + skip(identity: string): void { + const entry = this.#entries.get(identity); + clearTimeout(entry?.timer); + if (entry !== undefined) entry.timer = undefined; + } + + /** Its object ended: an entry whose window has passed gets no outcome any more and goes. */ + ended(identity: string): void { + if (this.#entries.has(identity) && !this.waiting(identity)) this.#entries.delete(identity); + } + + /** Forgets `identity` (success, a broken failure, or a window that passed with nothing to retry). */ + delete(identity: string): void { + clearTimeout(this.#entries.get(identity)?.timer); + this.#entries.delete(identity); + } + + /** Forgets every identity (the host closes). */ + clear(): void { + for (const identity of [...this.#entries.keys()]) this.delete(identity); + } +} diff --git a/packages/chord/src/next/host/ctx.ts b/packages/chord/src/next/host/ctx.ts new file mode 100644 index 000000000..4eb44020b --- /dev/null +++ b/packages/chord/src/next/host/ctx.ts @@ -0,0 +1,132 @@ +// The `ctx` a method handler gets (§8): its call's context, who called, the calling host's incarnation, and +// call-scoped references; and the abort signal of a call's context, linked to long-lived sources until it settles. + +import { type Context, type ContextKey, withAbortSignal, withoutAbortSignal } from "../../context/index.ts"; +import type { Contract, ParamsOf } from "../contracts/index.ts"; +import { ChordError, type Principal } from "../data/index.ts"; +import type { AnyHost, GetOptions } from "./host.ts"; +import type { Binding, Reference } from "./reference.ts"; + +/** A handler's context. */ +export interface Ctx

extends Context { + /** The caller as its link stamped it; on a local object's handler the host's principal (none on a client). */ + readonly principal: P | undefined; + /** The calling host's incarnation over a link, this host's otherwise (R20). */ + readonly incarnation: string; + /** + * A reference as the caller (`principal`) sees it, released when the handler settles; afterwards `get` throws + * `released`. + */ + get(Type: C, params: ParamsOf, options?: GetOptions): Promise>; +} + +/** Who makes a call: the principal it acts as and its host's incarnation. */ +export type Caller = { readonly principal: Principal | undefined; readonly incarnation: string }; + +/** The ctx of one handler run: its references are released by `settle` when the handler settled. */ +export class CallContext implements Ctx { + readonly principal: Principal | undefined; + readonly incarnation: string; + readonly #context: Context; + readonly #host: AnyHost; + /** The references taken with `get`; `undefined` once the handler settled. */ + #references: Binding[] | undefined = []; + /** Aborted when the handler settles: a `get` still waiting gives up and releases its reference. By the first. */ + #settling: AbortController | undefined; + + constructor(context: Context, caller: Caller, host: AnyHost) { + this.#context = context; + this.principal = caller.principal; + this.incarnation = caller.incarnation; + this.#host = host; + } + + get abortSignal(): AbortSignal | undefined { + return this.#context.abortSignal; + } + + get telemetry(): Context["telemetry"] { + return this.#context.telemetry; + } + + value(key: ContextKey): T | undefined { + return this.#context.value(key); + } + + toString(): string { + return `${this.#context}.Ctx`; + } + + async get(Type: C, params: ParamsOf, options: GetOptions = {}): Promise> { + // An ended handler creates nothing: its call's abort signal is checked before anything else. + this.#context.abortSignal?.throwIfAborted(); + if (this.#references === undefined) throw settled(Type); + // Made for the caller: its principal and its origin's incarnation travel with the reference's holds. + const caller = { principal: this.principal, incarnation: this.incarnation }; + this.#settling ??= new AbortController(); + const scoped = withLinkedAbortSignal(this.#settling.signal, this.#context); + let binding: Binding; + try { + binding = await this.#host.acquire(Type, params, caller, scoped.context, options); + } catch (error) { + throw this.#references === undefined ? settled(Type) : error; + } finally { + scoped.release(); + } + // A get that finished in the turn its handler settled: nobody releases its reference later. + if (this.#references === undefined) { + binding.release(); + throw settled(Type); + } + this.#references.push(binding); + // The binding was built from `Type`'s members. + return binding.reference as Reference; + } + + /** The handler settled: its references are released, waiting gets give up, and `get` throws from now on. */ + settle(): void { + const references = this.#references ?? []; + this.#references = undefined; + this.#settling?.abort(); + for (const binding of references) binding.release(); + } +} + +function settled(Type: Contract): ChordError { + return new ChordError({ + code: "released", + reason: "settled", + message: `${Type.name}: ctx.get after its handler settled`, + }); +} + +/** + * A child of `context` aborted by `signal` and by the caller's signal (the caller's reason first) until `release()`, + * which the call runs when it settles. Not `AbortSignal.any`: before Node 24.16.0 it keeps cleanup entries per call for + * as long as its sources live (nodejs/node#62367), and a call's sources (its attempt, its reference) live long. + * Without a caller's signal the child is aborted by `signal` itself and `release` does nothing. + */ +export function withLinkedAbortSignal( + signal: AbortSignal, + context: Context, +): { readonly context: Context; release(): void } { + const caller = context.abortSignal; + if (caller === undefined) return { context: withAbortSignal(signal, context), release: NOTHING }; + const linked = new AbortController(); + const listeners = [caller, signal].map((source) => { + const forward = (): void => { + release(); + linked.abort(source.reason); + }; + return [source, forward] as const; + }); + const release = (): void => { + for (const [source, forward] of listeners) source.removeEventListener("abort", forward); + }; + const aborted = listeners.find(([source]) => source.aborted); + if (aborted !== undefined) linked.abort(aborted[0].reason); + else for (const [source, forward] of listeners) source.addEventListener("abort", forward, { once: true }); + return { context: withAbortSignal(linked.signal, withoutAbortSignal(context)), release }; +} + +const NOTHING = (): void => {}; diff --git a/packages/chord/src/next/host/deadline.ts b/packages/chord/src/next/host/deadline.ts new file mode 100644 index 000000000..4625b06b8 --- /dev/null +++ b/packages/chord/src/next/host/deadline.ts @@ -0,0 +1,43 @@ +// An activation's `readyTimeout` deadline (slice 6a §9): one timer that eager migration work pauses and resumes with +// the budget it had left, so a long migration does not fail its object while other activations keep their bounds. + +/** A deadline of `ms` calling `expired` once when reached; paused time does not count. */ +export class Deadline { + readonly #expired: () => void; + /** The budget left when the timer last started (or paused). */ + #remaining: number; + #started = 0; + #timer: ReturnType | undefined; + #cleared = false; + + constructor(ms: number, expired: () => void) { + this.#expired = expired; + this.#remaining = ms; + this.#start(); + } + + /** Stops the clock; `resume` restarts it with the budget left. */ + pause(): void { + if (this.#timer === undefined) return; + clearTimeout(this.#timer); + this.#timer = undefined; + this.#remaining = Math.max(0, this.#remaining - (Date.now() - this.#started)); + } + + /** Restarts a paused clock. */ + resume(): void { + if (this.#timer === undefined && !this.#cleared) this.#start(); + } + + /** Ends the deadline: `expired` is never called. */ + clear(): void { + clearTimeout(this.#timer); + this.#timer = undefined; + this.#cleared = true; + } + + #start(): void { + this.#started = Date.now(); + this.#timer = setTimeout(this.#expired, this.#remaining); + } +} diff --git a/packages/chord/src/next/host/facet.ts b/packages/chord/src/next/host/facet.ts new file mode 100644 index 000000000..0055285ff --- /dev/null +++ b/packages/chord/src/next/host/facet.ts @@ -0,0 +1,71 @@ +// Facets (§16): a plugin's setup for one host, typed by the app's principal type. A setup only declares: the types it +// serves and the host's presence lists; it has no documents, routines or natives, so a facet cannot fail after `load`. + +import type { Watchable } from "../../watch.ts"; +import type { Contract } from "../contracts/index.ts"; +import type { Principal } from "../data/index.ts"; +import type { Factory } from "./attempt.ts"; +import type { Presence } from "./presence.ts"; + +/** What a setup serves types with (valid only while the setup runs), and the host's presence lists (always). */ +export interface FacetHost

{ + /** Serves a type on this host with `factory`; a remote type's factory gets the principal its object is for. */ + serve(Type: C, factory: Factory, options?: ServeOptions): void; + /** `host.references(Type)`: the references to objects of `Type` on this host (§19), for factories to derive from. */ + references(Type: C): Watchable[]>; +} + +/** What `host.serve` takes besides the type and the factory. */ +export type ServeOptions = { + /** + * How long, in milliseconds, a live object with params stays live after its last reference is released (0 by + * default): a reference within that time reuses it, with its documents and dependencies. + */ + readonly linger?: number; +}; + +/** A type a setup served: its contract, its factory and its options. */ +export type ServedEntry = { + readonly contract: Contract; + readonly factory: ServedFactory; + readonly options: ServeOptions; +}; + +/** A factory as a setup served it, of either kind (the host dispatches on its contract's kind). */ +type ServedFactory = (params: never, principalOrSelf: never, self: never) => unknown; + +/** A plugin's facet for hosts of the app's principal type `P` (type only: nothing is checked at runtime). */ +export type Facet

= { + readonly setup: (host: FacetHost

) => void; + /** Type only: the principal type, invariant, so a host takes only facets typed for exactly its `P`. */ + readonly "~principal"?: (principal: P) => P; +}; + +/** Defines a facet; base code binds `P` once, next to its principal type. */ +export function defineFacet

(setup: (host: FacetHost

) => void): Facet

{ + return { setup }; +} + +/** Runs a facet's setup and returns the types it served, in order; `references` is the host's. */ +export function setUp

( + name: string, + facet: Facet

, + references: (Type: C) => Watchable[]>, +): readonly ServedEntry[] { + const served: ServedEntry[] = []; + let open = true; + const host: FacetHost

= { + serve: (Type, factory, options = {}) => { + if (!open) throw new Error(`${name}: host.serve is valid only in the facet's setup`); + // Served types are dispatched by name and kind; the factory matches its own contract. + served.push({ contract: Type, factory: factory as ServedFactory, options }); + }, + references, + }; + try { + facet.setup(host); + } finally { + open = false; + } + return served; +} diff --git a/packages/chord/src/next/host/host.ts b/packages/chord/src/next/host/host.ts new file mode 100644 index 000000000..8312e70bd --- /dev/null +++ b/packages/chord/src/next/host/host.ts @@ -0,0 +1,862 @@ +// Hosts (§3): `createHost`, loading facets and the start barrier, links to children, the parent (connected again after +// a drop) and routes, the origins it relays holds for, getting and holding references as a principal, the heartbeat +// of every link, revocation (§4), presence (§19) and keep-alive counts, closing; one batch per host, the backoff map +// and the endings new instances wait for. + +import type { TelemetryContext } from "@earendil-works/pi-telemetry"; +import { BACKGROUND_CONTEXT, type Context, withAbortSignal, withoutAbortSignal } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Watchable } from "../../watch.ts"; +import type { Address, Contract, HostKind, ParamsOf } from "../contracts/index.ts"; +import { contractData, type Params } from "../contracts/internal.ts"; +import { ChordError, type FailureCause, type Principal, type PrincipalKeyFunction } from "../data/index.ts"; +import { + asChordError, + bindProvider, + canonicalJson, + checkParams, + identityKey, + invalid, + principalKey, +} from "../data/internal.ts"; +import { binaryCodec, type Codec, type Compression, type Link, type PartLimits } from "../link/index.ts"; +import { linkLimits } from "../link/internal.ts"; +import { + type HoldRequest, + HostLink, + type LinkRole, + Path, + type ProviderSide, + RemoteObject, +} from "../remote/internal.ts"; +import { Batch, type BatchError } from "../state/internal.ts"; +import type { ChordStore, Store } from "../store/store.ts"; +import { Activity, type ActivityCounts } from "./activity.ts"; +import { notLive } from "./attempt.ts"; +import { Backoff } from "./backoff.ts"; +import { type Caller, withLinkedAbortSignal } from "./ctx.ts"; +import { type Facet, type ServedEntry, setUp } from "./facet.ts"; +import type { CloseFailed } from "./native.ts"; +import { ChordObject, type Holder, type LocalContract, type RemoteContract, type Served } from "./object.ts"; +import { type Placed, Placement } from "./placement.ts"; +import { type Presence, References } from "./presence.ts"; +import { Binding, type CheckedParams, type Held, type Reference, type Target } from "./reference.ts"; +import { RevocationKeys, type RevokedBy } from "./revocation.ts"; +import { Route, type RouteOptions } from "./route.ts"; +import { uuidv7 } from "./uuid.ts"; + +/** What `createHost` takes; `R` is the app's revocation type (JSON). */ +export type HostOptions

= { + /** This host's address (`Session({ id })`): it serves only objects placed there. Without one, a client. */ + readonly address?: Address; + /** Who this host acts as for its own requests; a host without one (a client) has none. */ + readonly principal?: P; + /** The key of a principal in identities; L1's canonical JSON by default. The same function on every host. */ + readonly principalKey?: PrincipalKeyFunction

; + /** The telemetry adapter; NOOP by default. */ + readonly telemetry?: TelemetryContext; + /** Everything that fails in the background; every app handles it. */ + readonly onError: (error: HostError) => void; + /** How long a native's close, and an ending object's running calls, are waited for; 30 s by default. */ + readonly closeTimeout?: number; + /** How long an activation attempt, and a `get`, may take until the object is live; 60 s by default. */ + readonly readyTimeout?: number; + /** + * The heartbeat of every link: a link silent for `ping` ms (15 s by default) sends a `ping`; one silent for `close` + * ms (30 s by default), its handshake included, is closed. + */ + readonly heartbeat?: { readonly ping?: number; readonly close?: number }; + /** + * What revokes a principal (§4): `host.revoke(r)` ends the objects and closes the child links of every principal + * `p` whose `revokedBy(p)` holds `r` (compared as canonical JSON). The same function on every host. + */ + readonly revokedBy?: RevokedBy; + /** + * A synchronous compressor (`nodeCompression` for Node): JSON parts above 16 KiB this host originates are deflated + * once, and deflated parts it owns inflated in place. Without one (a browser), nothing is compressed and deflated + * parts are inflated asynchronously, later messages of their link waiting in order. + */ + readonly compression?: Compression; + /** + * The store persisted documents and tables live in (`openStore`); the app owns it and closes it after the host. + * One open host per store. A `storage_failed` of the store is reported once and closes the host. + */ + readonly store?: Store; +}; + +/** Where a reported part sits on its object: a routine (`on`, `onLive`) or a native, by kind, from 1 in declaration order. */ +export type Position = { readonly kind: "on" | "onLive" | "native"; readonly position: number }; + +/** + * What `onError` gets, by kind. `object` is the object's type, with its params key when it has params; `failure` is a + * `FailureCause`, `error` the plain error. + */ +export type HostError = + | BatchError + | { readonly kind: "activation_failed"; readonly object: string; readonly failure: FailureCause } + /** A live object failed at runtime (a `migrate` failing on access, a stored version newer than the code). */ + | { readonly kind: "object_failed"; readonly object: string; readonly failure: FailureCause } + /** The host's store stopped after a backend error: the host closes itself (once). */ + | { readonly kind: "storage_failed"; readonly error: unknown } + | { readonly kind: "close_failed"; readonly object: string; readonly native: Position; readonly error: unknown } + | { readonly kind: "close_timeout"; readonly object: string; readonly native: Position } + | { readonly kind: "drain_timeout"; readonly object: string } + | { readonly kind: "protocol_error"; readonly error: unknown } + | { readonly kind: "resolve_failed"; readonly address: string; readonly error: unknown } + /** + * A derive's compute threw (its owner's, once per failure episode: the first failure after a value or after its + * creation; a value ends the episode); `member` when it is an exposed state member. + */ + | { readonly kind: "derive_failed"; readonly object: string; readonly member?: string; readonly error: unknown } + /** A routine's call threw, or its source failed for good; `routine` is its kind and position. */ + | { readonly kind: "routine_failed"; readonly object: string; readonly routine: Position; readonly error: unknown } + /** A job's `run` threw (not after an abort): its outcome is `failed { threw }`. */ + | { + readonly kind: "job_failed"; + readonly object: string; + readonly job: string; + readonly id: string; + readonly error: unknown; + } + /** A job's `finish` threw: retried with backoff, blocked after five failures in a row. */ + | { + readonly kind: "job_finish_failed"; + readonly object: string; + readonly job: string; + readonly id: string; + readonly error: unknown; + } + /** A blocked run was dropped without `finish`: aborted, or its object ended. */ + | { readonly kind: "job_dropped"; readonly object: string; readonly job: string; readonly id: string }; + +/** + * What `accept` and `connect` take: the principal of the host at the other end, as this host stamps it; `trusted` skips + * the message check on the link (IPC to the host's own processes; never grants the principal fields); the codec both + * ends of the link use (`binaryCodec` by default); and the limits per part of what arrives on it (Q2: none on a + * trusted link, 16 MiB on the wire and 64 MiB decoded otherwise, unless set). + */ +export type LinkOptions

= { + readonly principal: P; + readonly trusted?: boolean; + readonly codec?: Codec; + readonly limits?: PartLimits; +}; + +/** What `host.get` and `host.hold` take besides the type, the params and the context. */ +export type GetOptions = { + /** + * A getter's deadline for its whole wait, and how long a call made while the reference is loading waits (the park + * deadline); the host's `readyTimeout` by default. + */ + readonly readyTimeout?: number; +}; + +/** + * A host whatever its principal type, as Chord's own parts use it: `ChordHost

` is invariant in `P` (`load` takes + * facets typed for exactly `P`). + */ +export type AnyHost = Omit; + +/** A host. `P` is the app's principal type, `R` its revocation type. */ +export interface Host

{ + /** + * Runs every facet's setup, then activates every served type with empty params and resolves when each has had its + * first outcome, live or failed (the start barrier; failures go to `onError`). A setup that throws rejects it and + * closes the host. Runs once. + */ + load(facets: readonly { readonly name: string; readonly facet: Facet

}[], context: Context): Promise; + /** + * A reference once its object is live; waits for `load` first. It acts as the host's principal and goes where its + * type's placement puts it: here, over a route, or over the parent link (`Placement`). A broken failure rejects at + * once; through transient ones it waits until its deadline, then rejects with the last failure's error. + */ + get(Type: C, params: ParamsOf, context: Context, options?: GetOptions): Promise>; + /** + * A reference at once, following its object until released: loading, live, failed (retried as its object is) and + * loading again while its link is down; it never rejects. Throws at once for invalid params and a local type not + * served here; before `load` resolved these are decided once it did, failing the reference broken. Its `variant` is `undefined` until its first binding. + */ + hold(Type: C, params: ParamsOf, context: Context, options?: GetOptions): Held; + /** + * A link from a child: this host serves its remote types over it, every hold and call arriving on it acting as + * `principal`. `context` is the parent of the work arriving on it (its cancellation is dropped). + */ + accept(link: Link, options: LinkOptions

, context: Context): void; + /** + * This host's parent link: remote types not served here and not routed are held over it, every such reference held + * again on a new one; holds arriving on it may name whom they act for. Throws while the previous one is open; once + * it closed (a drop), `connect` takes the next. `principal` is who the parent is. + */ + connect(link: Link, options: LinkOptions

, context: Context): void; + /** + * How this host reaches hosts of `Kind` (called before `load`, once per kind): `authorize` admits every hold routed + * over it, `resolve` makes the link of an address. Holds for an address wait while it resolves and again after its + * link drops (a backoff per address, 1 s doubling to 60 s), while they exist. `context` is the parent of the route's + * work (its `resolve`s, the holds and calls on its links), aborted for it when this host closes. + */ + route( + Kind: HostKind, + options: RouteOptions>, + context: Context, + ): void; + /** + * Revokes `revocation` (§4, R10, R11), synchronously: every remote object whose principal it revokes (`revokedBy`) + * ends at once, whatever references remain (holders get `forbidden`, running calls are aborted, an activation is + * abandoned); every child link whose principal it revokes closes (never the parent, never this host's own + * principal); and every route link is sent `revoke`, which the host there applies with its own `revokedBy`. Nothing + * is replayed to links made later. + */ + revoke(revocation: R, context: Context): void; + /** + * The references to objects of `Type` on this host (§19), one entry per reference: the object's params, the + * principal it acts as (remote types) and its origin, the incarnation of the host it was made for (two tabs are two + * origins). Changes in the batch where a reference binds or ends; a link drop is a departure. + */ + references(Type: C): Watchable[]>; + /** + * The work keeping this host busy, for the app's supervisor: holds arriving over links and served here, method + * calls running, running jobs and lingering objects; published at most once per microtask turn with changes. + */ + readonly activity: Watchable; + /** + * Closes its links, then ends every object at once, then drains and closes them one by one, the last to go live + * first (reason `close`). Rejects with an `AggregateError` of the closes that threw. + */ + close(context: Context): Promise; +} + +/** Creates a host. */ +export function createHost

( + options: HostOptions, +): Host { + return new ChordHost(options); +} + +/** No activation chain (Q5). */ +const NO_CHAIN: readonly string[] = []; + +/** The params of a type with empty params, as checked: nothing, and the empty key. */ +const EMPTY_PARAMS: CheckedParams = { params: {}, key: "" }; + +/** An npm package name, scoped or not. */ +const PLUGIN_NAME = /^(?:@[a-z0-9~-][a-z0-9._~-]*\/)?[a-z0-9~-][a-z0-9._~-]*$/; + +/** The host: its served types, its objects by identity, its batch, backoff and endings. */ +export class ChordHost

implements Host { + readonly batch: Batch; + readonly timeouts: { readonly close: number; readonly ready: number }; + readonly backoff = new Backoff((identity) => this.#due(identity)); + readonly address: Address | undefined; + readonly principal: P | undefined; + /** This run of the host, sent in every handshake (R20). */ + readonly incarnation = uuidv7(); + readonly compression: Compression | undefined; + /** The host's store, if it has one. */ + readonly store: ChordStore | undefined; + readonly #activity: Activity; + readonly #principalKey: PrincipalKeyFunction

| undefined; + readonly #revocationKeys: RevocationKeys; + /** The presence list of each type someone asked about (`references`), by type name. */ + readonly #references = new Map(); + readonly #onError: (error: HostError) => void; + readonly #types = new Map(); + /** By identity, in the order they went live (the others where they were created). */ + readonly #objects = new Map(); + /** Teardowns still running, by identity: the next instance of an identity waits for its previous ones. */ + readonly #endings = new Map>(); + readonly #loaded = Promise.withResolvers(); + /** Every open link: children, routes and the parent. */ + readonly #links = new Set(); + /** Where holds go that are not served here and not routed: the parent link, across reconnects. */ + readonly #parent = new Path(() => {}); + /** + * Remote references failed for a part past a link's limits: on no link's path (no reconnect holds them, no route + * resolves for them) until `ref.retry()` places them anew; ended at `close` like the others. + */ + readonly unplaced = new Path(() => {}); + readonly #routes = new Map(); + readonly #placement: Placement; + /** The downstream origins this host relays holds for, by their number on upstream links (never reused). */ + readonly relays = new Map(); + #nextOrigin = 1; + /** Aborted when the host closes: the context of every route's `resolve`. */ + readonly #routing = new AbortController(); + readonly #heartbeat: { readonly ping: number; readonly close: number }; + /** The one heartbeat timer, running while the host has a link. */ + #ticking: ReturnType | undefined; + #state: "new" | "loading" | "loaded" | "closed" = "new"; + /** The one closing, which every `close` returns. */ + #closing: Promise | undefined; + + constructor(options: HostOptions

) { + this.address = options.address; + this.principal = options.principal; + this.#principalKey = options.principalKey; + this.#onError = options.onError; + this.batch = new Batch((error) => this.report(error), options.telemetry?.metrics); + this.compression = options.compression; + this.#activity = new Activity(this.batch); + // `revokedBy` is called with this host's principals only, the `P`s its links stamp. + this.#revocationKeys = new RevocationKeys(options.revokedBy as RevokedBy | undefined); + this.timeouts = { close: options.closeTimeout ?? 30_000, ready: options.readyTimeout ?? 60_000 }; + this.#heartbeat = { ping: options.heartbeat?.ping ?? 15_000, close: options.heartbeat?.close ?? 30_000 }; + this.#placement = new Placement(this.address, this.#types, this.#routes, this.#parent); + // Chord's own: `openStore` made it. + this.store = options.store as ChordStore | undefined; + this.store?.attach((error) => this.#storageFailed(error)); + } + + async load(facets: readonly { readonly name: string; readonly facet: Facet

}[], context: Context): Promise { + this.batch.assertOutside("load"); + this.#assertOpen(); + if (this.#state !== "new") throw new Error("load runs once per host"); + this.#state = "loading"; + try { + this.#setUp(facets); + } catch (error) { + // The host closes as `close` would: its links too. Nothing was activated, so no close can fail. + this.#closing ??= this.#close(); + await this.#closing.catch(() => {}); + throw error; + } + const objects = [...this.#types.values()] + .filter((served) => served.kind === "local" && served.hostLifetime) + .map((served) => this.#object(served, EMPTY_PARAMS, undefined, context)); + await Promise.allSettled(objects.map(({ settled }) => settled)); + if (this.#state === "loading") this.#state = "loaded"; + this.#loaded.resolve(); + } + + async get( + Type: C, + params: ParamsOf, + context: Context, + options: GetOptions = {}, + ): Promise> { + const binding = await this.acquire(Type, params, this.#own(), context, options); + // The binding was built from `Type`'s members. + return binding.reference as Reference; + } + + hold(Type: C, params: ParamsOf, context: Context, options: GetOptions = {}): Held { + this.batch.assertOutside("hold"); + // The binding was built from `Type`'s members; it may be unbound, so `variant` may be undefined. + return this.#hold(Type, params, this.#own(), context, options).reference as Held; + } + + accept(link: Link, options: LinkOptions

, context: Context): void { + this.#assertOpen(); + this.#link(link, "child", options, context); + } + + connect(link: Link, options: LinkOptions

, context: Context): void { + this.#assertOpen(); + if (this.#parent.link !== undefined) throw new Error("connect: this host has an open parent link"); + const parent = this.#link(link, "parent", options, context); + if (parent.open) this.#parent.connected(parent); + } + + route( + Kind: HostKind, + options: RouteOptions>, + context: Context, + ): void { + this.#assertOpen(); + if (this.#state !== "new") throw new Error("route: routes are declared before load"); + if (this.#routes.has(Kind.kind)) throw new Error(`route: ${Kind.kind} is routed already`); + // The parent of the route's work: its `resolve`s and the holds and calls on its links, aborted when it closes. + const routing = withAbortSignal(this.#routing.signal, context); + // A route calls `authorize` with the principals of holds on this host, `P`s, and addresses of `Kind`. + const route = new Route( + Kind as unknown as HostKind, + options as unknown as RouteOptions, + this, + routing, + ); + this.#routes.set(Kind.kind, route); + } + + revoke(revocation: R, context: Context): void { + const key = canonicalJson(revocation, context); + const revoked = (principal: Principal): boolean => + !this.#isOwn(principal, context) && this.#revocationKeys.revokes(principal, key, context); + const error = (label: string): ChordError => + new ChordError({ code: "forbidden", reason: "revoked", message: `${label}: revoked` }); + for (const object of [...this.#objects.values()]) { + if (object.principal !== undefined && revoked(object.principal)) object.end(error(object.label)); + } + // Holds admitted before this revoke but still waiting for `load` would take revoked objects afterwards. + const waiting = error("a hold waiting for load"); + for (const link of this.#links) link.revokeWaiting(revoked, waiting); + // Down the route links before the child links close: a worker ends the principal's objects with the revocation's + // error before the closed origins release their holds. + for (const link of this.#links) if (link.role === "route") link.send({ kind: "revoke", revocation }); + for (const link of [...this.#links]) if (link.role === "child" && revoked(link.principal)) link.close(); + } + + get activity(): Watchable { + // Its value is an `ActivityCounts`. + return this.#activity as unknown as Watchable; + } + + references(Type: C): Watchable[]> { + let references = this.#references.get(Type.name); + if (references === undefined) { + references = new References(this.batch); + for (const object of this.#objects.values()) { + if (object.served.contract.name !== Type.name) continue; + for (const holder of object.holders) references.record(object, holder); + } + this.#references.set(Type.name, references); + } + // Entries of `Type`'s objects, acting as `P`s. + return references as unknown as Watchable[]>; + } + + async close(_context: Context): Promise { + this.batch.assertOutside("close"); + this.#closing ??= this.#close(); + return this.#closing; + } + + /** + * What a hold arriving before `load` resolved waits for; `undefined` once it resolved (or the host closed), so a + * later hold takes its reference at once, in the turn it arrived (R18's handover: a new hold before the old's release). + */ + get pendingLoad(): Promise | undefined { + return this.#state === "new" || this.#state === "loading" ? this.#loaded.promise : undefined; + } + + /** Whether the host closes: its objects end through `close`, not through its links. */ + get closing(): boolean { + return this.#state === "closed"; + } + + /** + * A reference to `Type` made by `caller` (its principal, its origin's incarnation), once it is live (`host.get`, + * `ctx.get`): a hold, waited for up to the getter's one deadline, across reconnects; a broken failure rejects at + * once, the deadline with the last failure's error. A getter that gives up releases its reference. + */ + async acquire( + Type: Contract, + params: unknown, + caller: Caller, + context: Context, + options: GetOptions, + ): Promise { + this.batch.assertOutside("get"); + const binding = this.#hold(Type, params, caller, context, options); + const timeout = options.readyTimeout ?? this.timeouts.ready; + const expired = new AbortController(); + const waiting = withLinkedAbortSignal(expired.signal, context); + const deadline = setTimeout(() => expired.abort(binding.lastFailure ?? notLive(binding.label, timeout)), timeout); + try { + await binding.live(waiting.context); + return binding; + } catch (error) { + binding.release(); + throw error; + } finally { + clearTimeout(deadline); + waiting.release(); + } + } + + /** Adds `delta` to the keep-alive count of `kind` (`host.activity`), where the work happens. */ + count(kind: keyof ActivityCounts, delta: number): void { + this.#activity.count(kind, delta); + } + + /** `holder` references `object`: listed in its type's presence, if someone asked for it. */ + referenced(object: ChordObject, holder: Holder, context: Context): void { + this.#references.get(object.served.contract.name)?.add(object, holder, context); + } + + /** `holders` of `object` ended: gone from its type's presence. */ + unreferenced(object: ChordObject, holders: Iterable, context: Context): void { + this.#references.get(object.served.contract.name)?.remove(holders, context); + } + + /** Reports a background failure to `onError`; a throwing `onError` has nowhere left to report to. */ + report(error: HostError): void { + try { + this.#onError(error); + } catch { + // Nothing to do: teardowns and attempts go on. + } + } + + /** + * An unbound reference to an object of `Type` made by `caller`, declared by `dependent` (`undefined` for + * `host.get`, `host.hold` and `ctx.get`). Throws where it cannot go (`#assertRoute`). + */ + reference(Type: Contract, dependent: ChordObject | undefined, caller: Caller, options: GetOptions): Binding { + this.#assertOpen(); + this.#assertRoute(Type); + return this.#binding(Type, dependent, caller, options); + } + + /** + * Binds `binding` to the object for (Type, params) as its principal where its placement puts it (`Placement`): the + * object here, created with its first attempt if there is none, or a remote object held over its path. Throws as + * `Placement.reference` does. + */ + bind(binding: Binding, Type: Contract, params: CheckedParams, context: Context): Target { + const { principal } = binding.caller; + const placed = this.#placement.reference(Type, params.params, principal, context); + let target: Target; + if (placed.kind === "path") { + const { caller, dependent } = binding; + const remote = new RemoteObject(this, Type, params, placed.address, caller, placed.path, dependent); + placed.path.attach(remote); + target = remote; + } else { + // Placed here: its type is served here. + const served = this.#types.get(Type.name)!; + target = this.#object(served, params, principal, context, NO_CHAIN, binding.dependent); + } + binding.bind(target); + return target; + } + + /** Where a hold arriving on `link` goes (`Placement.hold`); `released` once the host closed. */ + place(request: HoldRequest, link: HostLink, caller: Caller, context: Context): Placed { + this.#assertOpen(); + return this.#placement.hold(request, link, caller.principal, context); + } + + /** + * The object a hold placed here binds to, acting as `principal`: a remote type served here (otherwise + * `unavailable`) with the hold's version and fingerprint (`contract_mismatch`), valid params, and placed here, its + * placement computed from the params (`invalid/placement`); not an activation the hold's chain waits for + * (`invalid/cycle`, Q5). A new object keeps the hold's chain. + */ + serving(request: HoldRequest, params: unknown, principal: Principal, context: Context): ChordObject { + const { type, version, fingerprint, chain = NO_CHAIN } = request; + const served = this.#types.get(type); + if (served?.kind !== "remote") { + throw new ChordError({ code: "unavailable", message: `${type} is not served here` }); + } + this.#assertMatch(served, version, fingerprint); + const checked = checkParams(served.contract, params, context); + this.#placement.assertHere(served.contract, checked.params); + const object = this.#object(served, checked, principal, context, chain); + const activation = object.activationId; + if (activation !== undefined && chain.includes(activation)) throw this.cycle(chain, activation, [object.label]); + return object; + } + + /** + * The path a remote reference of `Type` with `params` acting as `principal` goes over now: `ref.retry()` places a + * reference failed for a part past the limits anew (a route may have forgotten its old path). Throws as + * `Placement.reference` does, and `unavailable` for a type served here now. + */ + pathOf(Type: Contract, params: ParamsOf, principal: Principal | undefined): Path { + // A retry has no context of its own: its admission checks count on none. + const placed = this.#placement.reference(Type, params, principal, BACKGROUND_CONTEXT); + if (placed.kind === "path") return placed.path; + throw new ChordError({ code: "unavailable", message: `${Type.name} is served here now` }); + } + + /** + * The `invalid/cycle` error of a cycle through `chain` (Q5): its entries from the activation `id` on, by label + * where the object is on this host ("…" for other hosts'), then `closing`. + */ + cycle(chain: readonly string[], id: string, closing: readonly string[]): ChordError { + const labels = new Map([...this.#objects.values()].map((each) => [each.activationId, each.label])); + const cycle = chain.slice(chain.indexOf(id)).map((entry) => labels.get(entry) ?? "…"); + return invalid("cycle", `dependency cycle: ${[...cycle, ...closing].join(" → ")}`); + } + + /** A new origin number for `side`, which relays holds upstream; registered in `relays` until it closes. */ + relayOrigin(side: ProviderSide): number { + const origin = this.#nextOrigin++; + this.relays.set(origin, side); + return origin; + } + + /** + * A route's link, its work under the route's `context`: registered with the host and its heartbeat; one closed + * already is returned closed. + */ + routeLink(link: Link, options: LinkOptions, context: Context): HostLink { + return this.#link(link, "route", options, context); + } + + /** A link completed its handshake: a route's resets its address's backoff. */ + linkOpened(link: HostLink): void { + if (link.role !== "route") return; + for (const route of this.#routes.values()) route.opened(link); + } + + /** + * Forgets a link once it closed: the origins relaying to it, then the parent's path or its route's address are + * told; the heartbeat may stop. + */ + unlink(link: HostLink): void { + this.#links.delete(link); + for (const relay of this.relays.values()) relay.unlinked(link); + if (this.#links.size === 0) { + clearInterval(this.#ticking); + this.#ticking = undefined; + } + if (link.role === "parent") this.#parent.disconnected(link); + if (link.role === "route") for (const route of this.#routes.values()) if (route.closed(link)) break; + } + + /** Moves `object` behind every object that went live before it (close ends in reverse); its backoff is over. */ + wentLive(object: ChordObject): void { + this.#objects.delete(object.identity); + this.#objects.set(object.identity, object); + this.backoff.delete(object.identity); + } + + /** Drops an ended object (once, before a next instance of its identity exists). */ + forget(object: ChordObject): void { + this.#objects.delete(object.identity); + this.backoff.ended(object.identity); + } + + /** Records a teardown for `identity`: its next attempt waits for it and for every earlier one still running. */ + ending(identity: string, teardown: Promise): void { + const previous = this.#endings.get(identity); + const ending = previous === undefined ? teardown : Promise.all([previous, teardown]).then(() => {}); + this.#endings.set(identity, ending); + void ending.then(() => { + if (this.#endings.get(identity) === ending) this.#endings.delete(identity); + }); + } + + /** What the next attempt for `identity` waits for: its previous instances' teardowns, if any still run. */ + endingOf(identity: string): Promise | undefined { + return this.#endings.get(identity); + } + + /** + * A reference to `Type` made by `caller` for `host.hold` and getters: params and where it goes checked at once + * (`invalid/params`, `#assertRoute`), then bound (a placement or route that fails fails the reference broken); + * before `load` resolved, both once it did (a host closed meanwhile releases it). + */ + #hold(Type: Contract, params: unknown, caller: Caller, context: Context, options: GetOptions): Binding { + this.#assertOpen(); + const checked = checkParams(Type, params, context); + if (this.pendingLoad === undefined) { + const binding = this.reference(Type, undefined, caller, options); + this.#bindOrFail(binding, Type, checked, context); + return binding; + } + const binding = this.#binding(Type, undefined, caller, options); + // Raced with the reference's end: a release before `load` resolved lets go of the binding, its contract and + // `context` at once (only the race's settled promise stays behind in `load`'s reactions). + void Promise.race([this.#loaded.promise, binding.ended()]).then(() => { + // Released while `load` ran: nothing is made for it. + if (binding.refState.kind === "ended") return; + if (this.#state === "closed") return binding.end(this.#closedError()); + this.#bindOrFail(binding, Type, checked, context); + }); + return binding; + } + + /** + * Binds a reference (`bind`); where it cannot go (`#assertRoute`, a placement or `authorize` that throws or refuses) + * fails it broken. + */ + #bindOrFail(binding: Binding, Type: Contract, checked: CheckedParams, context: Context): void { + try { + this.#assertRoute(Type); + this.bind(binding, Type, checked, context); + } catch (error) { + binding.refuse({ kind: "broken", error: asChordError(error, Type.name) }); + } + } + + /** A new reference's binding made by `caller`; its parked calls wait its `readyTimeout`. */ + #binding(Type: Contract, dependent: ChordObject | undefined, caller: Caller, options: GetOptions): Binding { + return new Binding(Type, this.batch, dependent, caller, options.readyTimeout ?? this.timeouts.ready); + } + + /** This host as the maker of its own references: its principal and incarnation. */ + #own(): Caller { + return { principal: this.principal, incarnation: this.incarnation }; + } + + /** Whether `principal` is this host's own (by principal key): revocation never touches it. */ + #isOwn(principal: Principal, context: Context): boolean { + const own = this.principal; + if (own === undefined) return false; + // Principals on this host are `P`s: its own, and the ones its links stamp. + return ( + principalKey(own, context, this.#principalKey) === principalKey(principal as P, context, this.#principalKey) + ); + } + + /** A new link of `role`, registered with the heartbeat while open. */ + #link(link: Link, role: LinkRole, options: LinkOptions, context: Context): HostLink { + const { principal, trusted = false, codec = binaryCodec } = options; + const limits = linkLimits(trusted, options.limits); + const made = new HostLink(this, link, { role, principal, trusted, codec, limits }, withoutAbortSignal(context)); + if (!made.open) return made; + this.#links.add(made); + if (this.#ticking === undefined) { + const { ping, close } = this.#heartbeat; + this.#ticking = setInterval(() => { + const now = Date.now(); + for (const each of [...this.#links]) each.tick(now, ping, close); + }, Math.min(ping, close) / 10); + } + return made; + } + + /** + * Throws where a reference to `Type` cannot go: a type served here needs the caller's version and fingerprint + * (`contract_mismatch`); a local type never crosses a link (`unavailable`). A remote type not served here goes on (a + * route or the parent link), open yet or not. + */ + #assertRoute(Type: Contract): void { + const served = this.#types.get(Type.name); + if (served === undefined) { + if (Type.kind === "local") { + throw new ChordError({ code: "unavailable", message: `${Type.name} is not served here` }); + } + return; + } + this.#assertMatch(served, Type.version, Type.fingerprint); + } + + /** + * The object for `params` (checked by L1) and, for a remote type, `principal` (its key is part of the identity), + * created if there is none; its first attempt starts at once, or when its identity's backoff window ends. A new one + * keeps the activation chain it is created for (Q5): `chain` (a hold's), or that of `dependent`'s activation when + * an activation declared it, so its own holds over links carry that chain on. + */ + #object( + served: Served, + params: CheckedParams, + principal: Principal | undefined, + context: Context, + chain: readonly string[] = NO_CHAIN, + dependent?: ChordObject, + ): ChordObject { + const { key, params: checked } = params; + const identity = + served.kind === "local" + ? identityKey(served.contract, key) + : // A host serving remote types has a principal, which its references act as unless a + // link stamped its own (only `P`s). + identityKey(served.contract, key, principalKey(principal as P, context, this.#principalKey)); + let object = this.#objects.get(identity); + if (object === undefined) { + // Shared by every getter: one getter's cancellation does not cancel it. + const owner = served.kind === "local" ? undefined : principal; + const created = dependent?.activationChain() ?? chain; + object = new ChordObject(this, served, checked, owner, key, identity, withoutAbortSignal(context), created); + this.#objects.set(identity, object); + if (!this.backoff.waiting(identity)) object.attempt(); + } + return object; + } + + /** + * Closes the links, then ends every object at once and tears them down one by one, the last to go live first, after + * the endings already running. + */ + async #close(): Promise { + this.#state = "closed"; + this.#routing.abort(this.#closedError()); + // Links first, children, routes, then the parent: no `gone` goes out for the objects ending below. + const links = [...this.#links]; + for (const role of ["child", "route", "parent"]) for (const link of links) if (link.role === role) link.close(); + // Remote references end now: their parked calls reject `released`, no later link holds them. + const closed = this.#closedError(); + const holders = [ + ...this.#parent.holders(), + ...this.unplaced.holders(), + ...[...this.#routes.values()].flatMap((route) => route.close()), + ]; + for (const holder of holders) if (holder.kind === "remote") holder.lost(closed); + this.backoff.clear(); + // Every object is sealed before any teardown releases a reference, so none ends in the background. + const teardowns = [...this.#objects.values()].reverse().map((object) => object.stop()); + await Promise.all(this.#endings.values()); + const failures: CloseFailed[] = []; + for (const tearDown of teardowns) failures.push(...(await tearDown())); + this.#types.clear(); + this.store?.detach(); + // Gets waiting for a load that will not come fail now. + this.#loaded.resolve(); + if (failures.length > 0) { + const message = `host.close: ${failures.length} native close(s) threw`; + throw new AggregateError( + failures.map(({ error }) => error), + message, + ); + } + } + + /** The store stopped after a backend error (it tells the host once): reported, and the host closes. */ + #storageFailed(error: unknown): void { + this.report({ kind: "storage_failed", error }); + this.#closing ??= this.#close(); + // The app's own `close` returns this closing and gets its failures. + this.#closing.catch(() => {}); + } + + /** A backoff window ended: the identity's object is retried; without one the entry goes. */ + #due(identity: string): void { + const object = this.#objects.get(identity); + if (object === undefined) this.backoff.delete(identity); + else object.attempt(); + } + + /** Runs every facet's setup, checking the plugin names (npm names, each once), and serves what they declared. */ + #setUp(facets: readonly { readonly name: string; readonly facet: Facet

}[]): void { + const names = new Set(); + for (const { name, facet } of facets) { + if (!PLUGIN_NAME.test(name)) throw new Error(`"${name}" is not a valid plugin name`); + if (names.has(name)) throw new Error(`plugin "${name}" is loaded twice`); + names.add(name); + for (const entry of setUp(name, facet, (Type) => this.references(Type))) this.#serve(entry); + } + } + + #serve({ contract, factory, options }: ServedEntry): void { + if (this.#types.has(contract.name)) throw new Error(`${contract.name} is served twice`); + // A client serves only local types (§3): a remote object is owned by a principal of the host serving it. + if (contract.kind === "remote" && this.principal === undefined) { + throw new Error(`${contract.name}: a host without a principal serves only local types`); + } + const common = { provider: bindProvider(contract, {}), linger: options.linger ?? 0 }; + // The contract's kind decides the factory's signature and the contract's shape. + const served: Served = + contract.kind === "local" + ? { + ...common, + kind: "local", + contract: contract as LocalContract, + factory: factory as Extract["factory"], + hostLifetime: contractData(contract).params.fields.length === 0, + } + : { + ...common, + kind: "remote", + contract: contract as RemoteContract, + factory: factory as Extract["factory"], + }; + this.#types.set(contract.name, served); + } + + #assertMatch(served: Served, version: number, fingerprint: string): void { + if (version !== served.contract.version || fingerprint !== served.contract.fingerprint) { + const message = `${served.contract.name} v${version} does not match the served v${served.contract.version}`; + throw new ChordError({ code: "contract_mismatch", message }); + } + } + + #assertOpen(): void { + if (this.#state === "closed") throw this.#closedError(); + } + + #closedError(): ChordError { + return new ChordError({ code: "released", reason: "closed", message: "the host is closed" }); + } +} diff --git a/packages/chord/src/next/host/index.ts b/packages/chord/src/next/host/index.ts new file mode 100644 index 000000000..6eb1a040c --- /dev/null +++ b/packages/chord/src/next/host/index.ts @@ -0,0 +1,24 @@ +// Hosts for apps: creating a host, facets, factories, natives, references and the ctx of method handlers, routines and +// their waiting helpers, revocation, presence and keep-alive counts. + +export type { ActivityCounts } from "./activity.ts"; +export type { Factory, Self, StoreSelf } from "./attempt.ts"; +export type { Ctx } from "./ctx.ts"; +export { defineFacet, type Facet, type FacetHost, type ServeOptions } from "./facet.ts"; +export { + createHost, + type GetOptions, + type Host, + type HostError, + type HostOptions, + type LinkOptions, + type Position, +} from "./host.ts"; +export type { Native, NativeCloseReason, NativeOptions } from "./native.ts"; +export type { Presence } from "./presence.ts"; +export type { RefState } from "./ref-state.ts"; +export type { Held, Reference, RefIn } from "./reference.ts"; +export type { RevokedBy } from "./revocation.ts"; +export type { RouteOptions } from "./route.ts"; +export type { OnHandler, OnLiveHandler, OnNativeHandler, RoutineDeclarations } from "./routines.ts"; +export { sleep, untilAborted } from "./sleep.ts"; diff --git a/packages/chord/src/next/host/internal.ts b/packages/chord/src/next/host/internal.ts new file mode 100644 index 000000000..a976a73a5 --- /dev/null +++ b/packages/chord/src/next/host/internal.ts @@ -0,0 +1,11 @@ +// Hosts for Chord's layers: what the remote sides of links use of a host, its objects and references; what jobs use of +// a handler's ctx, the backoff and ids. + +export { nextDelay } from "./backoff.ts"; +export { CallContext, type Caller } from "./ctx.ts"; +export type { AnyHost, ChordHost } from "./host.ts"; +export type { PinnedKey } from "./keyed-members.ts"; +export { type ChordObject, type Holder, settlement } from "./object.ts"; +export { LIVE, LOADING, type RefState } from "./ref-state.ts"; +export type { Binding, CheckedParams, Target } from "./reference.ts"; +export { uuidv7 } from "./uuid.ts"; diff --git a/packages/chord/src/next/host/keyed-members.ts b/packages/chord/src/next/host/keyed-members.ts new file mode 100644 index 000000000..e10166507 --- /dev/null +++ b/packages/chord/src/next/host/keyed-members.ts @@ -0,0 +1,268 @@ +// Keyed members across attempts (slice 6a §1.2, §1.4, §12): the keys of a local object's keyed members that references +// pin (`ref.member.at(key)`, `ref.member.watch(key)`), one member source per member and key, as state members have. +// While an attempt is live a key follows that attempt's keyed member (the object's own slot, or the provider's member +// it re-exposes, rule 22: one tracker at the end); while the object is failed it keeps its last value and gets no +// frames; when a new attempt goes live it follows that attempt's member of the same name, its key loaded before. + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import type { ChordError } from "../data/index.ts"; +import type { KeyedSlot } from "../documents/internal.ts"; +import type { Batch, DeriveSource } from "../state/internal.ts"; +import { MemberSource } from "./member-source.ts"; +import type { KeyedBinding } from "./members.ts"; + +/** A key of a keyed member, pinned: its source, and the release (idempotent). */ +export type PinnedKey = { readonly source: DeriveSource; readonly release: () => void }; + +/** A pinned key on one attempt's keyed member; `foreign` when the member is another object's (re-exposed). */ +type Following = { readonly pin: PinnedKey; readonly foreign: boolean }; + +/** + * Where one pinned key stands: its first pin in flight (`loading`: `abandon` rejects its users' waits), following an + * attempt's member, frozen while its object is failed or activating again (both with what a new attempt prepared to + * follow, `next`), dropped (its last user left, or its object failed or ended while it loaded), or ended with its + * object (`error`, for a pin whose wait resolved just before). + */ +type EntryState = + | { readonly kind: "loading"; readonly abandon: (error: unknown) => void } + | { + readonly kind: "bound"; + readonly source: MemberSource; + readonly following: Following; + readonly next: Following | undefined; + } + | { readonly kind: "frozen"; readonly source: MemberSource; readonly next: Following | undefined } + | { readonly kind: "dropped" } + | { readonly kind: "ended"; readonly error: ChordError }; + +/** One pinned key of one keyed member: its users, the wait of its first pin, and where it stands. */ +type Entry = { + readonly member: string; + readonly key: string; + users: number; + readonly ready: Promise; + state: EntryState; +}; + +/** The pinned keys of one object's keyed members, across its attempts; every pin below them released exactly once. */ +export class KeyedMembers { + readonly #batch: Batch; + /** The object's context: a key's first load runs under it, shared by every user. */ + readonly #context: Context; + readonly #entries = new Map>(); + + constructor(batch: Batch, context: Context) { + this.#batch = batch; + this.#context = context; + } + + /** Whether no key is pinned. */ + get empty(): boolean { + return this.#entries.size === 0; + } + + /** + * Pins `key` (a stored key) of `member`, which the live attempt has as `keyed`: its member source once the key is + * loaded; `context` ends only this caller's wait. Rejects when the object fails or ends while the first pin loads. + */ + async pin(member: string, key: string, keyed: KeyedBinding, context: Context): Promise { + const entry = this.#entries.get(member)?.get(key) ?? this.#create(member, key, keyed); + entry.users++; + try { + await awaitWithContext(entry.ready, context); + } catch (error) { + this.#release(entry); + throw error; + } + const { state } = entry; + // Loaded, but its object ended before this wait resumed: nothing to pin (the entry is gone). + if (state.kind === "ended") throw state.error; + // Loaded, and its object neither failed nor ended since (a failure while loading rejects `ready` first). + const source = (state as Extract).source; + let released = false; + const release = (): void => { + if (released) return; + released = true; + this.#release(entry); + }; + return { source, release }; + } + + /** + * Loads every pinned key on `members` (a new attempt's, while it activates, before it goes live); `undefined` when + * no key is pinned. Rejects when a load fails; `rebind` or `discard` then takes what it pinned. + */ + prepare(members: ReadonlyMap, context: Context): Promise | undefined { + if (this.#entries.size === 0) return undefined; + return (async () => { + for (const byKey of this.#entries.values()) { + for (const entry of byKey.values()) { + const keyed = members.get(entry.member); + if (keyed === undefined || (entry.state.kind !== "bound" && entry.state.kind !== "frozen")) continue; + const next = { pin: await pinOn(keyed, entry.key, context), foreign: keyed.kind === "reexposed" }; + const state = entry.state; + // Dropped while it loaded: the new pin goes at once. + if (state.kind !== "bound" && state.kind !== "frozen") { + next.pin.release(); + continue; + } + state.next?.pin.release(); + entry.state = { ...state, next }; + } + } + })(); + } + + /** The new attempt went live, inside a batch: every prepared key follows its new member (one frame). */ + rebind(): void { + for (const byKey of this.#entries.values()) { + for (const entry of byKey.values()) { + const state = entry.state; + if ((state.kind !== "bound" && state.kind !== "frozen") || state.next === undefined) continue; + state.source.bind(state.next.pin.source); + if (state.kind === "bound") state.following.pin.release(); + entry.state = { kind: "bound", source: state.source, following: state.next, next: undefined }; + } + } + } + + /** The new attempt did not go live: what `prepare` pinned is released. */ + discard(): void { + for (const byKey of this.#entries.values()) { + for (const entry of byKey.values()) { + const state = entry.state; + if ((state.kind !== "bound" && state.kind !== "frozen") || state.next === undefined) continue; + state.next.pin.release(); + entry.state = { ...state, next: undefined }; + } + } + } + + /** + * The object failed live (`error`): the keys keep their values and release what they follow; `foreign` takes only + * the keys of re-exposed members (no admitted commit of this object changes them), `all` every key. A first pin + * still loading is given up at once (its users' waits reject with `error`; the pin, when it lands, is released). + */ + freeze(which: "foreign" | "all", error: unknown): void { + for (const byKey of this.#entries.values()) { + for (const entry of [...byKey.values()]) { + const state = entry.state; + if (state.kind === "loading") { + this.#abandon(entry, state, error); + continue; + } + if (state.kind !== "bound" || (which === "foreign" && !state.following.foreign)) continue; + state.source.freeze(); + state.following.pin.release(); + entry.state = { kind: "frozen", source: state.source, next: state.next }; + } + } + } + + /** The object ended, inside a batch: every key ends with `error` and releases what it pins. */ + end(error: ChordError): void { + for (const byKey of this.#entries.values()) { + for (const entry of byKey.values()) { + const state = entry.state; + if (state.kind === "loading") state.abandon(error); + else if (state.kind === "bound" || state.kind === "frozen") state.source.objectEnded(error); + this.#releaseAll(entry); + entry.state = { kind: "ended", error }; + } + } + this.#entries.clear(); + } + + /** A new entry, its first pin started under the object's context. */ + #create(member: string, key: string, keyed: KeyedBinding): Entry { + const ready = Promise.withResolvers(); + // Each user awaits `ready`; a rejection nobody waits for is not unhandled. + ready.promise.catch(() => {}); + const entry: Entry = { + member, + key, + users: 0, + ready: ready.promise, + state: { kind: "loading", abandon: ready.reject }, + }; + pinOn(keyed, key, this.#context).then( + (pin) => { + // Given up meanwhile (its object failed or ended, or every user left): the late pin goes, once. + if (entry.state.kind !== "loading") { + pin.release(); + return; + } + const source = new MemberSource(pin.source, this.#batch); + entry.state = { + kind: "bound", + source, + following: { pin, foreign: keyed.kind === "reexposed" }, + next: undefined, + }; + ready.resolve(); + }, + (error) => { + if (entry.state.kind === "loading") this.#abandon(entry, entry.state, error); + }, + ); + let byKey = this.#entries.get(member); + if (byKey === undefined) { + byKey = new Map(); + this.#entries.set(member, byKey); + } + byKey.set(key, entry); + return entry; + } + + /** Gives up `entry`'s first pin with `error`: dropped and forgotten, its users' waits rejected. */ + #abandon(entry: Entry, state: Extract, error: unknown): void { + entry.state = { kind: "dropped" }; + this.#forget(entry); + state.abandon(error); + } + + /** One user of `entry` left; the last drops it. */ + #release(entry: Entry): void { + if (--entry.users > 0) return; + this.#forget(entry); + this.#releaseAll(entry); + } + + /** Drops `entry`: what it follows and will follow is released; its source keeps its value and follows nothing. */ + #releaseAll(entry: Entry): void { + const state = entry.state; + entry.state = { kind: "dropped" }; + if (state.kind !== "bound" && state.kind !== "frozen") return; + state.source.freeze(); + if (state.kind === "bound") state.following.pin.release(); + state.next?.pin.release(); + } + + #forget(entry: Entry): void { + const byKey = this.#entries.get(entry.member); + if (byKey?.get(entry.key) !== entry) return; + byKey.delete(entry.key); + if (byKey.size === 0) this.#entries.delete(entry.member); + } +} + +/** + * Pins `key` on `keyed`: on the object's own slot it is pinned and loaded; on a re-exposed member it is pinned on the + * provider's member (whose own keys follow its attempts). + */ +function pinOn(keyed: KeyedBinding, key: string, context: Context): Promise { + if (keyed.kind === "reexposed") return keyed.binding.pinKey(keyed.member, key, context); + return pinned(keyed.slot, key, context); +} + +/** Pins `key` on `slot` and loads it when it is not in memory; a failed load releases the pin. */ +async function pinned(slot: KeyedSlot, key: string, context: Context): Promise { + const release = slot.pin(key, context); + try { + await slot.ensure(key, context); + } catch (error) { + release(); + throw error; + } + return { source: slot.source(key), release }; +} diff --git a/packages/chord/src/next/host/member-source.ts b/packages/chord/src/next/host/member-source.ts new file mode 100644 index 000000000..c5f8e6b68 --- /dev/null +++ b/packages/chord/src/next/host/member-source.ts @@ -0,0 +1,221 @@ +// Member sources (slice 6a §1.2, §1.4): a local object's state member as its references and other objects' derives +// follow it across attempts. While an attempt is live it passes that attempt's member on; while the object is failed +// or activating again it keeps the last value and gets no frames; when a new attempt goes live it re-binds to that +// attempt's member of the same name (one frame from the last value), as a remote reference's re-hold does. + +import type { Context } from "../../context/index.ts"; +import { diffRevisions, type Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { Watch, type WatchCall, type WatchEnd } from "../../watch.ts"; +import type { ChordError } from "../data/index.ts"; +import { + type Batch, + type Dependent, + type DeriveSource, + handedOut, + Source, + type Subscriber, + type WatchStart, +} from "../state/internal.ts"; + +/** + * Where a member source stands: following the live attempt's member, frozen at its last value, failed (its value + * threw when frozen, or a new attempt's variant lacks the member: a later attempt may bring it back; `last` is the + * last good value, `undefined` when there was none) or ended with its object. + */ +type MemberState = + | { readonly kind: "bound"; readonly source: DeriveSource } + | { readonly kind: "frozen"; readonly value: JsonValue } + | { readonly kind: "failed"; readonly error: unknown; readonly last: JsonValue | undefined } + | { readonly kind: "ended"; readonly error: ChordError }; + +const ENDED: WatchEnd = { kind: "ended" }; + +/** + * One state member of a local object across its attempts. Bound, it is a dependent of the attempt's member (its own + * dependents are marked with it). Its watches and subscribers (holds over a link, member sources re-exposing it) + * subscribe to the bound member itself, so the member's frames reach them as they are, with no extra hop, and a derive + * member stays lazy while nothing follows it; a freeze or a re-bind moves them. + */ +export class MemberSource extends Source implements Dependent { + #state: MemberState; + /** Its watches and subscribers, each subscribed to the bound member while bound; frozen, they get nothing. */ + readonly #passed = new Set(); + /** What its watches had at a re-bind (`undefined` for no value: the member had failed), until the re-bind's frame. */ + #framed: { readonly value: JsonValue | undefined } | undefined; + + constructor(source: DeriveSource, batch: Batch) { + super(batch); + this.#state = { kind: "bound", source }; + this.#hook(source); + } + + get value(): JsonValue { + const state = this.#state; + if (state.kind === "bound") return state.source.value; + if (state.kind === "frozen") return state.value; + throw state.error; + } + + /** What a new watch starts at: the bound member's start, the frozen value, or the failure with its last value. */ + start(): WatchStart { + const state = this.#state; + if (state.kind === "bound") { + return state.source instanceof Source ? state.source.start() : { kind: "value", value: state.source.value }; + } + if (state.kind === "frozen") return { kind: "value", value: state.value }; + if (state.kind === "failed") return { kind: "failed", error: state.error, last: state.last }; + throw state.error; + } + + /** + * A watch subscribed to the bound member itself (as `Source.openWatch` starts one: on a failed member at its last + * good value, the failure first). + */ + openWatch(context: Context): Watch { + this.batch.assertOutside("watch"); + const start = this.start(); + if (start.kind === "failed" && start.last === undefined) throw start.error; + const value = start.kind === "value" ? start.value : start.last!; + const watch: Watch = new Watch(value, () => this.unsubscribe(watch), this.batch.reportListener); + this.subscribe(watch); + if (context.abortSignal !== undefined) watch.observeAbort(context.abortSignal); + if (start.kind === "failed") watch.fail(handedOut(start.error), context); + return watch; + } + + /** Whether a watch or a subscriber follows it (through the bound member). */ + get watched(): boolean { + return this.#passed.size > 0; + } + + /** Feeds `subscriber` the bound member's frames (it subscribes there) and the re-binds' frames. */ + subscribe(subscriber: Subscriber): void { + this.#passed.add(subscriber); + const state = this.#state; + if (state.kind === "bound" && state.source instanceof Source) state.source.subscribe(subscriber); + } + + /** As `Source.unsubscribe`. */ + unsubscribe(subscriber: Subscriber): void { + this.#passed.delete(subscriber); + const state = this.#state; + if (state.kind === "bound" && state.source instanceof Source) state.source.unsubscribe(subscriber); + } + + /** The bound member changed: its dependents are marked (its frames reach the subscribers directly). */ + invalidate(): void { + for (const dependent of this.dependents) dependent.invalidate(); + } + + /** The bound member's document was retired: its dependents carry the flag on. */ + sourceRetired(visited: Set): void { + for (const dependent of this.dependents) dependent.sourceRetired(visited); + } + + /** Its object failed live and its admitted commits adopted: it keeps the member's value (or error) from now on. */ + freeze(): void { + const state = this.#state; + if (state.kind !== "bound") return; + this.#leave(); + const start = state.source instanceof Source ? state.source.start() : undefined; + if (start === undefined || start.kind === "value") { + this.#state = { kind: "frozen", value: start === undefined ? state.source.value : start.value }; + } else { + this.#state = { kind: "failed", error: start.error, last: start.last }; + } + } + + /** + * A new attempt went live: follows its member `source`, inside a batch. Watches get one frame from the last value + * (a full one after a failure); derives downstream are marked and recompute, a failed one included (self-healing). + */ + bind(source: DeriveSource): void { + if (this.#state.kind === "ended") return; + this.#leave(); + if (this.watched) this.#framed = { value: this.#shown() }; + this.#state = { kind: "bound", source }; + this.#hook(source); + // Marks the derives downstream: a failed one recomputes too (self-healing). + this.changed(); + } + + /** + * A new attempt's variant lacks the member, inside a batch: reads throw `error`; watches and subscribers get one + * failure call and stay open (a later attempt may bring the member back, as `[["r", value]]`). + */ + vanish(error: ChordError, context: Context): void { + if (this.#state.kind === "ended") return; + const last = this.#lastGood(); + this.#leave(); + this.#state = { kind: "failed", error, last }; + this.#call({ kind: "failed", error: handedOut(error) }, context); + this.changed(); + } + + /** + * Its object ended, inside a batch: reads throw `error`, watches end `ended`, and derives over it are marked (their + * next read throws it, no cached value stays). Its other subscribers end with their owners (a hold with its object). + */ + objectEnded(error: ChordError): void { + this.#leave(); + this.#state = { kind: "ended", error }; + this.#framed = undefined; + // A copy: an ending watch unsubscribes itself. + for (const subscriber of [...this.#passed]) subscriber.end(ENDED); + this.#passed.clear(); + for (const dependent of this.dependents) dependent.invalidate(); + } + + /** Delivers a re-bind's frame: the diff from the watches' value to the new member's. */ + deliver(context: Context): void { + const framed = this.#framed; + this.#framed = undefined; + if (framed === undefined) return; + let after: JsonValue; + try { + after = this.value; + } catch (error) { + if (framed.value !== undefined) this.#call({ kind: "failed", error: handedOut(error) }, context); + return; + } + const before = framed.value; + if (after === before) return; + const ops: readonly Op[] = before === undefined ? [["r", after]] : diffRevisions(before, after); + if (ops.length > 0) this.#call({ kind: "value", value: after, ops, retired: false }, context); + } + + /** A call of its own (a re-bind's frame, a vanished member) to its watches and subscribers. */ + #call(call: WatchCall, context: Context): void { + for (const subscriber of this.#passed) subscriber.advance(call, context); + } + + /** The value its watches have before a re-bind; `undefined` when they had a failure (their next value is whole). */ + #shown(): JsonValue | undefined { + const start = this.start(); + return start.kind === "value" ? start.value : undefined; + } + + /** Its last good value, which watches opened while it is failed start at; `undefined` when it never had one. */ + #lastGood(): JsonValue | undefined { + const start = this.start(); + return start.kind === "value" ? start.value : start.last; + } + + /** Starts reading `source`: its changes mark this source's dependents, its frames reach the subscribers. */ + #hook(source: DeriveSource): void { + if (!(source instanceof Source)) return; + source.dependents.add(this); + for (const subscriber of this.#passed) source.subscribe(subscriber); + } + + /** Stops following the bound source, when bound: no more marks or frames from it. */ + #leave(): void { + const state = this.#state; + if (state.kind !== "bound") return; + const { source } = state; + if (!(source instanceof Source)) return; + source.dependents.delete(this); + for (const subscriber of this.#passed) source.unsubscribe(subscriber); + } +} diff --git a/packages/chord/src/next/host/members.ts b/packages/chord/src/next/host/members.ts new file mode 100644 index 000000000..7eaf1fc73 --- /dev/null +++ b/packages/chord/src/next/host/members.ts @@ -0,0 +1,124 @@ +// The member check (Q14): an implementation is one variant with exactly that variant's members, methods being +// functions and state members JSON watchables; and the resolution of every derive's declared sources. +// +// The rule for watches across attempts (slice 6a §1.2, §1.4): references and other objects' derives follow a local +// object's state member through its member source (`member-source.ts`), and a key of a keyed member they hold (`at`, +// `watch(key)`) through a member source per key (`keyed-members.ts`); both live as long as the object. While the +// object is failed or activating again they keep their last values and get no frames; when a new attempt goes live +// they re-bind by name to that attempt's member (a pinned key loaded on it first), whether a document or a derive: one +// frame from the last value, and every derive downstream recovers, one that failed on the old member's error included +// (decision "self healing"). A re-exposed member is the re-exposing object's: it freezes with that object, its +// provider's source goes on. At a live failure, members following another object's sources freeze at once; the +// object's own documents and derives freeze once its admitted commits adopted, so until then an own derive over +// another object's member may still change. Holds over a link follow the live attempt's members and take a snapshot at +// each binding. + +import type { Context } from "../../context/index.ts"; +import type { Contract } from "../contracts/index.ts"; +import { currentVariants, LOCAL, takesNoArgs, type VariantData } from "../contracts/internal.ts"; +import type { KeyedSlot } from "../documents/internal.ts"; +import type { Derive, DeriveSource } from "../state/internal.ts"; +import { type Binding, resolveKeyed, resolveWatchable } from "./reference.ts"; + +/** A checked implementation, as Chord calls and follows it. */ +export type Members = { + /** The methods, each called with its arguments (`null` for none) and a context. */ + readonly methods: ReadonlyMap; + /** The state members, resolved to what Chord follows. */ + readonly state: ReadonlyMap; + /** The keyed members (slice 6a §12): the object's own slot, or the provider's member it re-exposes. */ + readonly keyed: ReadonlyMap; +}; + +/** + * What a keyed member is: a keyed document handle's slot (`own`), or another object's keyed member re-exposed through + * a reference (`reexposed`), which reads end at the ultimate provider's slot (rule 22) and pinned keys follow. + */ +export type KeyedBinding = + | { readonly kind: "own"; readonly slot: KeyedSlot } + | { readonly kind: "reexposed"; readonly binding: Binding; readonly member: string }; + +/** A method as Chord calls it. */ +type Method = (args: unknown, context: Context) => Promise; + +/** + * Checks what a factory returned (plugin code): a local contract's members, or a remote contract's `{ variant, + * members }` naming one of its current variants. Errors name the variant or member (the failure cause adds the + * object). + */ +export function checkImplementation( + contract: Contract, + implementation: unknown, +): { readonly variant: string | undefined; readonly members: Members } { + const variants = currentVariants(contract); + if (contract.kind === "local") { + // The member check establishes the shape. + return { variant: undefined, members: checkMembers(variants.get(LOCAL)!, implementation as Implementation) }; + } + // Plugin code: a remote factory returns `{ variant, members }`, which the checks below establish. + const { variant, members } = implementation as { readonly variant: string; readonly members: Implementation }; + const data = variants.get(variant); + if (data === undefined) throw new Error(`variant ${JSON.stringify(variant)}: not a variant of ${contract.name}`); + return { variant, members: checkMembers(data, members) }; +} + +type Implementation = { readonly [member: string]: unknown }; + +/** + * Checks members against a variant's (exactly its members, methods being functions, state members JSON watchables and + * keyed members keyed document handles); a method without arguments is called with the context alone. + */ +function checkMembers({ state, methods, keyed }: VariantData, implementation: Implementation): Members { + for (const member of Object.keys(implementation)) { + if (!state.has(member) && !methods.has(member) && !keyed.has(member)) throw new Error(`${member}: not a member`); + } + // Own properties only: an inherited `toString` is no implementation of a member named so. + const own = (member: string): unknown => + Object.hasOwn(implementation, member) ? implementation[member] : undefined; + const checked = { + methods: new Map(), + state: new Map(), + keyed: new Map(), + }; + for (const [member, data] of methods) { + const method = own(member); + if (typeof method !== "function") throw new Error(`${member}: a method is a function`); + const call: Method = takesNoArgs(data) + ? (_args, context) => method(context) + : (args, context) => method(args, context); + checked.methods.set(member, call); + } + for (const member of state.keys()) { + const source = resolveWatchable(own(member)); + if (source === undefined) throw new Error(`${member}: a state member is a JSON watchable`); + checked.state.set(member, source); + } + for (const member of keyed.keys()) { + const resolved = resolveKeyed(own(member)); + if (resolved === undefined) throw new Error(`${member}: a keyed member is a keyed document handle`); + checked.keyed.set(member, resolved); + } + return checked; +} + +/** + * Resolves every derive's sources; errors name a derive by the member exposing it, else by its position. `report` + * makes each derive's failure report, with the member exposing it when there is one (`derive_failed`). + */ +export function attachDerives( + derives: readonly Derive[], + state: ReadonlyMap, + report: (member: string | undefined) => (error: unknown) => void, +): void { + const exposed = new Map([...state].map(([member, source]) => [source, member])); + derives.forEach((derive, position) => { + const member = exposed.get(derive); + const name = member ?? `derive #${position + 1}`; + const resolve = (source: unknown, index: number): DeriveSource => { + const resolved = resolveWatchable(source); + if (resolved === undefined) throw new Error(`${name}: source ${index} is not a JSON watchable`); + return resolved; + }; + derive.attach(resolve, report(member)); + }); +} diff --git a/packages/chord/src/next/host/native.ts b/packages/chord/src/next/host/native.ts new file mode 100644 index 000000000..c0dc190f4 --- /dev/null +++ b/packages/chord/src/next/host/native.ts @@ -0,0 +1,151 @@ +// Natives (§13): the handle `self.open` returns, a value opened during activation and closed with a reason when its +// attempt ends; `within`, which bounds a close or a drain by a timeout; and the drain's admission of running work. + +import { type Context, withAbortSignal } from "../../context/index.ts"; +import { ChordError } from "../data/index.ts"; +import type { HostError } from "./host.ts"; + +/** Why a native closes: its object ended or its attempt failed (`end`), the host closes (`close`), a reload (slice 7). */ +export type NativeCloseReason = { readonly kind: "end" } | { readonly kind: "close" } | { readonly kind: "reload" }; + +/** A native's handle. */ +export interface Native { + /** The opened value: throws `not_ready` until its open resolved and `released` once its close started. */ + readonly value: T; +} + +/** What `self.open` takes besides the open and the close. */ +export type NativeOptions = { + /** How long its close may take; the host's `closeTimeout` by default. */ + readonly closeTimeout?: number; +}; + +/** A close that threw, as reported. */ +export type CloseFailed = Extract; + +/** + * Where a native belongs: its object's label and its position in declaration order (from 1) for errors and reports, + * the object's context (every close runs under it, never under the aborted opener's), and where it reports. + */ +type NativeOwner = { + readonly object: string; + readonly position: number; + readonly context: Context; + readonly report: (error: HostError) => void; +}; + +/** What an attempt does with each of its natives, whatever their value's type. */ +export interface NativeSlot { + open(context: Context): Promise; + close(reason: NativeCloseReason): Promise; +} + +/** Where a native stands. `abandoned`: closed while its open was pending, so the late value is closed at once. */ +type NativeState = + | { readonly kind: "unopened" } + | { readonly kind: "opening" } + | { readonly kind: "open"; readonly value: T } + | { readonly kind: "abandoned"; readonly reason: NativeCloseReason } + | { readonly kind: "closed" }; + +/** One native of one attempt: opened at most once, closed exactly once. */ +export class NativeHandle implements Native, NativeSlot { + readonly #open: (context: Context) => Promise; + readonly #close: (value: T, reason: NativeCloseReason, context: Context) => void | Promise; + readonly #closeTimeout: number; + readonly #owner: NativeOwner; + #state: NativeState = { kind: "unopened" }; + + constructor( + open: (context: Context) => Promise, + close: (value: T, reason: NativeCloseReason, context: Context) => void | Promise, + closeTimeout: number, + owner: NativeOwner, + ) { + this.#open = open; + this.#close = close; + this.#closeTimeout = closeTimeout; + this.#owner = owner; + } + + get value(): T { + const state = this.#state; + if (state.kind === "open") return state.value; + const name = this.#name(); + if (state.kind === "unopened" || state.kind === "opening") { + throw new ChordError({ code: "not_ready", message: `${name}: not open yet` }); + } + throw new ChordError({ code: "released", reason: "closed", message: `${name}: closed` }); + } + + /** Opens the value; one that arrives after its attempt was abandoned is closed at once, a failure reported. */ + async open(context: Context): Promise { + this.#state = { kind: "opening" }; + await this.#opened(await this.#open(context)); + } + + /** Closes the value if it is open (a pending open's value is closed when it arrives); resolves with a throw. */ + async close(reason: NativeCloseReason): Promise { + const state = this.#state; + this.#state = state.kind === "opening" ? { kind: "abandoned", reason } : { kind: "closed" }; + return state.kind === "open" ? this.#closeValue(state.value, reason) : undefined; + } + + /** Keeps the value an open resolved with, or closes it at once when the native was closed meanwhile. */ + async #opened(value: T): Promise { + const state = this.#state; + if (state.kind !== "abandoned") { + this.#state = { kind: "open", value }; + return; + } + this.#state = { kind: "closed" }; + const failure = await this.#closeValue(value, state.reason); + if (failure !== undefined) this.#owner.report(failure); + } + + /** + * Runs the close under the object's context, aborted at the close timeout, and waits for it up to then; longer is + * reported. + */ + async #closeValue(value: T, reason: NativeCloseReason): Promise { + const { object, position, context, report } = this.#owner; + const at = { object, native: { kind: "native", position } } as const; + const timeout = new AbortController(); + const closing = (async () => this.#close(value, reason, withAbortSignal(timeout.signal, context)))(); + try { + if ((await within(closing, this.#closeTimeout)) === "timeout") { + timeout.abort(new ChordError({ code: "aborted", message: `${this.#name()}: close timed out` })); + report({ kind: "close_timeout", ...at }); + } + return undefined; + } catch (error) { + return { kind: "close_failed", ...at, error }; + } + } + + #name(): string { + return `${this.#owner.object} native #${this.#owner.position}`; + } +} + +/** Whether `promise` settled within `ms` (it rejects when `promise` does first); `promise` itself runs on. */ +export async function within(promise: Promise, ms: number): Promise<"settled" | "timeout"> { + let timer: ReturnType | undefined; + const timeout = new Promise<"timeout">((resolve) => { + timer = setTimeout(resolve, ms, "timeout"); + }); + try { + return await Promise.race([promise.then(() => "settled" as const), timeout]); + } finally { + clearTimeout(timer); + } +} + +/** Whether the `size` pieces of work in `drain` settled within `ms` (at once when there is none). */ +export async function drained( + drain: Iterable>, + size: number, + ms: number, +): Promise<"settled" | "timeout"> { + return size === 0 ? "settled" : within(Promise.allSettled(drain), ms); +} diff --git a/packages/chord/src/next/host/object.ts b/packages/chord/src/next/host/object.ts new file mode 100644 index 000000000..ebdf37b4a --- /dev/null +++ b/packages/chord/src/next/host/object.ts @@ -0,0 +1,691 @@ +// Objects (§6): one per identity, from its first attempt to its end. Its state across attempts (its documents, line +// and job runs, its member sources, slice 6a §1.2), its holders, retries after failures (backoff, dependency recovery +// R18, `ref.retry()`), a live object's failure (slice 6a §1.4), the waits and cycle checks of activations (Q5, its +// instance id and the activation chain it was created for), lingering after its last reference (§3, D8), the job runs +// that keep it alive (§15), and its ending, revocation included (§4, R11). + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import type { Contract, ParamsOf } from "../contracts/index.ts"; +import type { ContractShape } from "../contracts/internal.ts"; +import { ChordError, type FailureCause, isChordError, type Principal } from "../data/index.ts"; +import { invalid, type Provider } from "../data/internal.ts"; +import { Documents, type KeyedSlot } from "../documents/internal.ts"; +import type { JobDefinition } from "../jobs/index.ts"; +import { JobRecords } from "../jobs/internal.ts"; +import { CHAIN } from "../link/internal.ts"; +import type { DeriveSource } from "../state/internal.ts"; +import { Attempt, type Self } from "./attempt.ts"; +import type { Caller } from "./ctx.ts"; +import type { AnyHost } from "./host.ts"; +import { KeyedMembers, type PinnedKey } from "./keyed-members.ts"; +import { MemberSource } from "./member-source.ts"; +import type { KeyedBinding } from "./members.ts"; +import type { CloseFailed, NativeCloseReason } from "./native.ts"; +import { type Ended, type Failed, LIVE, LOADING, type RefState } from "./ref-state.ts"; +import type { Target } from "./reference.ts"; +import { uuidv7 } from "./uuid.ts"; + +/** A `local` contract, as a served type has it. */ +export type LocalContract = Contract; + +/** A `remote` contract, as a served type has it. */ +export type RemoteContract = Contract; + +/** + * A type served on a host, by kind: its contract, factory (a remote one gets the object's principal) and argument + * checks. Factories are plugin code: what they return is checked at activation. + */ +export type Served = + | (ServedType & { + readonly kind: "local"; + readonly factory: (params: ParamsOf, self: Self) => unknown; + /** A type with empty params: its object lives as long as the host (Q-A). */ + readonly hostLifetime: boolean; + }) + | (ServedType & { + readonly kind: "remote"; + readonly factory: (params: ParamsOf, principal: Principal, self: Self) => unknown; + }); + +type ServedType = { + readonly contract: C; + readonly provider: Provider; + /** How long a live object stays live after its last reference, in milliseconds (`serve`'s `linger`, 0 by default). */ + readonly linger: number; +}; + +/** What holds an object: a reference's binding, or a hold arriving over a link. */ +export interface Holder { + /** The object whose attempt declared the reference; retried when this object goes live (R18). */ + readonly dependent: ChordObject | undefined; + /** Whom the reference acts for, with its origin's incarnation (`host.references`). */ + readonly caller: Caller; + /** The object's state (`state`) changed: an attempt started after a failure, it went live, or it failed. */ + moved(): void; + /** The object ended with `error`. */ + end(error: ChordError): void; +} + +/** + * Where an object stands. `waiting`: for its identity's backoff window before its first attempt. `failed` has the + * state its references show, and keeps the failed attempt in `holding` after a dependency failure (R18): its + * references keep the dependencies. + */ +type ObjectState = + | { readonly kind: "waiting" } + | { readonly kind: "activating"; readonly attempt: Attempt } + | { readonly kind: "live"; readonly attempt: Attempt } + | { readonly kind: "failed"; readonly failure: Failed; readonly holding: Attempt | undefined } + | { readonly kind: "ended"; readonly ended: Ended }; + +/** + * What an ended object still has to do once its previous teardowns finished (`previous`): drain and close its attempt, + * or release a failed attempt's references; then drop its documents and job runs (`end` alone). + */ +type Teardown = + | { readonly kind: "none" } + | { readonly kind: "end"; readonly previous: Promise | undefined } + | { readonly kind: "close" | "release"; readonly attempt: Attempt; readonly previous: Promise | undefined }; + +const END: NativeCloseReason = { kind: "end" }; + +const CLOSE: NativeCloseReason = { kind: "close" }; + +/** + * One object. Each attempt is a fresh `Attempt`; the documents, the line and the job runs, the holders (and, after a + * dependency failure, the failed attempt's references) outlive it. A transient failure is retried after its backoff window, or at once when a + * dependency it failed on goes live or a reference asks (`retry`); a broken one stays failed until a reference asks. + * The object ends with its last reference, a live one after its type's `linger` (never, for a host-lifetime object), + * or when the host closes. + */ +export class ChordObject implements Target { + readonly host: AnyHost; + readonly served: Served; + readonly params: ParamsOf; + /** The principal in a remote object's identity, which its factory gets; `undefined` for a local object. */ + readonly principal: Principal | undefined; + readonly identity: string; + /** The object in errors, reports and cycles: its type, and its params key when it has params. */ + readonly label: string; + /** + * The activation chain it was created for (Q5): the instance ids of the activations waiting for it, from the hold + * that created it (a trusted link's) or the activation here that declared it; empty otherwise. + */ + readonly chain: readonly string[]; + /** Its documents and line, across its attempts. */ + readonly documents: Documents; + /** The context of every attempt: its creator's, without the creator's cancellation. */ + readonly #context: Context; + /** Its job runs by job name, across its attempts. */ + readonly #jobs = new Map(); + /** This instance's id (uuidv7), minted at its first attempt: a restarted object is a new instance (D4). */ + #instance: string | undefined; + #state: ObjectState = { kind: "waiting" }; + /** References and holds of this object. */ + readonly #holders = new Set(); + /** Objects this object's activation waits for; followed to find cycles (Q5). */ + readonly #waitingFor = new Set(); + /** The current attempt's outcome: resolves when it goes live, rejects with its error; renewed by every retry. */ + #outcome = settlement(); + /** The error of the last failed attempt, until one goes live. */ + #lastFailure: ChordError | undefined; + /** Ends the object when its type's `linger` passed after its last reference; cleared by a new one. */ + #lingering: ReturnType | undefined; + /** Its running and finishing job runs: they keep it alive like references (D4), without being references. */ + #runs = 0; + /** + * The attempt its member sources show: the live one, or the last one that went live while the object is failed or + * activating again (their last values); `undefined` before the first went live. + */ + #exposed: Attempt | undefined; + /** Its state members as references and other objects' derives follow them, by name; each made by its first use. */ + readonly #members = new Map(); + /** The keys of its keyed members that references hold, across attempts. */ + readonly #keys: KeyedMembers; + + constructor( + host: AnyHost, + served: Served, + params: ParamsOf, + principal: Principal | undefined, + key: string, + identity: string, + context: Context, + chain: readonly string[], + ) { + this.host = host; + this.served = served; + this.params = params; + this.principal = principal; + this.identity = identity; + this.label = key === "" ? served.contract.name : `${served.contract.name}(${key})`; + this.chain = chain; + this.#context = context; + this.#keys = new KeyedMembers(host.batch, context); + const { store } = host; + this.documents = new Documents({ + label: this.label, + batch: host.batch, + context, + store: + store === undefined + ? undefined + : { store, owner: `${host.address?.kind ?? "client"}:${served.contract.name}` }, + fail: (error) => { + if (this.#state.kind === "live") this.failLive(this.#state.attempt, error); + }, + }); + } + + /** The object's state as its references show it. */ + get state(): RefState { + const state = this.#state; + if (state.kind === "failed") return state.failure; + if (state.kind === "live") return LIVE; + return state.kind === "ended" ? state.ended : LOADING; + } + + /** Whether its state members have values: once live (a local object stays live until it ends). */ + get hydrated(): boolean { + return this.#state.kind === "live"; + } + + /** Whether its state members have values while it is not live: once an attempt went live, until it ends. */ + get lastValues(): boolean { + return this.#exposed !== undefined && this.#state.kind !== "ended"; + } + + /** The current attempt's outcome (the start barrier waits for the first). */ + get settled(): Promise { + return this.#outcome.promise; + } + + /** The error of the last failed attempt while none went live since. */ + get lastFailure(): ChordError | undefined { + return this.#lastFailure; + } + + /** Its instance id while an attempt activates; `undefined` otherwise (a hold's chain names only activations). */ + get activationId(): string | undefined { + return this.#state.kind === "activating" ? this.#instance : undefined; + } + + /** + * The chain a hold this object's activation makes over a link carries (Q5): its own chain and its instance id, the + * last 32; `undefined` when it is not activating. + */ + activationChain(): string[] | undefined { + const id = this.activationId; + if (id === undefined) return undefined; + const { chain } = this; + return [...(chain.length < CHAIN ? chain : chain.slice(chain.length - CHAIN + 1)), id]; + } + + /** Its holders, for a presence list made after they took it. */ + get holders(): ReadonlySet { + return this.#holders; + } + + /** The variant a live remote object's factory chose; `undefined` for a local object and before it is live. */ + get variant(): string | undefined { + return this.#state.kind === "live" ? this.#state.attempt.variant : undefined; + } + + /** The live object's state members, as Chord follows them. */ + get stateMembers(): ReadonlyMap { + return this.#live().stateMembers; + } + + /** + * Starts an attempt when waiting or failed; a trigger while an attempt runs (or once live) is ignored. After a + * failure the holders are told: their references are loading again. + */ + attempt(): void { + const state = this.#state; + if (state.kind !== "waiting" && state.kind !== "failed") return; + this.#instance ??= uuidv7(); + const attempt = new Attempt(this, state.kind === "failed" ? state.holding : undefined, this.#context); + this.#state = { kind: "activating", attempt }; + if (state.kind === "failed") { + this.#outcome = settlement(); + this.#notify(); + } + void this.#run(attempt); + } + + /** + * `ref.retry()` and a `retry` message: an attempt now when failed (a broken one too) or waiting, its backoff window + * skipped; ignored while an attempt runs and once live. + */ + retry(): void { + if (this.#state.kind !== "waiting" && this.#state.kind !== "failed") return; + this.host.backoff.skip(this.identity); + this.attempt(); + } + + /** Calls a method of the live object for `caller`. */ + async call(method: string, args: unknown, context: Context, caller: Caller): Promise { + return this.#live().call(method, args, context, caller); + } + + /** + * The state member `member` as references and other objects' derives follow it across attempts: its member source, + * with its last value while the object is not live (`lastValues`). + */ + member(member: string): DeriveSource { + // Made once per member and dropped at the end: a source found here is the one to follow. + const made = this.#members.get(member); + if (made !== undefined) return made; + const exposed = this.#exposed; + if (exposed === undefined || this.#state.kind === "ended") return this.#live().member(member); + return this.#memberSource(member, exposed); + } + + /** The keyed member `member` of the live object: its ultimate slot (slice 6a §12). */ + keyedMember(member: string): KeyedSlot { + return this.#live().keyedMember(member); + } + + /** Pins `key` of the live object's keyed member `member` across its attempts (`KeyedMembers.pin`). */ + async pinKey(member: string, key: string, context: Context): Promise { + const attempt = this.#live(); + const stored = attempt.keyedMember(member).key(key); + return this.#keys.pin(member, stored, attempt.keyedMembers.get(member)!, context); + } + + /** + * An activating attempt's keyed members are checked: the keys references pin are loaded on them before it goes + * live; `undefined` when none is held. + */ + prepareKeys(members: ReadonlyMap, context: Context): Promise | undefined { + return this.#keys.prepare(members, context); + } + + /** + * Counts `holder` as a reference to this object (the holder reads `state` itself), listed in its type's presence; a + * lingering object stays. + */ + hold(holder: Holder): void { + this.#holders.add(holder); + this.#stopLingering(); + // Last: a derive over the presence list may close a link synchronously, which may release this holder again. + this.host.referenced(this, holder, this.#context); + } + + /** `holder` was released: the object may end (`#idle`). */ + unhold(holder: Holder): void { + if (!this.#holders.delete(holder)) return; + this.#idle(); + // Last: a derive over the presence list may close a link synchronously, which releases other holders again. + this.host.unreferenced(this, [holder], this.#context); + } + + /** + * The records of job `definition`'s runs, made by its first declaration. A later declaration of another version is + * refused (versions of runs across attempts are slice 7's). + */ + jobRecords(definition: JobDefinition): JobRecords { + const records = this.#jobs.get(definition.name); + if (records === undefined) { + const owner = { + label: this.label, + host: this.host, + context: this.#context, + documents: this.documents, + countRuns: (delta: number) => this.countRuns(delta), + }; + const made = new JobRecords(definition, owner); + this.#jobs.set(definition.name, made); + return made; + } + if (records.definition.version !== definition.version) { + throw new Error( + `${this.label}: job "${definition.name}" is declared at version ${definition.version}, its runs are at ${records.definition.version}`, + ); + } + return records; + } + + /** + * A runtime failure of the live attempt `attempt` (a `migrate` failing on access, a stored version newer than the + * code): the attempt is sealed and the object `failed` with the broken `cause`; calls reject with it, holders are + * told, the state members keep their values once the admitted commits adopted, the natives close in the background + * (then its references are released), `onError` gets `object_failed` once. Documents and job runs stay. Recovery: + * `ref.retry()` (a new attempt, whose members the member sources follow once it is live). A failure of a sealed + * attempt is ignored. + */ + failLive(attempt: Attempt, error: ChordError): void { + const state = this.#state; + if (state.kind !== "live" || state.attempt !== attempt) return; + const cause: FailureCause = { kind: "broken", error }; + this.host.backoff.delete(this.identity); + const failure: Failed = { kind: "failed", cause, retryAt: undefined }; + this.#state = { kind: "failed", failure, holding: undefined }; + this.#lastFailure = error; + this.#outcome = settlement(); + this.#outcome.reject(error); + attempt.seal(error); + // Members following another object's sources freeze now: no commit of this object changes them. + this.#freeze(attempt, "foreign"); + // The commits admitted before the seal adopt and publish once (frames included); then the other members keep + // their values. A retry's factory and hydration wait for both (its ending), so no old commit lands in a new + // attempt. + const frozen = this.documents.settled().then(() => this.#freeze(attempt, "all")); + const closing = Promise.all([attempt.close(END), frozen]).then(([failures]) => { + attempt.detach(); + attempt.releaseReferences(); + return failures; + }); + this.host.ending(this.identity, this.#reported(closing)); + this.host.report({ kind: "object_failed", object: this.label, failure: cause }); + this.#notify(); + this.#idle(); + } + + /** + * Adds `delta` to its running and finishing job runs (and `host.activity.jobs`). A new run keeps a lingering object; + * the last one settling on an object without references ends it as its last release would (`#idle`). + */ + countRuns(delta: number): void { + this.#runs += delta; + this.host.count("jobs", delta); + // A run started while the object lingers keeps it, like a new reference. + if (delta > 0) this.#stopLingering(); + if (this.#runs === 0) this.#idle(); + } + + /** + * Throws `invalid/cycle` when waiting for `dependency` closes a chain of waits back to this object (Q5). Every other + * object on the chain fails broken with the same error, so no part of a cycle waits for a retry. Counts the objects + * it visits (`chord.cycle.visits`). Otherwise, a dependency whose activation is in this object's chain closes a + * cycle through other hosts (they fail through their dependency on this one). + */ + assertNoCycle(dependency: Target, context: Context): void { + const visited = new Set(); + const cycle = dependency.pathTo(this, visited); + context.telemetry.metrics?.counter("chord.cycle.visits").add(visited.size); + if (cycle === undefined) { + const across = dependency.activationId; + if (across === undefined || !this.chain.includes(across)) return; + throw this.host.cycle(this.chain, across, [this.label, dependency.label]); + } + const error = invalid("cycle", `dependency cycle: ${[this, ...cycle].map(({ label }) => label).join(" → ")}`); + for (const object of cycle) { + const state = object.#state; + if (object !== this && state.kind === "activating") { + state.attempt.abandon({ kind: "failed", cause: { kind: "broken", error } }); + } + } + throw error; + } + + /** + * Waits for `dependency`'s current attempt as part of this object's activation. Rejects at once when it is failed, + * and when its attempt fails, with `unavailable` naming it (R18), its error as the cause; a dependency that failed + * for a cycle rejects with that `invalid/cycle` itself. + */ + async waitFor(dependency: Target, context: Context): Promise { + this.#waitingFor.add(dependency); + try { + await awaitWithContext(dependency.settled, context); + } catch (error) { + if (isChordError(error, "invalid") && error.reason === "cycle") throw error; + const message = `${this.label}: dependency ${dependency.label} failed`; + throw new ChordError({ code: "unavailable", message }, { cause: error }); + } finally { + this.#waitingFor.delete(dependency); + } + } + + /** A dependency went live: retried at once (backoff window skipped) when it failed on a dependency (R18). */ + dependencyLive(): void { + if (this.#state.kind !== "failed" || this.#state.holding === undefined) return; + this.host.backoff.skip(this.identity); + this.attempt(); + } + + /** + * Ends the object (once): sealed now, its holders and running calls told `error` (`released` by default), torn down + * in the background with reason `end`, close failures reported. + */ + end(error: ChordError = this.ended()): void { + const teardown = this.#seal(error); + if (teardown.kind !== "none") this.host.ending(this.identity, this.#reported(this.#tearDown(teardown, END))); + } + + /** + * Ends the object for `host.close`: sealed now; returns its teardown (reason `close`), which the host runs once every + * object is sealed, so no reference it releases starts an ending in the background. + */ + stop(): () => Promise { + const teardown = this.#seal(this.ended()); + return () => this.#tearDown(teardown, CLOSE); + } + + /** The `released` error of this object. */ + ended(): ChordError { + return new ChordError({ code: "released", reason: "ended", message: `${this.label}: ended` }); + } + + async #run(attempt: Attempt): Promise { + const outcome = await attempt.activate(this.host.endingOf(this.identity)); + // The object may have ended after the activation's last step. + if (this.#state.kind !== "activating" || this.#state.attempt !== attempt) return; + switch (outcome.kind) { + case "live": + return this.#wentLive(attempt); + case "failed": + return this.#failed(attempt, outcome.cause, undefined); + case "dependency_failed": + return this.#failed(attempt, outcome.cause, attempt); + } + } + + #wentLive(attempt: Attempt): void { + this.#state = { kind: "live", attempt }; + this.#exposed = attempt; + if (this.#members.size > 0 || !this.#keys.empty) + this.host.batch.within(() => this.#rebind(attempt), this.#context); + this.#lastFailure = undefined; + this.host.wentLive(this); + this.#outcome.resolve(); + this.#notify(); + // R18: objects that failed because this one had failed are retried now. + for (const holder of [...this.#holders]) holder.dependent?.dependencyLive(); + attempt.live(); + } + + /** + * Records a failed attempt: reported, torn down in the background (reason `end`), its references released unless it + * is `holding` them (R18), its holders told. Transient: a backoff window, whose end is the failure's `retryAt`; + * broken: no automatic retry. + */ + #failed(attempt: Attempt, cause: FailureCause, holding: Attempt | undefined): void { + // What its activation loaded for the pinned keys goes; they stay frozen. + this.#keys.discard(); + if (cause.kind === "transient") this.host.backoff.failed(this.identity); + else this.host.backoff.delete(this.identity); + const failure: Failed = { kind: "failed", cause, retryAt: this.host.backoff.until(this.identity) }; + this.#state = { kind: "failed", failure, holding }; + this.#lastFailure = cause.error; + attempt.seal(this.ended()); + if (holding === undefined) attempt.releaseReferences(); + const closing = attempt.close(END).then((failures) => { + attempt.detach(); + return failures; + }); + this.host.ending(this.identity, this.#reported(closing)); + this.host.report({ kind: "activation_failed", object: this.label, failure: cause }); + this.#outcome.reject(cause.error); + this.#notify(); + // Job runs kept it while it activated; failed, they keep it no longer. + this.#idle(); + } + + /** The member source of `member`, made bound to `exposed`'s member by its first use. */ + #memberSource(member: string, exposed: Attempt): MemberSource { + let source = this.#members.get(member); + if (source === undefined) { + source = new MemberSource(exposed.member(member), this.host.batch); + this.#members.set(member, source); + } + return source; + } + + /** + * The attempt `attempt` failed live: its state members and pinned keys keep their values from now on (each made now + * when nothing used it yet, so reads while failed find it). `foreign`: those following another object's sources + * (at once); `all`: every one (once the admitted commits adopted). + */ + #freeze(attempt: Attempt, which: "foreign" | "all"): void { + if (this.#exposed !== attempt || this.#state.kind === "ended") return; + for (const [member, source] of attempt.stateMembers) { + if (which === "all" || !attempt.owns(source)) this.#memberSource(member, attempt).freeze(); + } + // A pin still loading is given up with the failure: a consumer activating on it fails transient (R18). + this.#keys.freeze(which, this.#lastFailure ?? this.ended()); + } + + /** + * A new attempt went live, inside a batch: every member source follows its member of the same name, every pinned key + * its keyed member's (loaded by its activation). + */ + #rebind(attempt: Attempt): void { + const members = attempt.stateMembers; + for (const [member, source] of this.#members) { + const found = members.get(member); + if (found !== undefined) { + source.bind(found); + continue; + } + const variant = attempt.variant === undefined ? "" : ` (variant ${attempt.variant})`; + source.vanish(invalid("member", `${this.label}${variant} has no state member "${member}"`), this.#context); + } + this.#keys.rebind(); + } + + /** Tells every holder that the state changed. */ + #notify(): void { + for (const holder of [...this.#holders]) holder.moved(); + } + + /** + * Marks the object ended: forgotten by the host, waits rejected, references to it ended with `error` (and gone from + * its type's presence), its job runs ended (signals fired, waiters rejected), its transactions refused, its attempt + * sealed. Returns what is left to tear down; it releases nothing itself. + */ + #seal(error: ChordError): Teardown { + const state = this.#state; + if (state.kind === "ended") return { kind: "none" }; + const previous = this.host.endingOf(this.identity); + this.#state = { kind: "ended", ended: { kind: "ended", error } }; + for (const records of this.#jobs.values()) records.end(error); + this.documents.seal(); + this.#stopLingering(); + this.host.forget(this); + this.#outcome.reject(error); + // In a batch: the derives over its members are marked, so none keeps a cached value. + this.host.batch.within(() => { + for (const source of this.#members.values()) source.objectEnded(error); + this.#keys.end(error); + }, this.#context); + this.#members.clear(); + this.#exposed = undefined; + const holders = [...this.#holders]; + this.#holders.clear(); + this.host.unreferenced(this, holders, this.#context); + for (const holder of holders) holder.end(error); + if (state.kind === "waiting") return { kind: "end", previous }; + if (state.kind === "failed") { + // The failed attempt is closed (or closing) since it failed; only the references it holds (R18) are left. + const { holding } = state; + return holding === undefined ? { kind: "end", previous } : { kind: "release", attempt: holding, previous }; + } + state.attempt.seal(error); + return { kind: "close", attempt: state.attempt, previous }; + } + + /** + * A remote object or a local one with params ends once it has no references and no running or finishing job runs: + * a live one once its type's `linger` passed without a new reference, any other at once. + */ + #idle(): void { + const kind = this.#state.kind; + // A run waiting for a live attempt keeps its object only while one is live or activating. + const running = this.#runs > 0 && (kind === "live" || kind === "activating"); + const kept = this.#holders.size > 0 || running || (this.served.kind === "local" && this.served.hostLifetime); + if (kept || this.#state.kind === "ended" || this.#lingering !== undefined) return; + if (this.#state.kind !== "live" || this.served.linger === 0) { + this.end(); + return; + } + // A new holder clears the timer (`hold`), so an expiry finds none. + this.#lingering = setTimeout(() => { + this.#stopLingering(); + this.end(); + }, this.served.linger); + this.host.count("lingering", 1); + } + + /** Its linger timer stops (a new holder, its expiry, its end): no longer counted as lingering. */ + #stopLingering(): void { + if (this.#lingering === undefined) return; + clearTimeout(this.#lingering); + this.#lingering = undefined; + this.host.count("lingering", -1); + } + + /** + * Once the previous teardowns finished (a failed attempt's closes may read the documents): drains and closes a + * sealed attempt, then releases its references (after the closes, which may use them), then waits for an admitted + * commit step to adopt and drops the documents and job runs. Resolves with the closes that threw. + */ + async #tearDown(teardown: Teardown, reason: NativeCloseReason): Promise { + if (teardown.kind === "none") return []; + await teardown.previous; + const failures = teardown.kind === "close" ? await teardown.attempt.close(reason) : []; + if (teardown.kind === "close") teardown.attempt.detach(); + if (teardown.kind !== "end") teardown.attempt.releaseReferences(); + await this.documents.settled(); + this.documents.end(); + for (const records of this.#jobs.values()) records.drop(); + this.#jobs.clear(); + return failures; + } + + /** Reports the close failures of a teardown nobody waits for. */ + async #reported(closing: Promise): Promise { + for (const failure of await closing) this.host.report(failure); + } + + /** + * The live attempt; `not_ready` before the object went live and while it activates again, the cause while failed, + * `released` once it ended. + */ + #live(): Attempt { + const state = this.#state; + if (state.kind === "live") return state.attempt; + if (state.kind === "ended") throw this.ended(); + if (state.kind === "failed") throw state.failure.cause.error; + throw new ChordError({ code: "not_ready", message: `${this.label}: not live yet` }); + } + + /** The chain of waits from this activation to `target` (see `Target`). */ + pathTo(target: ChordObject, visited: Set): ChordObject[] | undefined { + if (this === target) return [this]; + visited.add(this); + for (const next of this.#waitingFor) { + if (visited.has(next)) continue; + const rest = next.pathTo(target, visited); + if (rest !== undefined) return [this, ...rest]; + } + return undefined; + } +} + +/** A settlement nobody may wait for: its rejection is not unhandled. */ +export function settlement(): PromiseWithResolvers { + const settlement = Promise.withResolvers(); + settlement.promise.catch(() => {}); + return settlement; +} diff --git a/packages/chord/src/next/host/parked.ts b/packages/chord/src/next/host/parked.ts new file mode 100644 index 000000000..796ade209 --- /dev/null +++ b/packages/chord/src/next/host/parked.ts @@ -0,0 +1,98 @@ +// Parked calls (§3, §7): calls made on a loading reference, queued in the order they were made until it goes live, +// each with its own deadline from the moment it was called (the reference's `readyTimeout`, never renewed). + +import type { Context } from "../../context/index.ts"; +import { ChordError } from "../data/index.ts"; + +/** One queued call: what to send, how its caller is settled, its deadline (`Date.now()` time) and what ends its wait. */ +type ParkedCall = { + readonly method: string; + readonly args: unknown; + readonly context: Context; + readonly resolve: (value: unknown) => void; + readonly reject: (error: unknown) => void; + readonly due: number; + readonly deadline: ReturnType; + readonly cancel: (() => void) | undefined; +}; + +/** + * The queue of one reference's parked calls. A call leaves it when it is sent, at its deadline (`unavailable/ + * park_timeout`), when its caller cancels, or rejected with the queue (the reference failed or was released). + */ +export class ParkedCalls { + /** In the order the calls were made (a `Set` keeps insertion order and deletes in constant time). */ + readonly #calls = new Set(); + readonly #timeout: number; + readonly #label: string; + + constructor(timeout: number, label: string) { + this.#timeout = timeout; + this.#label = label; + } + + /** Whether calls are queued: a new call waits behind them, also while they are being sent. */ + get waiting(): boolean { + return this.#calls.size > 0; + } + + /** Queues a call; settles as the call it becomes once sent, or with why it left the queue. */ + park(method: string, args: unknown, context: Context): Promise { + const signal = context.abortSignal; + signal?.throwIfAborted(); + const { promise, resolve, reject } = Promise.withResolvers(); + const deadline = setTimeout(() => this.#expire(call), this.#timeout); + let cancel: (() => void) | undefined; + if (signal !== undefined) { + cancel = () => { + this.#take(call); + reject(signal.reason); + }; + signal.addEventListener("abort", cancel, { once: true }); + } + const due = Date.now() + this.#timeout; + const call: ParkedCall = { method, args, context, resolve, reject, due, deadline, cancel }; + this.#calls.add(call); + return promise; + } + + /** + * Sends the queued calls in order through `send`, each only while `live()` still holds: sending one runs code + * (a handler's synchronous part) that may release or drop the reference, and the rest then stays queued or is + * rejected by that; or that makes a new call, which is queued behind the rest and sent in this loop. A call whose + * deadline passed while earlier ones ran (their timers could not fire) is rejected instead of sent. + */ + flush(send: (method: string, args: unknown, context: Context) => Promise, live: () => boolean): void { + for (const call of this.#calls) { + if (!live()) return; + if (Date.now() >= call.due) { + this.#expire(call); + continue; + } + this.#take(call); + send(call.method, call.args, call.context).then(call.resolve, call.reject); + } + } + + /** Rejects every queued call with `error`. */ + reject(error: unknown): void { + for (const call of this.#calls) { + this.#take(call); + call.reject(error); + } + } + + /** `call`'s deadline passed: `unavailable/park_timeout`. */ + #expire(call: ParkedCall): void { + this.#take(call); + const message = `${this.#label}: a call waited ${this.#timeout} ms for its reference to go live`; + call.reject(new ChordError({ code: "unavailable", reason: "park_timeout", message })); + } + + /** Takes `call` out of the queue: its deadline and cancellation stop. */ + #take(call: ParkedCall): void { + this.#calls.delete(call); + clearTimeout(call.deadline); + if (call.cancel !== undefined) call.context.abortSignal?.removeEventListener("abort", call.cancel); + } +} diff --git a/packages/chord/src/next/host/placement.ts b/packages/chord/src/next/host/placement.ts new file mode 100644 index 000000000..67f4215ca --- /dev/null +++ b/packages/chord/src/next/host/placement.ts @@ -0,0 +1,112 @@ +// Placement (§3, section 1's table): where a reference made on a host, or a hold arriving on one of its links, goes: +// here, over the route of its placement's address, or over the parent link; and the check that an object served here +// is placed here. + +import type { Context } from "../../context/index.ts"; +import type { Address, Contract, ParamsOf } from "../contracts/index.ts"; +import { contractData } from "../contracts/internal.ts"; +import { ChordError, type Principal } from "../data/index.ts"; +import { invalid } from "../data/internal.ts"; +import type { WireAddress } from "../link/internal.ts"; +import type { HoldRequest, HostLink, Path } from "../remote/internal.ts"; +import type { Served } from "./object.ts"; +import type { Route } from "./route.ts"; + +/** Where something goes: here, or over a path (with the address it was placed at, if its type has placement). */ +export type Placed = + | { readonly kind: "here" } + | { readonly kind: "path"; readonly path: Path; readonly address: WireAddress | undefined }; + +/** A host's placement: its address, its served types, its routes and its parent path, as the host keeps them. */ +export class Placement { + readonly #address: Address | undefined; + readonly #types: ReadonlyMap; + readonly #routes: ReadonlyMap; + readonly #parent: Path; + + constructor( + address: Address | undefined, + types: ReadonlyMap, + routes: ReadonlyMap, + parent: Path, + ) { + this.#address = address; + this.#types = types; + this.#routes = routes; + this.#parent = parent; + } + + /** + * Where a reference to `Type` with `params` made as `principal` goes: here when its type is served here and placed + * here (or this host has no address, or the type no placement); otherwise over the route of its placement's kind, + * or over the parent link (placement decides where an object lives: a worker reaches another worker's object of a + * type it serves too). Throws `invalid/placement` for a placement whose params do not fit its kind, `forbidden` when + * the route refuses, and what a placement or `authorize` throws. + */ + reference(Type: Contract, params: ParamsOf, principal: Principal | undefined, context: Context): Placed { + const address = placementOf(Type, params); + if (this.#types.has(Type.name) && this.#placedHere(address)) return HERE; + return { kind: "path", path: this.#pathTo(address, principal, context) ?? this.#parent, address }; + } + + /** + * Where a hold arriving on `link` acting as `principal` goes: here when its type is served here; over the route of + * the address it names; else over the parent link, unless it came on the parent link or there is none + * (`unavailable`). Throws as `reference` does. + */ + hold(request: HoldRequest, link: HostLink, principal: Principal | undefined, context: Context): Placed { + if (this.#types.has(request.type)) return HERE; + const { address } = request; + const routed = this.#pathTo(address, principal, context); + if (routed !== undefined) return { kind: "path", path: routed, address }; + if (link.role === "parent" || this.#parent.link === undefined) { + throw new ChordError({ code: "unavailable", message: `${request.type} is not served here` }); + } + return { kind: "path", path: this.#parent, address }; + } + + /** + * Throws `invalid/placement` for a hold of `Type` with `params` (checked) served here but placed on another host + * (misrouted; never forwarded). The placement is computed here from the params: the address a hold names only + * routes it. + */ + assertHere(Type: Contract, params: ParamsOf): void { + const address = placementOf(Type, params); + if (this.#placedHere(address)) return; + throw invalid("placement", `${Type.name}: placed on ${address!.kind}, this host is ${this.#address!.key}`); + } + + /** + * The path to `address` when this host routes its kind (its params checked, `authorize` run for `principal`); + * `undefined` without an address or a route for its kind. + */ + #pathTo(address: WireAddress | undefined, principal: Principal | undefined, context: Context): Path | undefined { + if (address === undefined) return undefined; + return this.#routes.get(address.kind)?.path(address.params, principal, context); + } + + /** Whether an object placed at `address` lives here: this host has no address, the type no placement, or equal. */ + #placedHere(address: WireAddress | undefined): boolean { + const own = this.#address; + return own === undefined || address === undefined || sameAddress(own, address); + } +} + +const HERE: Placed = { kind: "here" }; + +/** The address `Type`'s placement computes for `params`; `undefined` for a type without placement. */ +function placementOf(Type: Contract, params: ParamsOf): WireAddress | undefined { + const placement = contractData(Type).placement; + if (placement === undefined) return undefined; + // Plugin code: what it returns is checked where it is routed or served. + const { kind, params: values } = placement(params) as { readonly kind: string; readonly params: unknown }; + return { kind, params: values }; +} + +/** Whether `address` names the host at `own`: the same kind and the same value for each of its params. */ +function sameAddress(own: Address, address: WireAddress): boolean { + if (own.kind !== address.kind || typeof address.params !== "object" || address.params === null) return false; + const values = address.params as { readonly [field: string]: unknown }; + const fields = Object.keys(own.params); + return Object.keys(values).length === fields.length && fields.every((field) => values[field] === own.params[field]); +} diff --git a/packages/chord/src/next/host/presence.ts b/packages/chord/src/next/host/presence.ts new file mode 100644 index 000000000..768153f56 --- /dev/null +++ b/packages/chord/src/next/host/presence.ts @@ -0,0 +1,132 @@ +// Presence (§19): `host.references(Type)`, a JSON watchable of the references to objects of one type on this host, one +// entry per reference with the object's params, the reference's principal (remote types) and its incarnation. + +import type { Context } from "../../context/index.ts"; +import type { Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Contract, ParamsOf } from "../contracts/index.ts"; +import type { Principal } from "../data/index.ts"; +import { Source } from "../state/internal.ts"; +import type { ChordObject, Holder } from "./object.ts"; + +/** + * One reference to an object of `C`: the object's params, the principal it acts as (for a remote type), and its + * incarnation: the host that made it as its caller sees it (a hold's caller over a link; two tabs are two + * incarnations), the word `ctx.incarnation` uses. + */ +export type Presence = { + readonly params: ParamsOf; + readonly principal?: P; + readonly incarnation: string; +}; + +/** + * The references of one type, made by the host for its first `host.references(Type)` and kept from then on: one entry + * per reference, added when it binds and removed when it ends, in the batch where that happens (derives over it + * recompute there). While watched, a batch's changes are folded into the published list once, with one splice op per + * removal and one for the additions; otherwise the list is built on its next read. + */ +export class References extends Source { + readonly #entries = new Map(); + /** The list as last published or read; `undefined` once it changed while nobody watched it. */ + #list: readonly Presence[] | undefined; + /** Entries added since `#list`, in order (only while it is defined). */ + #added: Presence[] = []; + /** Entries of `#list` removed since. */ + readonly #removed = new Set(); + /** The ops folding `#list` forward, for the current batch's frame. */ + #ops: Op[] = []; + + get value(): JsonValue { + this.#fold(); + // Entries are JSON: params, a principal and a string. + return this.#list as unknown as JsonValue; + } + + /** `holder` references `object`, added in the batch running now or the next one. */ + add(object: ChordObject, holder: Holder, context: Context): void { + this.batch.within(() => { + const entry = this.record(object, holder); + if (this.#follows()) this.#added.push(entry); + this.changed(); + }, context); + } + + /** `holders` of `object` ended, removed in one batch. */ + remove(holders: Iterable, context: Context): void { + this.batch.within(() => { + const follows = this.#follows(); + for (const holder of holders) { + const entry = this.#entries.get(holder); + if (entry === undefined) continue; + this.#entries.delete(holder); + if (!follows) continue; + const added = this.#added.indexOf(entry); + if (added === -1) this.#removed.add(entry); + else this.#added.splice(added, 1); + } + this.changed(); + }, context); + } + + /** Records `holder` without a batch (the list being made) and returns its entry. */ + record(object: ChordObject, holder: Holder): Presence { + const { principal, incarnation } = holder.caller; + const entry: Presence = { + params: object.params, + ...(object.served.kind === "remote" && principal !== undefined ? { principal } : {}), + incarnation, + }; + this.#entries.set(holder, entry); + return entry; + } + + deliver(context: Context): void { + const value = this.value; + const ops = this.#ops; + this.#ops = []; + // Listed once per change: the batch's first delivery carries all of them. + if (ops.length > 0) this.publish(value, ops, false, context); + } + + /** + * Whether a change is folded into the published list: while watched (a watch read the list first). Otherwise the + * list is dropped and built on its next read. + */ + #follows(): boolean { + if (this.watched && this.#list !== undefined) return true; + this.#list = undefined; + return false; + } + + /** + * Folds the changes since `#list` into a new list, once: a removal is one splice op at its index, the additions one + * splice op at the end. Builds the list from the entries when there is none. + */ + #fold(): void { + if (this.#list === undefined) { + this.#list = [...this.#entries.values()]; + this.#added = []; + this.#removed.clear(); + this.#ops = []; + return; + } + if (this.#added.length === 0 && this.#removed.size === 0) return; + let list = this.#list; + if (this.#removed.size > 0) { + const kept: Presence[] = []; + for (const entry of list) { + if (this.#removed.has(entry)) this.#ops.push(["p", [], kept.length, 1, []]); + else kept.push(entry); + } + this.#removed.clear(); + list = kept; + } + if (this.#added.length > 0) { + this.#ops.push(["p", [], list.length, 0, this.#added as unknown as JsonValue[]]); + list = [...list, ...this.#added]; + this.#added = []; + } + this.#list = list; + } +} diff --git a/packages/chord/src/next/host/ref-state.ts b/packages/chord/src/next/host/ref-state.ts new file mode 100644 index 000000000..75f9fa42e --- /dev/null +++ b/packages/chord/src/next/host/ref-state.ts @@ -0,0 +1,91 @@ +// The state of a reference (§7): loading, live, failed or ended; and `ref.state`, the native watchable that follows +// it. + +import type { Context } from "../../context/index.ts"; +import { NativeWatch, type NativeWatchable } from "../../watch.ts"; +import type { ChordError, FailureCause } from "../data/index.ts"; + +/** + * Where a reference stands. `loading`: its object activates or retries, it waits for `load` or a parent link, or its + * link is down (the reference's coarser view of its object's waiting and activating); `failed`: its object failed and + * no attempt runs (`retryAt` undefined: broken, no automatic retry); `live`: bound; `ended`: for good, with the error + * that says why (`released` without a reason: released by its holder; `released/ended`: its object ended; + * `released/closed`: its host closed; `forbidden/revoked`). + */ +export type RefState = + | { readonly kind: "loading" } + | { readonly kind: "live" } + | { readonly kind: "failed"; readonly cause: FailureCause; readonly retryAt: number | undefined } + | { readonly kind: "ended"; readonly error: ChordError }; + +/** A failed reference's state. */ +export type Failed = Extract; + +/** An ended reference's state. */ +export type Ended = Extract; + +/** The states without fields, one object each: a change of state is a new object (a native watch compares by identity). */ +export const LOADING: RefState = { kind: "loading" }; + +export const LIVE: RefState = { kind: "live" }; + +const IGNORE = (): void => {}; + +/** + * `ref.state`: its value is the reference's state, read through `read`; each watch gets every new state, in its own + * context, and ends `ended` when the reference is released (the end is the notification). + */ +export class ReferenceState implements NativeWatchable { + readonly #read: () => RefState; + /** Where a listener's throw goes (the host); dropped on release, so a released reference keeps no host. */ + #report: (error: unknown) => void; + /** Throws `invalid/reentrant` inside the host's batch (a watch taken in a derive compute); dropped on release. */ + #assertOutside: () => void; + /** The watches with the contexts they were taken in, which their frames carry. */ + readonly #watches = new Map, Context>(); + /** The state the watches have: a change is announced only when the state is another object. */ + #last: RefState; + + constructor(read: () => RefState, report: (error: unknown) => void, assertOutside: () => void) { + this.#read = read; + this.#report = report; + this.#assertOutside = assertOutside; + this.#last = read(); + } + + get value(): RefState { + return this.#read(); + } + + async watch(context: Context): Promise> { + this.#assertOutside(); + const state = this.#read(); + const watch: NativeWatch = new NativeWatch(state, () => this.#watches.delete(watch), this.#report); + // An ended reference has nothing more to announce. + if (state.kind === "ended") { + watch.end({ kind: "ended" }); + return watch; + } + this.#watches.set(watch, context); + if (context.abortSignal !== undefined) watch.observeAbort(context.abortSignal); + return watch; + } + + /** The reference's state may have changed: every watch gets it when it is another state. */ + changed(): void { + const state = this.#read(); + if (state === this.#last) return; + this.#last = state; + for (const [watch, context] of this.#watches) watch.advance(state, context); + } + + /** The reference ended (`ended`, the reference's state): every watch ends `ended`. */ + end(ended: Ended): void { + // An ended reference keeps nothing of its object: not the last failure's error, not the host. + this.#last = ended; + this.#report = IGNORE; + this.#assertOutside = IGNORE; + // Ending detaches, which deletes the current entry (safe while iterating) and runs no listener. + for (const watch of this.#watches.keys()) watch.end({ kind: "ended" }); + } +} diff --git a/packages/chord/src/next/host/reference.ts b/packages/chord/src/next/host/reference.ts new file mode 100644 index 000000000..142dae425 --- /dev/null +++ b/packages/chord/src/next/host/reference.ts @@ -0,0 +1,674 @@ +// References (§7): what `get` and `hold` return, a plain object built from the contract's members, and its binding to +// the object: counted as a reference, it follows the object's state, parks calls while loading, keeps the +// per-reference watches and is released once. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { NativeWatchable, Watch, Watchable } from "../../watch.ts"; +import type { Contract, Implementation, ParamsOf } from "../contracts/index.ts"; +import { + type CurrentVariants, + currentVariants, + type MethodData, + type ShapeOf, + takesNoArgs, + type VariantImplementation, +} from "../contracts/internal.ts"; +import { ChordError, type FailureCause } from "../data/index.ts"; +import { Keyed, type KeyedSlot, type SealToken, Singleton } from "../documents/internal.ts"; +import { type Batch, Constant, type DeriveSource, Source } from "../state/internal.ts"; +import { type Caller, withLinkedAbortSignal } from "./ctx.ts"; +import type { PinnedKey } from "./keyed-members.ts"; +import type { KeyedBinding } from "./members.ts"; +import type { ChordObject, Holder } from "./object.ts"; +import { ParkedCalls } from "./parked.ts"; +import { type Ended, type Failed, LOADING, ReferenceState, type RefState } from "./ref-state.ts"; + +/** + * A reference: a local contract's members, or for a remote contract the members of the variant its object chose, + * narrowed by `variant`; and its state, `retry()` and `release()`. Every use after release throws. + */ +export type Reference = ShapeOf["kind"] extends "local" + ? Implementation & Lifecycle + : InVariants; + +/** What `host.hold` returns: a reference whose `variant` is `undefined` until its first binding. */ +export type Held = ShapeOf["kind"] extends "local" + ? Implementation & Lifecycle + : InVariants; + +/** A remote reference narrowed to some of its variants. */ +export type RefIn = Extract, { readonly variant: V }>; + +/** One member of a remote reference's union per variant; `Unbound` is what `variant` may also be. */ +type InVariants = { + [V in keyof CurrentVariants]: { readonly variant: V | Unbound } & VariantImplementation[V]> & + Lifecycle; +}[keyof CurrentVariants]; + +/** What every reference has besides its members. */ +type Lifecycle = { + /** The reference's state: loading, live, failed or released; watches get every change. */ + readonly state: NativeWatchable; + /** Asks a failed object for an attempt now (the backoff skipped, a broken one renewed); otherwise nothing. */ + retry(): void; + release(): void; +}; + +/** + * What a binding is bound to: an object on this host (`ChordObject`), or an object held over a path, the parent link + * or a route address (`RemoteObject`). It tells its holders of each change of `state`. + */ +export interface Target { + /** The object in errors: its type, and its params key when it has params. */ + readonly label: string; + /** Its state as its references show it. */ + readonly state: RefState; + /** Whether its state members have values: live once, for a remote object bound once. */ + readonly hydrated: boolean; + /** The variant bound last; `undefined` for a local object and before the first binding. */ + readonly variant: string | undefined; + /** The error of its last failed attempt while none went live since. */ + readonly lastFailure: ChordError | undefined; + /** Its instance id while it activates on this host (Q5); `undefined` otherwise and over a link. */ + readonly activationId: string | undefined; + /** The current attempt's outcome, which a dependent's activation waits for (R18). */ + readonly settled: Promise; + /** Counts `binding` as a reference; the binding reads `state` itself. */ + hold(binding: Binding): void; + /** `binding` was released. */ + unhold(binding: Binding): void; + /** Calls a method of the live object for `caller`. */ + call(method: string, args: unknown, context: Context, caller: Caller): Promise; + /** + * The state member `member` as Chord follows it: a local object's member source (across its attempts), or the + * replica of the last binding over a link. + */ + member(member: string): DeriveSource; + /** + * Whether its members keep their last values while it is failed, for same-host reads and watches (a local object + * once an attempt went live, slice 6a §1.4); a remote object's failure is its reads' error. + */ + readonly lastValues: boolean; + /** The keyed member `member` of the live object: its ultimate slot (slice 6a §12; only local objects have any). */ + keyedMember(member: string): KeyedSlot; + /** + * Pins `key` of the keyed member `member` (a reference's `at(key)`, a key watch): its source across the object's + * attempts once loaded (`keyed-members.ts`); only a live local object takes it. + */ + pinKey(member: string, key: string, context: Context): Promise; + /** `ref.retry()`: an attempt now if failed; ignored otherwise. */ + retry(): void; + /** + * The chain of activation waits from this object to `target`, both included (Q5); `undefined` when there is none. + * `visited` keeps a search over shared dependencies linear in the waits. + */ + pathTo(target: ChordObject, visited: Set): ChordObject[] | undefined; +} + +/** A reference's params as L1 checked them, with their identity key. */ +export type CheckedParams = { readonly params: ParamsOf; readonly key: string }; + +/** + * Where a binding stands. `unbound`: waiting for `load` (a `host.hold`) or for its object's activation to bind it (a + * `self.get`); `refused`: a `host.hold` whose type turned out not to be gettable here once `load` resolved; a released + * binding keeps nothing but its name and its end (`ref.state`'s `ended`). `dependent` is the object whose attempt declared the reference with `self.get`, + * `undefined` for `host.get`, `host.hold` and `ctx.get`. + */ +type BindingState = + | { readonly kind: "unbound"; readonly batch: Batch; readonly dependent: ChordObject | undefined } + | { + readonly kind: "bound"; + readonly batch: Batch; + readonly dependent: ChordObject | undefined; + readonly target: Target; + } + | { readonly kind: "refused"; readonly batch: Batch; readonly failure: Failed } + | { readonly kind: "released"; readonly state: Ended }; + +/** What a wait for a state decides at each change. */ +type Decision = + | { readonly kind: "wait" } + | { readonly kind: "done" } + | { readonly kind: "failed"; readonly error: unknown }; + +const WAIT: Decision = { kind: "wait" }; + +const DONE: Decision = { kind: "done" }; + +const IGNORE = (): void => {}; + +/** A wait for a state: checked at every change, rejected with the reference's end. */ +type Waiter = { readonly check: () => void; readonly end: (error: ChordError) => void }; + +/** + * A reference's binding to its object: counted by the object as one of its references while bound, and told of each + * change of its state. Calls run while live, park while loading (sent in order once live) and reject while failed. + * Released, or ended with its object, it ends the watches taken through it and rejects the calls made through it. + */ +export class Binding implements Holder { + readonly reference: Reference; + /** Who calls through this reference: the principal it was got as, and this host's incarnation. */ + readonly caller: Caller; + /** The contract's name, for errors. */ + readonly #name: string; + /** How long a call made while loading waits: the reference's `readyTimeout`. */ + readonly #parkTimeout: number; + #state: BindingState; + readonly #watches = new Set>(); + /** + * Aborted when the reference ends: its calls' callers are rejected and their handlers' contexts aborted. Made by the + * first call and dropped at the end, so a released reference keeps no error (whose stack keeps its frames). + */ + #calls: AbortController | undefined; + /** Calls made while loading; made by the first. */ + #parked: ParkedCalls | undefined; + /** Waits for a state (getters, member watches before the first binding); each checks the state at every change. */ + readonly #waiters = new Set(); + /** `ref.state`, made on first access. */ + #refState: ReferenceState | undefined; + /** Resolved when the reference ends; made by the first `ended()`. */ + #ended: PromiseWithResolvers | undefined; + /** The keys its object's factory pinned with `ref.member.at(key)`: pinned once bound, released with it. */ + readonly #keys: KeyedMemberAt[] = []; + + constructor( + contract: Contract, + batch: Batch, + dependent: ChordObject | undefined, + caller: Caller, + parkTimeout: number, + ) { + this.#name = contract.name; + this.caller = caller; + this.#parkTimeout = parkTimeout; + this.#state = { kind: "unbound", batch, dependent }; + const reference: Record = { release: () => this.release(), retry: () => this.retry() }; + Object.defineProperty(reference, "state", { get: () => this.#watchableState(), enumerable: true }); + // Every member of every variant (one kind and signature per name, D11); the bound variant decides which work. + const methods = new Map(); + for (const variant of currentVariants(contract).values()) { + for (const member of variant.state.keys()) reference[member] = new ReferenceMember(this, member); + for (const member of variant.keyed.keys()) reference[member] = new ReferenceKeyed(this, member); + for (const [member, method] of variant.methods) methods.set(member, method); + } + for (const [member, method] of methods) { + // A method without arguments omits them; `null` is their internal and wire form. + reference[member] = takesNoArgs(method) + ? async (context: Context) => this.#call(member, null, context) + : async (args: unknown, context: Context) => this.#call(member, args, context); + } + if (contract.kind === "remote") { + Object.defineProperty(reference, "variant", { get: () => this.#variant(), enumerable: true }); + } + // Built from the contract's own member lists. + this.reference = reference as Reference; + } + + /** The object whose attempt declared this reference, until it is released. */ + get dependent(): ChordObject | undefined { + const state = this.#state; + return state.kind === "unbound" || state.kind === "bound" ? state.dependent : undefined; + } + + /** The reference's state. */ + get refState(): RefState { + const state = this.#state; + if (state.kind === "bound") return state.target.state; + if (state.kind === "refused") return state.failure; + return state.kind === "unbound" ? LOADING : state.state; + } + + /** The object in errors: its label once bound, else the contract's name. */ + get label(): string { + return this.#state.kind === "bound" ? this.#state.target.label : this.#name; + } + + /** The error of the object's last failed attempt while none went live since. */ + get lastFailure(): ChordError | undefined { + const state = this.#state; + if (state.kind === "refused") return state.failure.cause.error; + return state.kind === "bound" ? state.target.lastFailure : undefined; + } + + /** Binds to `target`, which counts this binding as a reference; the reference takes its state. */ + bind(target: Target): void { + const state = this.#state; + if (state.kind !== "unbound") return; + this.#state = { ...state, kind: "bound", target }; + target.hold(this); + this.moved(); + } + + /** A `host.hold` that cannot be bound once `load` resolved: failed with `cause`, which no retry changes. */ + refuse(cause: FailureCause): void { + const state = this.#state; + if (state.kind !== "unbound") return; + this.#state = { kind: "refused", batch: state.batch, failure: { kind: "failed", cause, retryAt: undefined } }; + this.moved(); + } + + /** + * The state may have changed: `ref.state`'s watches are told; live sends the parked calls in order, failed + * rejects them with the cause; waits for a state decide. + */ + moved(): void { + this.#refState?.changed(); + const state = this.refState; + if (state.kind === "live") { + this.#parked?.flush( + (method, args, context) => this.#send(method, args, context), + () => this.refState.kind === "live", + ); + } else if (state.kind === "failed") { + this.#parked?.reject(state.cause.error); + } + for (const waiter of [...this.#waiters]) waiter.check(); + } + + /** Resolves when the reference is live; rejects with a broken failure's error, or its end. Getters wait for it. */ + live(context: Context): Promise { + return this.#until((state) => { + if (state.kind === "live") return DONE; + const broken = state.kind === "failed" && state.cause.kind === "broken"; + return broken ? { kind: "failed", error: state.cause.error } : WAIT; + }, context); + } + + /** + * The state member behind `member`: its last value while loading, and while failed when its object keeps one (a + * local object that went live once, slice 6a §1.4); throws `released`, `not_ready` before the first binding, and + * the failure's error otherwise. + */ + member(member: string): DeriveSource { + const state = this.#state; + if (state.kind === "bound" && state.target.lastValues) return state.target.member(member); + return this.#target().member(member); + } + + /** The keyed member `member`'s slot (slice 6a §12), as `member` resolves a state member. */ + keyedSlot(member: string): KeyedSlot { + return this.#target().keyedMember(member); + } + + /** Pins `key` of the keyed member `member` on the bound object (`Target.pinKey`), as `keyedSlot` resolves. */ + pinKey(member: string, key: string, context: Context): Promise { + try { + return this.#target().pinKey(member, key, context); + } catch (error) { + return Promise.reject(error); + } + } + + /** The bound object unless it failed: throws `released`, the failure's error, `not_ready` before the binding. */ + #target(): Target { + const state = this.#state; + switch (state.kind) { + case "bound": { + const current = state.target.state; + if (current.kind === "failed") throw current.cause.error; + return state.target; + } + case "unbound": + throw new ChordError({ code: "not_ready", message: `${this.#name}: not live yet` }); + case "refused": + throw state.failure.cause.error; + case "released": + throw state.state.error; + } + } + + /** + * A watch of `member` that ends when this reference is released. Before the first binding it waits for it (and + * rejects when the reference fails or is released first); an object that keeps its last values (`lastValues`) + * gives a watch at once, which gets frames again when the object is live again. + */ + async watch(member: string, context: Context): Promise> { + // Checked here too: a constant's own watch has no batch to check. A released reference throws below. + if (this.#state.kind !== "released") this.#state.batch.assertOutside("watch"); + if (this.#state.kind !== "bound" || !this.#state.target.lastValues) await this.hydrated(context); + return this.follow(this.member(member).openWatch(context)); + } + + /** Resolves once the object's members have values; rejects when the reference fails or is released first. */ + async hydrated(context: Context): Promise { + if (this.#state.kind === "bound" && this.#state.target.hydrated) return; + await this.#until((state) => { + if (state.kind === "live") return DONE; + if (state.kind === "failed") return { kind: "failed", error: state.cause.error }; + return WAIT; + }, context); + } + + /** + * Ends `watch` when this reference is released; returns it. A watch opened while the reference was released (an + * async keyed watch racing `release`) is ended at once, its pin with it, and the caller gets `released`. + */ + follow>(watch: W): W { + if (this.#state.kind === "released") { + watch.end({ kind: "ended" }); + throw this.#state.state.error; + } + this.#watches.add(watch); + void watch.closed.then(() => this.#watches.delete(watch)); + return watch; + } + + /** + * `ref.member.at(key)`: only while its object's factory runs (the reference not yet bound); pinned once bound + * (`pinKeys`), released with the reference. + */ + at(member: string, key: string): KeyedMemberAt { + if (this.#state.kind !== "unbound" || this.#state.dependent === undefined) { + throw new Error(`${this.#name}: ${member}.at(key) is valid only in the factory`); + } + const at = new KeyedMemberAt(this, member, key); + this.#keys.push(at); + return at; + } + + /** + * Pins the keys of `at(key)` on the live object's keyed members and loads those not in memory, before the + * dependent goes live (slice 6a §12); `undefined` when there are none. A failed load rejects. + */ + pinKeys(context: Context): Promise | undefined { + if (this.#keys.length === 0) return undefined; + return (async () => { + for (const at of this.#keys) await at.pin(this, context); + })(); + } + + /** + * Resolves when the reference ends (released, or its object ended); never rejects. A wait that must not keep the + * reference races it (a `host.hold` waiting for `load`). + */ + ended(): Promise { + if (this.#state.kind === "released") return Promise.resolve(); + this.#ended ??= Promise.withResolvers(); + return this.#ended.promise; + } + + /** `ref.retry()`: the object is asked for an attempt now; it ignores this unless failed. */ + retry(): void { + if (this.#state.kind === "bound") this.#state.target.retry(); + } + + /** Releases the reference (idempotent); the object may end when it was its last. */ + release(): void { + const state = this.#state; + if (state.kind === "released") return; + state.batch.assertOutside("release"); + // First: an object over a link rejects the calls itself and sends no cancel for them. + if (state.kind === "bound") state.target.unhold(this); + this.end(this.#released()); + } + + /** + * Ends the reference without releasing it from its object (it ended): watches end, calls (running and parked) + * reject with `error`, `ref.state` is `ended` with it. + */ + end(error: ChordError): void { + // Kept by `ref.state`: its stack is made text now, so the error holds none of the frames' objects. + void error.stack; + const ended: Ended = { kind: "ended", error }; + this.#state = { kind: "released", state: ended }; + for (const at of this.#keys) at.release(); + this.#calls?.abort(error); + this.#calls = undefined; + this.#parked?.reject(error); + this.#parked = undefined; + for (const watch of this.#watches) watch.end({ kind: "ended" }); + this.#refState?.end(ended); + // Each removes itself; promise reactions run later. + for (const waiter of this.#waiters) waiter.end(error); + this.#ended?.resolve(); + this.#ended = undefined; + } + + #call(method: string, args: unknown, context: Context): Promise { + const state = this.refState; + switch (state.kind) { + case "live": + // Calls queued before it went live are still being sent: a new one never overtakes them. + if (this.#parked?.waiting) return this.#parked.park(method, args, context); + return this.#send(method, args, context); + case "loading": + this.#parked ??= new ParkedCalls(this.#parkTimeout, this.label); + return this.#parked.park(method, args, context); + case "failed": + throw state.cause.error; + case "ended": + throw state.error; + } + } + + /** Calls the live object; the call is aborted when the reference ends. */ + #send(method: string, args: unknown, context: Context): Promise { + // Only a bound reference is live. + const { target } = this.#state as Extract; + this.#calls ??= new AbortController(); + const linked = withLinkedAbortSignal(this.#calls.signal, context); + return target.call(method, args, linked.context, this.caller).finally(linked.release); + } + + /** + * Settles once `decide` does for the reference's state, checked now and at every change; rejects when the reference + * ends (with its error, `released` when it is released already) or `context` aborts. + */ + #until(decide: (state: RefState) => Decision, context: Context): Promise { + if (this.#state.kind === "released") return Promise.reject(this.#state.state.error); + const { promise, resolve, reject } = Promise.withResolvers(); + const signal = context.abortSignal; + const settle = (done: () => void): void => { + this.#waiters.delete(waiter); + signal?.removeEventListener("abort", abort); + done(); + }; + const waiter: Waiter = { + check: () => { + const decision = decide(this.refState); + if (decision.kind === "done") settle(resolve); + else if (decision.kind === "failed") settle(() => reject(decision.error)); + }, + end: (error) => settle(() => reject(error)), + }; + const abort = (): void => settle(() => reject(signal?.reason)); + this.#waiters.add(waiter); + signal?.addEventListener("abort", abort, { once: true }); + if (signal?.aborted) abort(); + else waiter.check(); + return promise; + } + + #watchableState(): ReferenceState { + if (this.#refState !== undefined) return this.#refState; + const state = this.#state; + // A released reference's state never changes: its watches end at once, with no listener to report. Bound, not a + // closure: `read` would share the closure's scope and keep the batch after release. + const [report, assertOutside] = + state.kind === "released" + ? [IGNORE, IGNORE] + : [state.batch.reportListener, state.batch.assertOutside.bind(state.batch, "watch")]; + this.#refState = new ReferenceState(() => this.refState, report, assertOutside); + return this.#refState; + } + + #variant(): string | undefined { + const state = this.#state; + if (state.kind === "released") throw state.state.error; + return state.kind === "bound" ? state.target.variant : undefined; + } + + /** The error of this reference's release by its holder (`released`, no reason). */ + #released(): ChordError { + return new ChordError({ code: "released", message: `${this.#name}: released` }); + } +} + +/** A state member of a reference: the provider's member, followed through this reference. */ +class ReferenceMember { + readonly #binding: Binding; + readonly #member: string; + + constructor(binding: Binding, member: string) { + this.#binding = binding; + this.#member = member; + } + + get value(): JsonValue { + return this.source.value; + } + + watch(context: Context): Promise> { + return this.#binding.watch(this.#member, context); + } + + /** The provider's member: a source or a constant. */ + get source(): DeriveSource { + return this.#binding.member(this.#member); + } +} + +/** A keyed member of a reference (slice 6a §12): reads by key behave as calls (they wait while loading). */ +class ReferenceKeyed { + readonly #binding: Binding; + readonly #member: string; + + constructor(binding: Binding, member: string) { + this.#binding = binding; + this.#member = member; + } + + /** The provider's slot: the ultimate provider's, through re-exposures (rule 22). */ + get slot(): KeyedSlot { + return this.#binding.keyedSlot(this.#member); + } + + /** This member as another object re-exposing it has it: its keys held through this reference. */ + get keyed(): KeyedBinding { + return { kind: "reexposed", binding: this.#binding, member: this.#member }; + } + + async get(key: string, context: Context): Promise { + await this.#binding.hydrated(context); + const slot = this.slot; + return slot.get(key, ownAccess(slot), context); + } + + /** + * A watch of the document at `key` across the object's attempts (it holds the key through the object), `undefined` + * when absent; it ends with this reference. + */ + async watch(key: string, context: Context): Promise | undefined> { + await this.#binding.hydrated(context); + const slot = this.slot; + if ((await slot.get(key, ownAccess(slot), context)) === undefined) return undefined; + const held = await this.#binding.pinKey(this.#member, key, context); + // Retired between the read and now: absent. + if (slot.current(slot.key(key)) === undefined) { + held.release(); + return undefined; + } + let watch: Watch; + try { + watch = held.source.openWatch(context); + } catch (error) { + held.release(); + throw error; + } + void watch.closed.then(held.release); + return this.#binding.follow(watch); + } + + at(key: string): Watchable { + // The pinned key's values are the member's documents. + return this.#binding.at(this.#member, key) as unknown as Watchable; + } +} + +/** A consumer reads its provider's keyed documents as the provider's own code may: current version, own token. */ +function ownAccess(slot: KeyedSlot): { readonly token: SealToken; readonly version: number } { + return { token: slot.owner.own, version: slot.definition.current.version }; +} + +/** Where a `ref.member.at(key)` stands: not held yet, held on the object's keyed member, or released for good. */ +type AtState = + | { readonly kind: "unpinned" } + | { readonly kind: "pinned"; readonly source: DeriveSource; readonly release: () => void } + | { readonly kind: "released" }; + +/** + * `ref.member.at(key)`: the key's source across the object's attempts once the reference is bound and the key held + * (`pin`), until the reference ends (`release`, exactly once): then it drops the source (a retained reference keeps no + * provider). + */ +class KeyedMemberAt { + readonly #binding: Binding; + readonly member: string; + readonly key: string; + #state: AtState = { kind: "unpinned" }; + + constructor(binding: Binding, member: string, key: string) { + this.#binding = binding; + this.member = member; + this.key = key; + } + + /** The key's source; `not_ready` before it is held, `released` after. */ + get source(): DeriveSource { + const state = this.#state; + if (state.kind === "pinned") return state.source; + const what = `${this.member}.at(${this.key})`; + if (state.kind === "released") throw new ChordError({ code: "released", message: `${what}: released` }); + throw new ChordError({ code: "not_ready", message: `${what}: not loaded yet` }); + } + + get value(): JsonValue { + return this.source.value; + } + + async watch(context: Context): Promise> { + return this.#binding.follow(this.source.openWatch(context)); + } + + /** Holds the key through `binding` (once), loaded; nothing once released (a hold that lands then is released). */ + async pin(binding: Binding, context: Context): Promise { + if (this.#state.kind !== "unpinned") return; + const held = await binding.pinKey(this.member, this.key, context); + if (this.#state.kind !== "unpinned") { + held.release(); + return; + } + this.#state = { kind: "pinned", source: held.source, release: held.release }; + } + + /** Releases the pin and drops the source (once). */ + release(): void { + const state = this.#state; + this.#state = { kind: "released" }; + if (state.kind === "pinned") state.release(); + } +} + +/** + * What a keyed member's value is (slice 6a §12): a keyed document handle's slot, or a reference's keyed member + * (re-exposed); `undefined` for anything else. + */ +export function resolveKeyed(value: unknown): KeyedBinding | undefined { + if (value instanceof Keyed) return { kind: "own", slot: value.slot }; + if (value instanceof ReferenceKeyed) return value.keyed; + return undefined; +} + +/** + * The watchable Chord follows behind a state member or a derive source (Q14): a source or a constant itself, a + * singleton handle's slot, a reference member's provider member, a keyed member's pinned key (a key source itself for + * `handle.at(key)`); `undefined` for anything else, a native watchable included. + */ +export function resolveWatchable(value: unknown): DeriveSource | undefined { + if (value instanceof Singleton) return value.slot; + if (value instanceof ReferenceMember) return value.source; + if (value instanceof KeyedMemberAt) return value.source; + if (value instanceof Source || value instanceof Constant) return value; + return undefined; +} diff --git a/packages/chord/src/next/host/revocation.ts b/packages/chord/src/next/host/revocation.ts new file mode 100644 index 000000000..6f77aace1 --- /dev/null +++ b/packages/chord/src/next/host/revocation.ts @@ -0,0 +1,35 @@ +// Revocation keys (§4, R10, R11): what `revokedBy` says revokes a principal, as canonical JSON, computed once per +// principal object. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Principal } from "../data/index.ts"; +import { canonicalJson } from "../data/internal.ts"; + +/** What revokes a principal: the app's typed values, the same function on every host. */ +export type RevokedBy

= (principal: P) => readonly R[]; + +/** A host's revocation keys: each principal's, cached weakly by the principal object. */ +export class RevocationKeys { + readonly #revokedBy: RevokedBy | undefined; + readonly #keys = new WeakMap>(); + + constructor(revokedBy: RevokedBy | undefined) { + this.#revokedBy = revokedBy; + } + + /** + * Whether `key` (a revocation's canonical JSON) is one of `principal`'s revocation keys: the canonical JSON of each + * value `revokedBy` returns for it, computed on its first use. + */ + revokes(principal: Principal, key: string, context: Context): boolean { + const revokedBy = this.#revokedBy; + if (revokedBy === undefined) return false; + let keys = this.#keys.get(principal); + if (keys === undefined) { + keys = new Set(revokedBy(principal).map((revocation) => canonicalJson(revocation, context))); + this.#keys.set(principal, keys); + } + return keys.has(key); + } +} diff --git a/packages/chord/src/next/host/route.ts b/packages/chord/src/next/host/route.ts new file mode 100644 index 000000000..e4f24d337 --- /dev/null +++ b/packages/chord/src/next/host/route.ts @@ -0,0 +1,160 @@ +// Routes (§3): how a host reaches hosts of one kind. `authorize` admits each hold routed over the route; `resolve` +// makes the link of one address, at most once at a time, again after the link drops while holds exist, with a +// backoff per address. + +import type { Context } from "../../context/index.ts"; +import type { Address, HostKind, KindShape } from "../contracts/index.ts"; +import { kindParams, type ParamsData } from "../contracts/internal.ts"; +import { ChordError, type Principal } from "../data/index.ts"; +import { checkAddress } from "../data/internal.ts"; +import type { Codec, Link, PartLimits } from "../link/index.ts"; +import { type HostLink, Path, type Upstream } from "../remote/internal.ts"; +import { Backoff } from "./backoff.ts"; +import type { AnyHost } from "./host.ts"; + +/** What `host.route` takes for a kind. */ +export type RouteOptions

= { + /** Whether `principal` may reach `address`; runs for every hold routed over the route, before anything starts. */ + authorize(principal: P, address: Address): boolean; + /** + * The link to `address`: `principal` is who the host at the other end is, `trusted` skips the message check on it, + * `limits` bound its parts (as `accept`'s). `context` is the route's, aborted when this host closes. Runs at most + * once at a time per address. + */ + resolve( + address: Address, + context: Context, + ): Promise<{ + readonly link: Link; + readonly principal: P; + readonly trusted?: boolean; + readonly codec?: Codec; + readonly limits?: PartLimits; + }>; +}; + +/** One address of a route: its path (link and holders) and whether its `resolve` runs. */ +type Routed = { readonly address: Address; readonly path: Path; resolving: boolean }; + +/** + * A host's route for one kind. An address is kept while it has holders, a link or a running `resolve`: a resolve whose + * holds were all released meanwhile finishes, and its link is kept until it closes. + */ +export class Route { + readonly kind: string; + readonly #params: ParamsData; + readonly #options: RouteOptions; + readonly #host: AnyHost; + /** The route's context, aborted when the host closes: every `resolve` and its links' work run under it. */ + readonly #context: Context; + readonly #addresses = new Map(); + /** The addresses' links, which tell their address when they open or close. */ + readonly #links = new Map(); + /** Per address key, separate from objects' identities. */ + readonly #backoff = new Backoff((key) => this.#due(key)); + + constructor(kind: HostKind, options: RouteOptions, host: AnyHost, context: Context) { + this.kind = kind.kind; + this.#params = kindParams(kind); + this.#options = options; + this.#host = host; + this.#context = context; + } + + /** + * The path to the address with `params`, for a hold acting as `principal`: the params checked against the kind + * (`invalid/placement`), then `authorize` (`forbidden` when it refuses; a throw is the hold's failure). + */ + path(params: unknown, principal: Principal | undefined, context: Context): Path { + const address = checkAddress(this.kind, this.#params, params, context); + if (principal === undefined || !this.#options.authorize(principal, address)) { + throw new ChordError({ code: "forbidden", message: `${address.key}: not authorized` }); + } + let routed = this.#addresses.get(address.key); + if (routed === undefined) { + const made: Routed = { address, path: new Path(() => this.#check(made)), resolving: false }; + routed = made; + this.#addresses.set(address.key, made); + } + return routed.path; + } + + /** One of this route's links completed its handshake: its address's backoff is over. */ + opened(link: HostLink): void { + const routed = this.#links.get(link); + if (routed !== undefined) this.#backoff.delete(routed.address.key); + } + + /** One of this route's links closed: the address waits its backoff, then resolves again while it has holders. */ + closed(link: HostLink): boolean { + const routed = this.#links.get(link); + if (routed === undefined) return false; + this.#links.delete(link); + if (!this.#host.closing) this.#backoff.failed(routed.address.key); + routed.path.disconnected(link); + return true; + } + + /** The host closes: no address resolves again. Returns every holder of every path, for the host to end. */ + close(): Upstream[] { + this.#backoff.clear(); + return [...this.#addresses.values()].flatMap(({ path }) => path.holders()); + } + + /** + * An address's holders or link changed: it resolves when it has holders, no link, no `resolve` running and no + * backoff window; it is forgotten when it has neither holders, a link nor a running `resolve`. + */ + #check(routed: Routed): void { + const { path, address } = routed; + if (path.link !== undefined || routed.resolving || this.#host.closing) return; + if (!path.held) { + this.#addresses.delete(address.key); + return; + } + if (!this.#backoff.waiting(address.key)) void this.#resolve(routed); + } + + /** A backoff window ended. */ + #due(key: string): void { + const routed = this.#addresses.get(key); + if (routed === undefined) this.#backoff.delete(key); + else this.#check(routed); + } + + /** + * Runs `resolve` once for the address. A rejection, or a link closed already, starts the address's backoff; a link + * returned after the host closed is closed at once. + */ + async #resolve(routed: Routed): Promise { + const key = routed.address.key; + routed.resolving = true; + let resolved: Awaited["resolve"]>>; + try { + resolved = await this.#options.resolve(routed.address, this.#context); + } catch (error) { + routed.resolving = false; + if (this.#host.closing) return; + this.#host.report({ kind: "resolve_failed", address: key, error }); + this.#backoff.failed(key); + this.#check(routed); + return; + } + routed.resolving = false; + if (this.#host.closing) { + resolved.link.close(); + return; + } + const { link: made, ...options } = resolved; + const link = this.#host.routeLink(made, options, this.#context); + if (!link.open) { + this.#backoff.failed(key); + this.#check(routed); + return; + } + this.#links.set(link, routed); + // A peer that greeted first had its `hello` handled while the link was made, before it was registered here. + if (link.greeted) this.#backoff.delete(key); + routed.path.connected(link); + } +} diff --git a/packages/chord/src/next/host/routines.ts b/packages/chord/src/next/host/routines.ts new file mode 100644 index 000000000..824e0ee38 --- /dev/null +++ b/packages/chord/src/next/host/routines.ts @@ -0,0 +1,432 @@ +// Routines (§14): the standing behaviour an object or a facet declares, `self.on(source, fn)` and `self.onLive(fn)`. +// They start once their owner went live and stop when it ends (their ctx aborted, the drain waits for running calls); +// each call gets its own ctx; a call that throws is reported and called again after a backoff. + +import type { Context } from "../../context/index.ts"; +import { type Op, PendingOps } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { + NativeWatch, + type NativeWatchable, + type NativeWatchCall, + type Watchable, + type WatchCall, + type WatchEnd, +} from "../../watch.ts"; +import type { Principal } from "../data/index.ts"; +import { Source } from "../state/internal.ts"; +import { nextDelay } from "./backoff.ts"; +import { CallContext, type Caller, type Ctx } from "./ctx.ts"; +import type { AnyHost, Position } from "./host.ts"; +import { resolveWatchable } from "./reference.ts"; +import { sleep } from "./sleep.ts"; + +/** + * Where routines run: their owner's label (for reports), its host, whom their calls act for, and the context of their + * calls, aborted when the owner ends. `admit` adds a running call to the owner's drain. + */ +export type RoutineOwner = { + readonly label: string; + readonly host: AnyHost; + readonly caller: Caller; + readonly context: Context; + /** + * Adds a running call to the drain; `cleanup` (its bookkeeping) runs once, when it settles or when the attempt ends + * with it still running. + */ + admit(running: Promise, cleanup: () => void): void; +}; + +/** + * A routine's handler of a JSON source: the value call (the value, the ops since the previous call, whether the + * source's document was retired since then) and the call's ctx. Failures are not calls of a handler (design §11): the + * routine waits for the next value. + */ +export type OnHandler = (call: WatchValueCall, ctx: C) => void | Promise; + +/** A JSON watch's value call: what an `on` handler gets. */ +export type WatchValueCall = Extract, { readonly kind: "value" }>; + +/** A native watch's value call. */ +export type NativeWatchValueCall = Extract, { readonly kind: "value" }>; + +/** A routine's handler of a native source: the value call (the value) and the call's ctx. */ +export type OnNativeHandler = (call: NativeWatchValueCall, ctx: C) => void | Promise; + +/** An `onLive` routine: called once its owner is live. */ +export type OnLiveHandler = (ctx: C) => void | Promise; + +/** What the declarations of an owner (`on`, `onLive`) take; part of the factory's `self` and the facet's. */ +export interface RoutineDeclarations

{ + /** + * A routine over `source`: `fn` gets its current value first (ops `[["r", value]]`), then one call per change, + * never overlapping: changes during a call fold into one next call (the newest value, the ops since the previous + * call, `retired` when any of them was a retirement). After a call that threw, it is called again with the whole + * newest value (`retired` again if the call that threw had it) after a backoff (1 s doubling to 60 s), or at once on + * a change during the backoff. Ends with its source. A JSON watchable is one whose watch is Chord's shared `Watch` + * (every Chord and Durable source). + */ + on(source: Watchable, fn: OnHandler>): void; + /** + * A routine over a native source (a registry value, `ref.state`): one whose watch is Chord's shared `NativeWatch`. + * As above without ops: after a busy call, one call with the newest value. + */ + on(source: NativeWatchable, fn: OnNativeHandler>): void; + /** + * A routine called once its owner is live. Returning ends it; a throw is reported and restarts it after a backoff + * (1 s doubling to 60 s, reset after a run of a minute); the owner's end aborts its ctx. + */ + onLive(fn: OnLiveHandler>): void; +} + +/** A declared routine: its kind and position in reports (`{ kind: "on", position: 2 }`), its source and its handler. */ +type Declaration = + | { + readonly kind: "on"; + readonly at: Position; + readonly source: Watchable | NativeWatchable; + readonly fn: (...args: never[]) => void | Promise; + } + | { readonly kind: "onLive"; readonly at: Position; readonly fn: OnLiveHandler }; + +/** How one routine call ended. */ +type Called = { readonly kind: "done" } | { readonly kind: "failed"; readonly error: unknown }; + +const DONE: Called = { kind: "done" }; + +/** One run of an `onLive` that lasted this long resets its backoff. */ +const LONG_RUN = 60_000; + +/** An owner's routines: declared while its factory (or setup) runs, started once when it went live. */ +export class Routines { + readonly #declared: Declaration[] = []; + #ons = 0; + #lives = 0; + + /** Declares `on(source, fn)`; whether the source is JSON or native is decided by its watch (`OnRoutine`). */ + on(source: Watchable | NativeWatchable, fn: (...args: never[]) => void | Promise): void { + this.#declared.push({ kind: "on", at: { kind: "on", position: ++this.#ons }, source, fn }); + } + + /** Declares `onLive(fn)`. */ + onLive(fn: OnLiveHandler): void { + this.#declared.push({ kind: "onLive", at: { kind: "onLive", position: ++this.#lives }, fn }); + } + + /** + * Starts every routine for `owner`, which went live; they stop when its context is aborted. A routine whose + * synchronous start ends the owner (an `onLive` releasing its last reference) starts none of the later ones. + */ + start(owner: RoutineOwner): void { + for (const declaration of this.#declared) { + if (owner.context.abortSignal?.aborted) return; + if (declaration.kind === "on") void new OnRoutine(owner, declaration).start(); + else void live(owner, declaration); + } + } +} + +/** + * Runs `onLive`'s handler until it returns or its owner ends; a throw is reported and the handler restarts after the + * backoff. + */ +async function live(owner: RoutineOwner, { at, fn }: Extract): Promise { + let delay: number | undefined; + while (!owner.context.abortSignal?.aborted) { + const started = Date.now(); + const called = await call(owner, fn); + if (called.kind === "done" || owner.context.abortSignal?.aborted) return; + failed(owner, at, called.error); + delay = nextDelay(Date.now() - started >= LONG_RUN ? undefined : delay); + try { + await sleep(delay, owner.context); + } catch { + return; + } + } +} + +/** + * Runs one routine call under its own ctx (its `ctx.get` references released when it settles), admitted to the owner's + * drain before the handler runs: a handler may end its own owner before its first await. Resolves with how it ended; + * never rejects. + */ +function call(owner: RoutineOwner, invoke: (ctx: Ctx) => void | Promise): Promise { + const ctx = new CallContext(owner.context, owner.caller, owner.host); + const settled = Promise.withResolvers(); + owner.admit(settled.promise, () => ctx.settle()); + const finish = (called: Called): void => { + ctx.settle(); + settled.resolve(called); + }; + // Plugin code: a synchronous throw becomes the rejection. + (async () => invoke(ctx))().then( + () => finish(DONE), + (error: unknown) => finish({ kind: "failed", error }), + ); + return settled.promise; +} + +/** Reports a failed routine call or source. */ +function failed(owner: RoutineOwner, routine: Position, error: unknown): void { + owner.host.report({ kind: "routine_failed", object: owner.label, routine, error }); +} + +/** + * Where an `on` routine stands. `starting`: taking its watch; `idle`: waiting for a change; `calling`: a call runs + * (changes meanwhile wait, folded, for the next); `settling`: a call succeeded while changes arrived, and the next one + * waits for one macrotask, so every value its watch still buffers (a native watch replays them one per microtask) is + * taken first; `backoff`: the last call threw and its timer calls again, unless a change during the backoff calls + * first; `ended`: its watch ended (the owner ended, or the source ended or failed). + */ +type OnState = + | { readonly kind: "starting" | "idle" | "calling" | "ended" } + | { readonly kind: "settling" | "backoff"; readonly timer: ReturnType }; + +/** + * What the next call's ops are relative to: `none` before the first call and after a throw (the handler may not have + * applied what it got: the next call gets the whole value), `last` after a call that succeeded. + */ +type Basis = "none" | "last"; + +/** + * One `on` routine. Its watch's listener never throws (a throwing listener would end the watch) and never waits: it + * keeps the newest frame, folding JSON ops, and the routine makes one call at a time from it, the first call and the + * retries after a throw through the same pending frame. + */ +class OnRoutine { + readonly #owner: RoutineOwner; + readonly #declaration: Extract; + #state: OnState = { kind: "starting" }; + /** A native source's watch yields values without ops; decided by its watch's class. */ + #native = false; + /** The newest value no call got yet. */ + #pending: { readonly value: unknown } | undefined; + /** The ops of the frames since the last call (JSON sources), folded. */ + readonly #ops = new PendingOps(); + /** One of the frames since the last call was a retirement (JSON sources). */ + #retired = false; + #basis: Basis = "none"; + /** The value of the last call: a retry after a throw calls with it. */ + #last: unknown; + /** The last call reported a retirement: a retry after a throw reports it again. */ + #lastRetired = false; + #delay: number | undefined; + /** The source failed after its last value: no call until its next value (a retry waits too). */ + #failed = false; + + constructor(owner: RoutineOwner, declaration: Extract) { + this.#owner = owner; + this.#declaration = declaration; + } + + /** + * Takes the watch and calls with its value. On a failed Chord source it waits for its first good value instead (a + * source that never had one is followed until it has). A source that cannot be watched otherwise is reported; an + * owner or source that ended meanwhile ends the routine. Never rejects. + */ + async start(): Promise { + const { source, at } = this.#declaration; + let failing = this.#failing(); + while (failing?.kind === "no_value") { + if (!(await this.#firstValue(failing.source))) { + this.#state = { kind: "ended" }; + return; + } + // Asked again: a failure may have followed the first value before this resumed (one receive turn). + failing = this.#failing(); + } + let watch: Awaited>; + try { + watch = await source.watch(this.#owner.context); + } catch (error) { + this.#state = { kind: "ended" }; + if (!this.#aborted()) failed(this.#owner, at, error); + return; + } + this.#native = watch instanceof NativeWatch; + void watch.closed.then((end) => this.#end(end)); + try { + // Both listeners return at once. + watch.start((call: WatchCall | NativeWatchCall) => { + if (call.kind === "value") this.#frame(call); + else this.#failure(); + return Promise.resolve(); + }); + } catch { + // The watch ended while it was taken (the owner or the source ended): `closed` says how. + return; + } + // A watch opened on a failed source holds its last good value, a stale one: its first call is the failure. + if (failing?.kind === "failed") { + this.#state = { kind: "idle" }; + return; + } + this.#pending = { value: watch.value }; + this.#next(); + } + + /** + * Whether the source is a Chord source failed now (`failed`), or one with no value yet (`no_value`, with the source to + * follow); `undefined` otherwise (natives and Durable's views tell their failures through their watches). + */ + #failing(): { readonly kind: "failed" } | { readonly kind: "no_value"; readonly source: Source } | undefined { + let source: unknown; + try { + source = resolveWatchable(this.#declaration.source); + } catch { + // Not bound or ended: the watch says so. + return undefined; + } + if (!(source instanceof Source)) return undefined; + let start: ReturnType; + try { + start = source.start(); + } catch { + return undefined; + } + if (start.kind === "value") return undefined; + return start.last === undefined ? { kind: "no_value", source } : { kind: "failed" }; + } + + /** + * Follows `source`, failed with no value, until its first good value (`true`), or until it or the owner ends + * (`false`): a subscriber of its own, since a watch has nothing to start at. + */ + #firstValue(source: Source): Promise { + const signal = this.#owner.context.abortSignal; + return new Promise((resolve) => { + const done = (got: boolean): void => { + source.unsubscribe(subscriber); + signal?.removeEventListener("abort", aborted); + resolve(got); + }; + const subscriber = { + advance: (call: WatchCall) => { + if (call.kind === "value") done(true); + }, + end: () => done(false), + }; + const aborted = (): void => done(false); + signal?.addEventListener("abort", aborted, { once: true }); + if (signal?.aborted) aborted(); + else source.subscribe(subscriber); + }); + } + + /** + * The source failed: what was pending goes (a value a failure followed is stale), a pending retry or settle is + * dropped, and the next call waits for the next value, as a whole value; a running call runs on. + */ + #failure(): void { + const state = this.#state; + if (state.kind === "ended") return; + this.#failed = true; + this.#pending = undefined; + this.#ops.take(); + this.#basis = "none"; + if (state.kind === "backoff" || state.kind === "settling") { + clearTimeout(state.timer); + this.#state = { kind: "idle" }; + } + } + + /** + * A frame of the watch: kept as the newest; it is called at once when nothing runs or waits, during a backoff too + * (the backoff ends); while a call runs or settles it waits for the next call. + */ + #frame(call: WatchValueCall | NativeWatchValueCall): void { + const state = this.#state; + if (state.kind === "ended") return; + this.#failed = false; + this.#pending = { value: call.value }; + if (!this.#native) { + // A JSON watch's call, as its class decided. + const { ops, retired } = call as WatchValueCall; + this.#ops.push(ops); + if (retired) this.#retired = true; + } + if (state.kind === "backoff") clearTimeout(state.timer); + if (state.kind === "backoff" || state.kind === "idle") this.#next(); + } + + /** Makes the next call from the pending frame, if there is one and the owner still runs; otherwise idles. */ + #next(): void { + const pending = this.#pending; + if (pending === undefined || this.#aborted()) { + this.#state = { kind: "idle" }; + return; + } + this.#pending = undefined; + const { value } = pending; + this.#state = { kind: "calling" }; + this.#last = value; + void call(this.#owner, this.#invocation(value)).then((called) => this.#called(called)); + } + + /** + * The handler's call with `value`: a native one with the value alone, a JSON one with the folded ops or the whole + * value, and whether a retirement came since the last call. + */ + #invocation(value: unknown): (ctx: Ctx) => void | Promise { + const { fn } = this.#declaration; + // The overloads typed `fn` for its source's kind, which the watch decided. + if (this.#native) return (ctx) => (fn as OnNativeHandler)({ kind: "value", value }, ctx); + const taken = this.#ops.take(); + // A JSON source's values are JSON. + const ops: readonly Op[] = + this.#basis === "none" || taken.kind === "replace" ? [["r", value as JsonValue]] : taken.ops; + const retired = this.#retired; + this.#retired = false; + this.#lastRetired = retired; + return (ctx) => (fn as OnHandler)({ kind: "value", value, ops, retired }, ctx); + } + + /** + * A call settled. After a success the next frame's ops build on it, and changes that arrived during the call are + * called after one macrotask (`settling`). After a throw (reported) the backoff always starts, also when changes + * arrived during the call (a handler that changes its own source and throws would feed itself otherwise); its timer + * calls with the newest value, or the last call's, as a whole value. + */ + #called(called: Called): void { + if (this.#state.kind === "ended") return; + if (called.kind === "done") { + this.#basis = "last"; + this.#delay = undefined; + if (this.#pending === undefined) this.#state = { kind: "idle" }; + else this.#state = { kind: "settling", timer: setTimeout(() => this.#next(), 0) }; + return; + } + if (this.#aborted()) return; + failed(this.#owner, this.#declaration.at, called.error); + this.#basis = "none"; + // The handler may not have applied the retirement either. + if (this.#lastRetired) this.#retired = true; + this.#delay = nextDelay(this.#delay); + // The source failed meanwhile: the retry waits for its next value, never the stale last one. + if (this.#failed) { + this.#state = { kind: "idle" }; + return; + } + const timer = setTimeout(() => { + this.#pending ??= { value: this.#last }; + this.#next(); + }, this.#delay); + this.#state = { kind: "backoff", timer }; + } + + /** The watch ended: the owner ended (aborted), or the source ended (no report) or failed (reported once). */ + #end(end: WatchEnd): void { + const state = this.#state; + this.#state = { kind: "ended" }; + if (state.kind === "settling" || state.kind === "backoff") clearTimeout(state.timer); + this.#pending = undefined; + this.#ops.take(); + this.#retired = false; + if (end.kind === "failed" && !this.#aborted()) failed(this.#owner, this.#declaration.at, end.error); + } + + #aborted(): boolean { + return this.#owner.context.abortSignal?.aborted === true; + } +} diff --git a/packages/chord/src/next/host/sleep.ts b/packages/chord/src/next/host/sleep.ts new file mode 100644 index 000000000..56cf18407 --- /dev/null +++ b/packages/chord/src/next/host/sleep.ts @@ -0,0 +1,43 @@ +// Waiting helpers for routines, jobs and handlers (§14): `sleep(ms, ctx)` and `untilAborted(ctx)`, which end with +// their context, leaving no timer or listener behind. Portable: timers and `AbortSignal` only. + +import type { Context } from "../../context/index.ts"; + +/** The longest delay one timer takes (a longer one fires at once). */ +const MAX_TIMER = 2 ** 31 - 1; + +/** + * Resolves after `ms` milliseconds; rejects with the abort reason once `context` is aborted, clearing its timer and + * listener. Delays above the timer limit (2^31-1 ms) are chained timers. + */ +export function sleep(ms: number, context: Context): Promise { + const signal = context.abortSignal; + if (signal?.aborted) return Promise.reject(signal.reason); + const { promise, resolve, reject } = Promise.withResolvers(); + let timer: ReturnType | undefined; + const abort = (): void => { + clearTimeout(timer); + reject(signal?.reason); + }; + const wait = (left: number): void => { + timer = setTimeout( + () => { + if (left > MAX_TIMER) return wait(left - MAX_TIMER); + signal?.removeEventListener("abort", abort); + resolve(); + }, + Math.min(left, MAX_TIMER), + ); + }; + signal?.addEventListener("abort", abort, { once: true }); + wait(ms); + return promise; +} + +/** Resolves once `context` is aborted (never without an abort signal); leaves no listener behind. */ +export function untilAborted(context: Context): Promise { + const signal = context.abortSignal; + if (signal === undefined) return new Promise(() => {}); + if (signal.aborted) return Promise.resolve(); + return new Promise((resolve) => signal.addEventListener("abort", () => resolve(), { once: true })); +} diff --git a/packages/chord/src/next/host/uuid.ts b/packages/chord/src/next/host/uuid.ts new file mode 100644 index 000000000..1112f4bb9 --- /dev/null +++ b/packages/chord/src/next/host/uuid.ts @@ -0,0 +1,38 @@ +// UUIDv7 (§3, code copied from pi-ai's `uuidv7`): time-ordered ids, minted for host incarnations. + +const MAX_SEQUENCE = (1n << 41n) - 1n; + +/** The last timestamp used: a clock going backwards keeps the ids ordered. */ +let lastTimestamp = -1; + +/** A random start, then incremented per id, so ids within one millisecond stay ordered and unique. */ +let sequence: bigint | undefined; + +/** A time-ordered UUIDv7. */ +export function uuidv7(): string { + const timestamp = Math.max(Date.now(), lastTimestamp); + lastTimestamp = timestamp; + const bytes = new Uint8Array(16); + globalThis.crypto.getRandomValues(bytes); + if (sequence === undefined) { + sequence = + (BigInt(bytes[1]!) << 32n) | + (BigInt(bytes[2]!) << 24n) | + (BigInt(bytes[3]!) << 16n) | + (BigInt(bytes[4]!) << 8n) | + BigInt(bytes[5]!); + } else { + if (sequence === MAX_SEQUENCE) throw new RangeError("UUIDv7 generator sequence exhausted"); + sequence++; + } + const time = BigInt(timestamp); + for (let index = 5; index >= 0; index--) bytes[index] = Number(time >> BigInt((5 - index) * 8)) & 0xff; + bytes[6] = 0x70 | Number((sequence >> 37n) & 0x0fn); + bytes[7] = Number((sequence >> 29n) & 0xffn); + bytes[8] = 0x80 | Number((sequence >> 23n) & 0x3fn); + bytes[9] = Number((sequence >> 15n) & 0xffn); + bytes[10] = Number((sequence >> 7n) & 0xffn); + bytes[11] = Number((sequence & 0x7fn) << 1n) | (bytes[11]! & 0x01); + const hex = Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")); + return `${hex.slice(0, 4).join("")}-${hex.slice(4, 6).join("")}-${hex.slice(6, 8).join("")}-${hex.slice(8, 10).join("")}-${hex.slice(10).join("")}`; +} diff --git a/packages/chord/src/next/jobs/README.md b/packages/chord/src/next/jobs/README.md new file mode 100644 index 000000000..3fba3ca35 --- /dev/null +++ b/packages/chord/src/next/jobs/README.md @@ -0,0 +1,112 @@ +# jobs + +Jobs in memory (§15, slice 5): owned work a call starts that outlives the call, in Durable's sandwich: intent (a +transaction), run (any effect), finish (a transaction on the current code). Persistence is slice 6, reload rules slice +7. + +## For apps (`index.ts`) + +```ts +import { defineJob } from "./index.ts"; + +const ShellState = defineState("chat.shell.state", { schema: obj({ output: str() }), version: 1, initial: () => ({ output: "" }) }); +const ShellJob = defineJob("chat.shell", { + version: 1, + args: obj({ command: str(), author: Author }), // closed, JSON only: checked at every start + result: obj({ exit: int(), output: str() }), // not validated + state: ShellState, // optional: one document per run + concurrency: 1, // optional: running plus finishing runs per object +}); + +host.serve(ChatStore, (_, self) => { + const entries = self.state(Entries); + const shell = self.job(ShellJob, { + run: async ({ command }, ctx) => { // outside transactions; ctx.jobId, ctx.state + const out = new OutputBuffer({ maxBytes: 64 * 1024, maxLines: 2_000, retain: "tail" }); + const write = async () => { const text = out.snapshot().text; await ctx.state.change((d) => { d.output = text; }, ctx); return text.length; }; + const progress = new Progress(write, report, 100); // the author throttles state changes + try { return { exit: await runProcess(command, ctx.abortSignal, (chunk) => { out.push(chunk); progress.mark(); }), output: out.snapshot().text }; } + finally { await progress.stop(); await write(); } // the last output reaches outcome.state, after an abort too + }, + finish: async (outcome, tx) => { // one transaction with the cleanup: only `await tx.*` + (await tx.draft(entries)).list.push(entryOf(outcome)); + tx.start(notify, { text: outcome.args.command }); // chaining: atomic with this outcome + }, + }); + return { + shells: shell.running, // a JSON watchable: views filter it + runShell: async ({ command, author, requestId }, ctx) => + (await shell.start({ command, author }, ctx, { key: requestId })).id, + abortShell: async ({ id }) => { shell.abort(id); return null; }, + }; +}); +``` + +- `defineJob(name, { version, args, result, state?, concurrency?, accepts?, persist? })`: `args` closed (every object + `additionalProperties: false`) and without `bytes()` (`running` carries them); `state` a singleton definition; + `concurrency` a positive integer; `accepts` `{ [older version]: { args, result } }`, both functions, checked now and + used from slice 7 on; `persist: { resume: "rerun" | "abandon" }` is refused before slice 6. Declarations fail with + an `Error` naming the path (`chat.shell.accepts[2]: not an older version`). +- `self.job(Job, { run, finish? })` in an object's factory (not in a facet), once per job name: a `JobHandle`. + - `start(args, ctx, { key? })`: one transaction on the object's line; resolves with a `Run { id, outcome(ctx) }` + after the commit. Only a live object takes starts (`not_ready` before, `released` once it ended). + - `tx.start(job, args, { key? })` inside `self.tx` (and in `finish`): the same, with the transaction's other writes; + its `Run` at once. A discarded transaction's `Run` rejects `released`. + - In order: the args checked (`invalid/args`; one compiled check per definition); a key of an unsettled run (one + started earlier in the same transaction too, a blocked one too) returns that run, args not compared; at the + `concurrency` limit (running and finishing runs plus earlier starts in the transaction) `unavailable/busy`; then + the run (id uuidv7) and its state document (`initial()`). The run whose `finish` the transaction is counts for + neither: `finish` may start the same job again, with the same key too, and gets a new run. A start that only + dedupes commits nothing. + - `outcome(id, ctx)`: waits for an unsettled run; a settled or unknown id rejects `gone`. + - `abort(id)`: fires a running run's signal (before `run` started too: it starts aborted); drops a blocked run (no + `finish`; its record, key and state document go; `job_dropped`; its waiters reject `aborted`); ignored for + finishing runs and unknown ids. + - `running`: one entry per unsettled run in start order, `{ id, version, key?, args, state, phase: "running" | + "finishing" } | { id, version, phase: "blocked", reason: "finish_failed" }` (`state` only with a state + document). +- `run(args, ctx)`: `ctx` is a handler's `Ctx` (the object's principal for a remote type, the host's for a local one; + the host's incarnation; `ctx.get` released when `run` settles) plus `jobId` and, with a state document, `state: { + value, change(fn, ctx) }`. Its abort signal fires on `abort(id)`, on the object's end and at `host.close`. + `ctx.state.change(fn, ctx)`: a transaction drafting only the run's document, under `ctx` without its abort signal + (the run's own ctx is fine), accepted while `run` runs (also after an abort) and refused `released/settled` once it + settled; a change admitted + before `run` settled commits before the outcome is recorded. Every change is a commit (and a message while watched): + throttle and await or catch them. +- Outcomes: `{ id, args, state } & ({ kind: "done", result } | { kind: "failed", error })`, `error` being `{ kind: + "threw", error }` (`run` threw: reported `job_failed`; anything but a ChordError or PluginError becomes a + PluginError `internal` with its message) or `{ kind: "aborted", reason: "abort" }` (`run` threw after its signal + fired; a result returned after an abort is kept). +- `finish(outcome, tx)` runs when `run` settled (`phase: "finishing"`); its commit deletes the record and the key + and retires the state document; without `finish` the commit only cleans up. A throw changes nothing: retried after + 1 s doubling to 60 s, each failure reported `job_finish_failed`; after five in a row the run is `blocked`: outcome, + state and key kept, not counted for concurrency, its waiters waiting (their own ctx ends a wait). +- `run.outcome(ctx)` resolves once `finish` committed; the caller's ctx ends only its wait, which then keeps nothing + of it (no listener on its signal). A retained `Run` answers late waits from how the run settled; a rejection's error + keeps its stack's receivers (the transaction that ended the run) until its `stack` is read (V8), as every error. +- Keep-alive: running and finishing runs keep their object alive (a count, not references: never in + `host.references`); an object without references ends after its last run settles (after its `linger`), dropping + blocked runs (`job_dropped`). A run is counted in its start's commit step, before the commit publishes (a link + closing while it sends may release the last reference) and before any run of the same commit settles (a `finish` + chaining another job keeps the object). A run started while the object lingers keeps it. `host.activity.jobs` + counts them. +- The object's end (revocation, `host.close`, the last release): runs' signals fire with the end's error, the drain + waits for `run` and a `finish` running then (bounded by `closeTimeout`; admitted before `run` runs, which may end + its own object), nothing finishes after it (its commit is refused, not reported), waiters reject with the end's + error; blocked runs are dropped (`job_dropped`), at every end. `running` then reads `released` and keeps no run. + +## For Chord's layers (`internal.ts`) + +- `Job`: one job of one object (an `Attempt` makes it for `self.job`). `live()` once the object is live, `seal(error)` + when it ends, `end()` when its documents are dropped; `stage(tx, args, ctx, options)` is `start`'s and `tx.start`'s + work inside a transaction. Its records change only in staged changes (`documents/`'s `Staged`): `create`, + `outcome`, `delete`, `block`, `drop`, adopted in the commit step (a new run is counted there); the actions after the + commit start `run` (queued, after the transaction settled), settle waiters, uncount runs and report. A job stages + only when it records something (`Transaction.staged` reads what it staged so far). +- `JobOwner`: what a job needs of its object: label, host, caller, the object's context (without an abort signal), + documents, `admit` (the drain) and `countRuns` (keep-alive and `host.activity.jobs`). +- Inside: `JobRun` (one run: record, phase, abort signal, invocation fence closed as `run` settles, `running` entry, + the `Run` handle over its detachable waiters), the + run states (a keyed `DocumentHandle` by run id, apart from `self.state`'s, never drafted while absent) and `Running` + (`running`'s list, copied per change; while watched it records the ops of each change, a run's state ops re-pathed + under its entry, delivered as one frame per batch). diff --git a/packages/chord/src/next/jobs/definition.ts b/packages/chord/src/next/jobs/definition.ts new file mode 100644 index 000000000..c0dc86082 --- /dev/null +++ b/packages/chord/src/next/jobs/definition.ts @@ -0,0 +1,123 @@ +// Job definitions (§15): `defineJob(name, …)`, the checks of a declaration, and the types of what a run ends with. + +import { checkKeys, entriesOf } from "../contracts/declare.ts"; +import type { Static, TSchema } from "../contracts/index.ts"; +import { checkName } from "../contracts/internal.ts"; +import { type Snapshot, snapshotArguments, snapshotSchema } from "../contracts/snapshot.ts"; +import type { ChordError, PluginError } from "../data/index.ts"; +import type { JsonObject, StateDefinition } from "../documents/index.ts"; + +/** What `defineJob` takes besides the name. */ +export type JobOptions< + A extends TSchema, + R extends TSchema, + D extends StateDefinition | undefined, +> = { + /** The job's version; every change of its args, result or state bumps it. */ + readonly version: number; + /** The run's arguments: closed objects, JSON only (`running` carries them), checked at every start. */ + readonly args: A; + /** What `run` returns; not validated. */ + readonly result: R; + /** A singleton document definition: one document per run, `ctx.state` in `run`, `outcome.state` in `finish`. */ + readonly state?: D; + /** At most this many runs running or finishing per object; a start beyond it rejects `unavailable/busy`. */ + readonly concurrency?: number; + /** Per older version, how its args and result map forwards (used from slice 7 on; checked now). */ + readonly accepts?: JobAccepts; + /** How a persisted run resumes after a crash; refused before slice 6. */ + readonly persist?: { readonly resume: "rerun" | "abandon" }; +}; + +/** Per older version, its mappings forwards: its args to the current args, its result to the current result. */ +export type JobAccepts = { + readonly [version: number]: { readonly args: (old: never) => Static; readonly result: (old: never) => Static }; +}; + +/** The value of a job's state document: the definition's document, `undefined` without one. */ +type StateOf = D extends StateDefinition ? T : undefined; + +/** What a run ended with, as `finish` and `outcome` get it: its args and state, and its result or its error. */ +export type Outcome = { readonly id: string; readonly args: A; readonly state: S } & ( + | { readonly kind: "done"; readonly result: R } + | { readonly kind: "failed"; readonly error: JobError } +); + +/** + * Why a run failed: `run` threw (reported as `job_failed`; anything but a ChordError or PluginError becomes a + * PluginError `internal` with its message), or it was aborted (`abort`: `abort(id)`; `reload` and `restart` come with + * slices 7 and 6). + */ +export type JobError = + | { readonly kind: "threw"; readonly error: ChordError | PluginError } + | { readonly kind: "aborted"; readonly reason: "abort" | "reload" | "restart" }; + +/** Declares a job: owned work a call starts that outlives it (§15). */ +export function defineJob< + const A extends TSchema, + const R extends TSchema, + D extends StateDefinition | undefined = undefined, +>(name: string, options: JobOptions): JobDefinition, Static, StateOf> { + // Declaration policy: JavaScript callers reach every check below. + checkName(name, "A job name"); + checkKeys(options, ["version", "args", "result", "state", "concurrency", "accepts", "persist"], name); + const { version, state, concurrency, accepts, persist } = options; + if (!Number.isSafeInteger(version) || version < 1) throw new Error(`${name}.version: expected a positive integer`); + const args = snapshotArguments(options.args, `${name}.args`); + if (args.bytes) throw new Error(`${name}.args: job arguments are JSON (running carries them); no bytes()`); + const result = snapshotSchema(options.result, `${name}.result`); + if (state?.keyed) throw new Error(`${name}.state: expected a singleton document definition`); + if (concurrency !== undefined && !(Number.isSafeInteger(concurrency) && concurrency > 0)) { + throw new Error(`${name}.concurrency: expected a positive integer`); + } + if (accepts !== undefined) checkJobAccepts(accepts, version, `${name}.accepts`); + if (persist !== undefined) throw new Error(`${name}.persist: persisted jobs are not supported before slice 6`); + return new JobDefinition(name, version, args, result, state, concurrency, accepts ?? {}); +} + +/** Checks an accepts table's shape: older versions only, each with an `args` and a `result` function. */ +function checkJobAccepts(accepts: unknown, version: number, path: string): void { + for (const [key, adapter] of entriesOf(accepts, path)) { + const older = Number(key); + if (!Number.isSafeInteger(older) || older < 1 || older >= version) { + throw new Error(`${path}[${key}]: not an older version`); + } + checkKeys(adapter, ["args", "result"], `${path}[${key}]`); + const { args, result } = adapter as { readonly args?: unknown; readonly result?: unknown }; + if (typeof args !== "function") throw new Error(`${path}[${key}].args: expected a function`); + if (typeof result !== "function") throw new Error(`${path}[${key}].result: expected a function`); + } +} + +/** A declared job; `A`, `R` and `S` are its args, result and state document (`undefined` without one). */ +export class JobDefinition { + readonly name: string; + readonly version: number; + /** The args schema, Chord's copy; its compiled check is cached by this snapshot. */ + readonly args: Snapshot; + readonly result: Snapshot; + readonly state: StateDefinition | undefined; + readonly concurrency: number | undefined; + /** The older versions' mappings, checked at declaration; slice 7 maps older runs with them. */ + readonly accepts: JobAccepts; + /** Type only: the args, result and state types. */ + declare readonly "~types"?: { readonly args: A; readonly result: R; readonly state: S }; + + constructor( + name: string, + version: number, + args: Snapshot, + result: Snapshot, + state: StateDefinition | undefined, + concurrency: number | undefined, + accepts: JobAccepts, + ) { + this.name = name; + this.version = version; + this.args = args; + this.result = result; + this.state = state; + this.concurrency = concurrency; + this.accepts = accepts; + } +} diff --git a/packages/chord/src/next/jobs/index.ts b/packages/chord/src/next/jobs/index.ts new file mode 100644 index 000000000..5a8cd4739 --- /dev/null +++ b/packages/chord/src/next/jobs/index.ts @@ -0,0 +1,13 @@ +// Jobs for apps: declaring a job, its handle, its runs, their outcomes and `running`. + +export { + defineJob, + type JobAccepts, + type JobDefinition, + type JobError, + type JobOptions, + type Outcome, +} from "./definition.ts"; +export type { JobCtx, JobHandle, JobImplementation, RunState, StartOptions } from "./job.ts"; +export type { Run } from "./run.ts"; +export type { RunningEntry } from "./running.ts"; diff --git a/packages/chord/src/next/jobs/internal.ts b/packages/chord/src/next/jobs/internal.ts new file mode 100644 index 000000000..ffbd89a05 --- /dev/null +++ b/packages/chord/src/next/jobs/internal.ts @@ -0,0 +1,4 @@ +// Jobs for Chord's layers: a job's records on its object and the handle an attempt keeps, and what they need. + +export { type AnyJobImplementation, Job, type JobOwner } from "./job.ts"; +export { JobRecords, type RecordsOwner } from "./records.ts"; diff --git a/packages/chord/src/next/jobs/job.ts b/packages/chord/src/next/jobs/job.ts new file mode 100644 index 000000000..12e4b7403 --- /dev/null +++ b/packages/chord/src/next/jobs/job.ts @@ -0,0 +1,319 @@ +// Jobs (§15), in memory: the handle `self.job` returns, one per attempt, over the job's records on its object. It +// holds the implementation and the invocations: `run` runs outside transactions, `finish` in one with the cleanup, +// both on the attempt that is live; the attempt's seal cuts its invocations off (slice 6a §1.3). + +import { type Context, withAbortSignal, withoutAbortSignal } from "../../context/index.ts"; +import type { Draft } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Watchable } from "../../watch.ts"; +import { ChordError, isChordError, PluginError, type Principal } from "../data/index.ts"; +import type { JsonObject, Tx } from "../documents/index.ts"; +import type { Documents, SealToken, Transaction } from "../documents/internal.ts"; +import { type Ctx, sleep } from "../host/index.ts"; +import { type AnyHost, CallContext, type Caller, nextDelay } from "../host/internal.ts"; +import type { Outcome } from "./definition.ts"; +import type { JobRecords, Settled } from "./records.ts"; +import type { AnyOutcome, JobRun, Run } from "./run.ts"; +import type { RunningEntry } from "./running.ts"; + +/** A job's handle, `self.job(Job, { run, finish? })`; `A`, `R` and `S` are its args, result and state. */ +export interface JobHandle { + /** + * Starts a run in one transaction on the object's line and resolves with its `Run` after the commit: the args + * checked (`invalid/args`); a `key` of an unsettled run returns that run; at the `concurrency` limit (running plus + * finishing runs) it rejects `unavailable/busy`. `run` starts after the commit. + */ + start(args: A, context: Context, options?: StartOptions): Promise>>; + /** Waits for the outcome of the unsettled run `id`; a settled or unknown id rejects `gone`. */ + outcome(id: string, context: Context): Promise>; + /** + * Fires a running run's abort signal; drops a blocked run (no `finish`; its waiters reject `aborted`); ignored for + * finishing runs and unknown ids. + */ + abort(id: string): void; + /** The unsettled runs, one entry each in start order, tagged by phase (a JSON watchable, a state member). */ + readonly running: Watchable[]>; +} + +/** What `start` and `tx.start` take besides the args. */ +export type StartOptions = { + /** Names one run: a start with the key of an unsettled run returns that run (args not compared). */ + readonly key?: string; +}; + +/** What `self.job` takes: the run and the optional finish. */ +export type JobImplementation = { + /** The work, outside transactions; its ctx's abort signal fires on `abort(id)` and on its object's end. */ + readonly run: (args: A, ctx: JobCtx) => Promise; + /** Records the outcome in one transaction (only `await tx.*`); the run's record and state document go with it. */ + readonly finish?: (outcome: Outcome, tx: Tx) => void | Promise; +}; + +/** A run's ctx: a handler's ctx (the object's principal or the host's), its run's id and, with a state, the state. */ +export type JobCtx

= Ctx

& { readonly jobId: string } & ([S] extends [undefined] + ? unknown + : { readonly state: RunState }); + +/** A run's state document. */ +export interface RunState { + /** The document's value. */ + readonly value: S; + /** + * One transaction on the object's line drafting only this document, under `context` without its abort signal: + * admitted while `run` runs (also after an abort, with the run's own ctx), refused `released` once it settled. + */ + change(fn: (draft: Draft) => T, context: Context): Promise; +} + +/** Where a job's handle runs its work: its attempt, as the job sees it. */ +export type JobOwner = { + readonly host: AnyHost; + /** Whom runs act for: the object's principal (remote) or the host's (local), from this host. */ + readonly caller: Caller; + /** The object's context, without an abort signal: runs, state changes and finishes run under it. */ + readonly context: Context; + /** Aborted when the attempt is sealed: a finish's backoff waits under it. */ + readonly signal: AbortSignal; + /** The attempt's seal token: its starts and finishes are transactions under it. */ + readonly token: SealToken; + readonly documents: Documents; + /** + * Adds running work (an invocation and its finish, a deferred finish) to the attempt's drain; `cleanup` (its + * bookkeeping) runs once, when it settles or when the attempt ends with it still running. + */ + admit(running: Promise, cleanup?: () => void): void; +}; + +/** A job's implementation as Chord calls it, whatever the job's types. */ +export type AnyJobImplementation = { + readonly run: (args: JsonValue, ctx: JobContext) => Promise; + readonly finish?: (outcome: AnyOutcome, tx: Tx) => void | Promise; +}; + +/** Failed finishes in a row after which a run is blocked. */ +const FINISH_ATTEMPTS = 5; + +/** + * A job as one attempt declares it: its implementation and the invocations and finishes running on the attempt. Starts + * are taken while the attempt is live; the object's records keep the runs. + */ +export class Job implements JobHandle { + readonly #records: JobRecords; + readonly #implementation: AnyJobImplementation; + readonly #owner: JobOwner; + /** `declared` until the attempt is live (starts are refused), `sealed` once it is sealed. */ + #state: "declared" | "live" | "sealed" = "declared"; + /** Runs whose `run` runs on this attempt. */ + readonly #invocations = new Set(); + /** Runs whose finish runs on this attempt. */ + readonly #finishing = new Set(); + + constructor(records: JobRecords, implementation: AnyJobImplementation, owner: JobOwner) { + this.#records = records; + this.#implementation = implementation; + this.#owner = owner; + } + + get running(): Watchable[]> { + // Its value is the list of entries `JobRun.entryFor` builds. + return this.#records.running as unknown as Watchable[]>; + } + + async start(args: JsonValue, context: Context, options: StartOptions = {}): Promise> { + const { documents, token } = this.#owner; + return documents.tx((tx) => this.stage(tx, args, context, options), token, context); + } + + outcome(id: string, context: Context): Promise { + return this.#records.outcome(id, context); + } + + abort(id: string): void { + this.#records.abort(id); + } + + /** The attempt went live: starts are taken, and the records' waiting runs are invoked or finished here. */ + live(): void { + if (this.#state !== "declared") return; + this.#state = "live"; + this.#records.attach(this); + } + + /** + * Stages a run in `tx` (`start`, `tx.start`), in order: the args checked, dedupe by key (runs started earlier in + * `tx` included), the concurrency check, then the record and the state document, created in the commit. The run + * whose finish `tx` is counts for neither: `finish` may start the same job again, with the same key too. Returns + * the run's handle at once; a start that only dedupes stages nothing. + */ + stage(tx: Transaction, args: unknown, context: Context, options: StartOptions): Run { + // Declaration policy: a handle belongs to its object (types cannot say whose transaction this is). + if (tx.documents !== this.#owner.documents) { + throw new Error(`${this.#owner.documents.label}: a transaction starts only its own object's jobs`); + } + if (this.#state === "sealed") { + throw new ChordError({ + code: "released", + reason: "ended", + message: `${this.#records.name}: its attempt ended`, + }); + } + if (this.#state === "declared") { + throw new ChordError({ code: "not_ready", message: `${this.#records.name}: its object is not live yet` }); + } + return this.#records.stage(tx, args, context, options); + } + + /** + * The attempt is sealed with `error`: starts are refused, running invocations are cut off (their signals fire, they + * end `aborted` with reason `restart` and finish on the next live attempt), and the runs finishing here wait for it. + */ + seal(error: ChordError): void { + if (this.#state === "sealed") return; + this.#state = "sealed"; + this.#records.detach(this, this.#finishing); + this.#finishing.clear(); + for (const run of this.#invocations) this.#records.cut(run, error); + this.#invocations.clear(); + } + + /** + * Runs `run` for a committed run (with its signal aborted already after an early abort). The invocation and the + * run's finish are one piece of work in the drain, admitted before plugin code runs: `run` may end its own object + * before its first await. + */ + invoke(run: JobRun): void { + const { context, caller, host } = this.#owner; + const ctx = new JobContext(withAbortSignal(run.abort.signal, context), caller, host, run.id, this.#stateOf(run)); + const work = Promise.withResolvers(); + this.#owner.admit(work.promise, () => ctx.settle()); + this.#invocations.add(run); + run.invocation = "running"; + const invocation = (async () => { + try { + return await this.#implementation.run(run.args, ctx); + } finally { + // The fence closes as `run` settles: a change made later is refused. + run.invocation = "settled"; + } + })(); + invocation + .then( + (result) => this.#settled(run, ctx, { kind: "done", result }), + (error: unknown) => this.#settled(run, ctx, this.#failed(run, error)), + ) + .then(work.resolve); + } + + /** + * Runs the finish of `run`, whose outcome is recorded: `finish` with the cleanup in one transaction under the + * attempt's token, retried with backoff after a throw (reported), blocked after five in a row. The attempt's seal + * stops it without a report or a count; the next live attempt finishes the run. + */ + finish(run: JobRun): void { + const work = this.#finish(run); + this.#owner.admit(work); + } + + /** How a throwing `run` ended: `aborted` after its signal fired, otherwise `threw` (reported). */ + #failed(run: JobRun, error: unknown): Settled { + if (run.abort.signal.aborted) return { kind: "failed", error: { kind: "aborted", reason: "abort" } }; + this.#records.report({ kind: "job_failed", id: run.id, error }); + const thrown = isChordError(error) || error instanceof PluginError ? error : internal(error); + return { kind: "failed", error: { kind: "threw", error: thrown } }; + } + + /** + * `run` settled: its `ctx.get` references go; its outcome is recorded unless the attempt's seal cut it off (which + * recorded its own), then the run finishes here while this attempt is live. + */ + async #settled(run: JobRun, ctx: JobContext, ended: Settled): Promise { + ctx.settle(); + if (run.token.sealed) return; + this.#invocations.delete(run); + this.#records.settled(run, ended); + } + + async #finish(run: JobRun): Promise { + const { documents, token, context, signal } = this.#owner; + this.#finishing.add(run); + let delay: number | undefined; + try { + for (let failures = 1; ; failures++) { + try { + return await documents.tx( + async (tx) => { + const staging = this.#records.staging(tx); + staging.finishing = run; + await this.#implementation.finish?.(run.outcome!, tx); + staging.changes.push({ kind: "delete", run }); + }, + token, + context, + ); + } catch (error) { + if (this.#state === "sealed") return; + this.#records.report({ kind: "job_finish_failed", id: run.id, error }); + if (failures === FINISH_ATTEMPTS) return await this.#records.record(run, "block"); + delay = nextDelay(delay); + try { + await sleep(delay, withAbortSignal(signal, context)); + } catch { + return; + } + } + } + } finally { + this.#finishing.delete(run); + } + } + + /** `ctx.state` of `run`, when the job has a state: changes under the run's own token while `run` runs. */ + #stateOf(run: JobRun): RunState | undefined { + const records = this.#records; + const states = records.states; + if (states === undefined) return undefined; + const { documents } = this.#owner; + const refused = (): ChordError => + new ChordError({ + code: "released", + reason: "settled", + message: `${records.name}: run ${run.id} settled; its state is closed`, + }); + return { + get value(): JsonObject { + const value = records.stateOf(run.id); + if (value === undefined) throw refused(); + return value; + }, + change: (fn: (draft: Draft) => T, context: Context): Promise => { + if (run.invocation !== "running") return Promise.reject(refused()); + // Never the abort signal: a run records its progress also after an abort. + const unaborted = withoutAbortSignal(context); + return documents.tx(async (tx) => fn(await tx.draftSlot(states, run.id)), run.token, unaborted); + }, + }; + } +} + +/** A plugin's throw that is neither a ChordError nor a PluginError, as a run's outcome keeps it. */ +function internal(error: unknown): PluginError { + return new PluginError("internal", error instanceof Error ? error.message : String(error)); +} + +/** A run's ctx: the handler's ctx of its object, its id and, with a state, its state document. */ +export class JobContext extends CallContext { + readonly jobId: string; + declare readonly state: RunState | undefined; + + constructor( + context: Context, + caller: Caller, + host: AnyHost, + jobId: string, + state: RunState | undefined, + ) { + super(context, caller, host); + this.jobId = jobId; + if (state !== undefined) this.state = state; + } +} diff --git a/packages/chord/src/next/jobs/records.ts b/packages/chord/src/next/jobs/records.ts new file mode 100644 index 000000000..b1e8e306b --- /dev/null +++ b/packages/chord/src/next/jobs/records.ts @@ -0,0 +1,446 @@ +// A job's records on its object (§15, slice 6a §1.2): its runs by id and by key, the active count, the run state +// documents, `running` and the runs waiting for a live attempt, across the object's attempts. Records and state +// documents change only in transactions on the object's line (staged changes adopted in the commit step). + +import type { Context } from "../../context/index.ts"; +import { type Op, type Prepared, type Tracker, track } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { ChordError } from "../data/index.ts"; +import { check, validatorFor } from "../data/internal.ts"; +import type { JsonObject, StateDefinition } from "../documents/index.ts"; +import type { DocumentSlot, Documents, Staged, Transaction } from "../documents/internal.ts"; +import type { AnyHost } from "../host/internal.ts"; +import { uuidv7 } from "../host/internal.ts"; +import type { JobDefinition, JobError } from "./definition.ts"; +import type { Job, StartOptions } from "./job.ts"; +import { type AnyOutcome, JobRun, type Run } from "./run.ts"; +import { Running } from "./running.ts"; + +/** What a job's records need of their object. */ +export type RecordsOwner = { + readonly label: string; + readonly host: AnyHost; + /** The object's context, without an abort signal: outcomes are recorded under it. */ + readonly context: Context; + readonly documents: Documents; + /** Counts running and finishing runs: they keep the object alive (`host.activity.jobs`). */ + countRuns(delta: number): void; +}; + +/** How a run's invocation ended, before its outcome is recorded. */ +export type Settled = + | { readonly kind: "done"; readonly result: unknown } + | { readonly kind: "failed"; readonly error: JobError }; + +/** A change of the records, staged in a transaction and adopted in its commit step. */ +type RecordChange = + | { readonly kind: "create"; readonly run: JobRun; readonly state: JsonObject | undefined } + | { readonly kind: "outcome"; readonly run: JobRun; readonly outcome: AnyOutcome } + | { readonly kind: "delete" | "block" | "drop"; readonly run: JobRun }; + +/** One transaction's changes of one job: the runs it started (dedupe, concurrency) and whose finish it is. */ +export type Staging = Staged & { + readonly started: JobRun[]; + readonly changes: RecordChange[]; + finishing: JobRun | undefined; +}; + +const RESTART: Settled = { kind: "failed", error: { kind: "aborted", reason: "restart" } }; + +/** + * The records of one job name on one object. The live attempt's `Job` invokes and finishes its runs; between live + * attempts, committed runs not yet invoked and runs whose outcome is recorded wait for the next one. + */ +export class JobRecords { + readonly definition: JobDefinition; + readonly running: Running; + readonly #owner: RecordsOwner; + readonly #states: RunStates | undefined; + readonly #runs = new Map(); + readonly #keys = new Map(); + /** Running and finishing runs: concurrency and keep-alive count them. */ + #active = 0; + /** The live attempt's job; `undefined` between live attempts. */ + #job: Job | undefined; + /** Runs waiting for a live attempt: to be invoked (`invocation` pending) or finished (phase `finishing`). */ + readonly #waiting = new Set(); + /** The object's end error, once it ended. */ + #ended: ChordError | undefined; + + constructor(definition: JobDefinition, owner: RecordsOwner) { + this.definition = definition; + this.#owner = owner; + const { state } = definition; + this.#states = + state === undefined + ? undefined + : new RunStates(state, owner.documents, (id, ops) => this.#stateChanged(id, ops)); + this.running = new Running(owner.documents.batch, this.name); + } + + /** The job in errors: its object and its name. */ + get name(): string { + return `${this.#owner.label} job ${this.definition.name}`; + } + + /** Waits for the outcome of the unsettled run `id`; a settled or unknown id rejects `gone`. */ + outcome(id: string, context: Context): Promise { + const run = this.#runs.get(id); + if (run === undefined) { + return Promise.reject(new ChordError({ code: "gone", message: `${this.name}: no unsettled run ${id}` })); + } + return run.handle.outcome(context); + } + + /** Fires a running run's signal; drops a blocked run; ignored for finishing runs, unknown ids and after the end. */ + abort(id: string): void { + const run = this.#runs.get(id); + if (run === undefined || this.#ended !== undefined) return; + if (run.phase === "running") { + run.abort.abort(new ChordError({ code: "aborted", message: `${this.name}: run ${id} aborted` })); + } else if (run.phase === "blocked") { + void this.record(run, "drop"); + } + } + + /** See `Job.stage`: the checks and the staged creation, against the object's runs. */ + stage(tx: Transaction, args: unknown, context: Context, options: StartOptions): Run { + const definition = this.definition; + // Plugin code, not type-checked: the args are checked as a method's are (R7). + check(validatorFor(definition.args, context), args, "args", context); + const staged = tx.staged(this); + const { key } = options; + if (key !== undefined) { + const known = staged?.started.find((run) => run.key === key) ?? this.#keys.get(key); + if (known !== undefined && known !== staged?.finishing) return known.handle; + } + const { concurrency } = definition; + const active = this.#active + (staged?.started.length ?? 0) - (staged?.finishing === undefined ? 0 : 1); + if (concurrency !== undefined && active >= concurrency) { + const message = `${this.name}: at its concurrency limit (${concurrency})`; + throw new ChordError({ code: "unavailable", reason: "busy", message }); + } + const state = definition.state?.initial(context); + // Checked above: the args are JSON (the definition refuses bytes). + const run = new JobRun(uuidv7(), definition.version, key, args as JsonValue); + const staging = this.staging(tx); + staging.started.push(run); + staging.changes.push({ kind: "create", run, state }); + return run.handle; + } + + /** This job's changes in `tx`, made on the first. */ + staging(tx: Transaction): Staging { + return tx.stage(this, () => { + const changes: RecordChange[] = []; + return { + started: [], + changes, + finishing: undefined, + adopt: () => { + // A step admitted before the object ended adopts after it: the records are not changed any more. + if (this.#ended !== undefined) return; + for (const change of changes) this.#adopt(change); + }, + committed: () => { + const ended = this.#ended; + if (ended === undefined) { + for (const change of changes) this.#committed(change); + return; + } + // Its runs end as the object's other runs did. + for (const change of changes) if (change.kind === "create") change.run.reject(ended); + }, + discarded: () => { + for (const change of changes) if (change.kind === "create") change.run.reject(this.#discarded()); + }, + }; + }); + } + + /** `job`, of the attempt that went live, invokes and finishes the runs from now on; the waiting ones first. */ + attach(job: Job): void { + if (this.#ended !== undefined) return; + this.#job = job; + const waiting = [...this.#waiting]; + this.#waiting.clear(); + for (const run of waiting) this.#dispatch(run, job); + } + + /** + * `job`'s attempt was sealed: its running invocations are cut off (`cut`), the runs it was finishing wait for the + * next live attempt. + */ + detach(job: Job, finishing: Iterable): void { + if (this.#job === job) this.#job = undefined; + for (const run of finishing) this.#need(run); + } + + /** + * A run's invocation settled: its outcome is recorded (phase `finishing`) in a transaction of the object behind the + * state changes admitted before, so `outcome.state` has them; then it is finished on the live attempt. + */ + settled(run: JobRun, ended: Settled): void { + void this.#recordOutcome(run, ended); + } + + /** + * A running run whose attempt was sealed while the object continues: its signal fires with `error`, its invocation + * is fenced (later state changes refused, the open one discarded, its settling ignored), and it ends `failed { + * aborted, reason: "restart" }`, finished on the next live attempt. + */ + cut(run: JobRun, error: ChordError): void { + run.invocation = "settled"; + this.#owner.documents.fence(run.token); + run.abort.abort(error); + if (this.#ended === undefined) void this.#recordOutcome(run, RESTART); + } + + /** + * Records `kind` for `run` in a transaction of the object: `block` a finishing run, `drop` a blocked one (its + * record, key and state document go), if it still is one there. The object's end ends it. + */ + async record(run: JobRun, kind: "block" | "drop"): Promise { + const { documents, context } = this.#owner; + try { + await documents.tx( + (tx) => { + const current = this.#runs.get(run.id) === run; + if (current && run.phase === (kind === "block" ? "finishing" : "blocked")) { + this.staging(tx).changes.push({ kind, run }); + } + }, + documents.own, + context, + ); + } catch { + // The object ended: its runs went with it. + } + } + + /** The value of run `id`'s state document, `undefined` without one or once retired. */ + stateOf(id: string): JsonObject | undefined { + return this.#states?.value(id); + } + + /** The run state documents, for `ctx.state.change`; `undefined` without a state definition. */ + get states(): DocumentSlot | undefined { + return this.#states; + } + + /** + * The object ends: running runs' signals fire, every waiter rejects with `error`, blocked runs are dropped + * (reported); nothing finishes. The records stay readable until `drop`. + */ + end(error: ChordError): void { + if (this.#ended !== undefined) return; + this.#ended = error; + this.#job = undefined; + this.#waiting.clear(); + for (const run of this.#runs.values()) { + if (run.phase === "blocked") this.report({ kind: "job_dropped", id: run.id }); + run.abort.abort(error); + run.reject(error); + } + if (this.#active > 0) this.#owner.countRuns(-this.#active); + } + + /** Its object's documents are dropped: the runs, their state documents and `running`'s watches go. */ + drop(): void { + this.#runs.clear(); + this.#keys.clear(); + this.#states?.drop(); + this.running.end(); + } + + /** Reports a run's failure or drop to the host. */ + report( + error: + | { readonly kind: "job_failed" | "job_finish_failed"; readonly id: string; readonly error: unknown } + | { readonly kind: "job_dropped"; readonly id: string }, + ): void { + this.#owner.host.report({ ...error, object: this.#owner.label, job: this.definition.name }); + } + + /** Records the outcome of `run`, then has it finished on the live attempt. */ + async #recordOutcome(run: JobRun, ended: Settled): Promise { + const { documents, context } = this.#owner; + try { + await documents.tx( + (tx) => { + const outcome = { id: run.id, args: run.args, state: this.stateOf(run.id), ...ended }; + this.staging(tx).changes.push({ kind: "outcome", run, outcome }); + }, + documents.own, + context, + ); + } catch { + // The object ended: nothing finishes. + return; + } + this.#need(run); + } + + /** `run` needs a live attempt (to be invoked or finished): now, or once one attaches. */ + #need(run: JobRun): void { + if (this.#ended !== undefined || this.#runs.get(run.id) !== run) return; + const job = this.#job; + if (job === undefined) this.#waiting.add(run); + else this.#dispatch(run, job); + } + + /** Invokes or finishes `run` on `job`, as its record stands. */ + #dispatch(run: JobRun, job: Job): void { + if (this.#runs.get(run.id) !== run) return; + if (run.phase === "finishing") job.finish(run); + else if (run.phase === "running" && run.invocation === "pending") job.invoke(run); + } + + /** + * Adopts one record change, inside the commit step: the maps, the active count, the state document, `running`. A + * new run is counted for its object here, before the commit's publication (a link closing while it sends may + * release the object's last reference) and before any run of the same commit settles in `#committed`. + */ + #adopt(change: RecordChange): void { + const { run } = change; + switch (change.kind) { + case "create": + this.#runs.set(run.id, run); + if (run.key !== undefined) this.#keys.set(run.key, run); + this.#active++; + this.#owner.countRuns(1); + if (change.state !== undefined) this.#states?.create(run.id, change.state); + run.entry = run.entryFor(change.state); + this.running.add(run.entry); + return; + case "outcome": + run.outcome = change.outcome; + run.phase = "finishing"; + this.#replaceEntry(run); + return; + case "block": + run.phase = "blocked"; + this.#active--; + this.#replaceEntry(run); + return; + case "delete": + case "drop": + this.#runs.delete(run.id); + // The finishing run's key may name its successor already, started in the same commit. + if (run.key !== undefined && this.#keys.get(run.key) === run) this.#keys.delete(run.key); + if (change.kind === "delete") this.#active--; + this.#states?.retire(run.id); + this.running.remove(run.entry!); + run.entry = undefined; + return; + } + } + + /** The actions after a record change committed. */ + #committed(change: RecordChange): void { + const { run } = change; + switch (change.kind) { + case "create": + // After the commit, outside the batch, once the transaction settled; on the live attempt, or the next. + queueMicrotask(() => this.#need(run)); + return; + case "outcome": + return; + case "delete": + this.#owner.countRuns(-1); + run.resolve(run.outcome!); + return; + case "block": + this.#owner.countRuns(-1); + return; + case "drop": + this.report({ kind: "job_dropped", id: run.id }); + run.reject(new ChordError({ code: "aborted", message: `${this.name}: run ${run.id} dropped` })); + return; + } + } + + /** A run's state document changed by `ops`, inside the commit step: its entry is replaced. */ + #stateChanged(id: string, ops: readonly Op[]): void { + const run = this.#runs.get(id)!; + const before = run.entry!; + run.entry = run.entryFor(this.stateOf(id)); + this.running.stateChanged(before, run.entry, ops); + } + + /** A run's record changed, inside the commit step: its entry is replaced. */ + #replaceEntry(run: JobRun): void { + const before = run.entry!; + run.entry = run.entryFor(this.stateOf(run.id)); + this.running.replace(before, run.entry); + } + + #discarded(): ChordError { + return new ChordError({ + code: "released", + reason: "discarded", + message: `${this.name}: the start's transaction was discarded`, + }); + } +} + +/** + * A job's state documents, one per run by its id: Chord's own keyed slot, apart from the object's `self.state` slots. + * Created with the run and retired with its record; never drafted while absent, so a retired one is never recreated. + * Each commit tells the job, whose `running` carries the state. + */ +class RunStates implements DocumentSlot { + readonly owner: Documents; + /** In memory only (persisted jobs are slice 6b's). */ + readonly persisted = undefined; + readonly #definition: StateDefinition; + readonly #trackers = new Map>(); + readonly #changed: (id: string, ops: readonly Op[]) => void; + + constructor( + definition: StateDefinition, + owner: Documents, + changed: (id: string, ops: readonly Op[]) => void, + ) { + this.#definition = definition; + this.owner = owner; + this.#changed = changed; + } + + /** The value of run `id`'s document; `undefined` once retired. */ + value(id: string): JsonObject | undefined { + return this.#trackers.get(id)?.value; + } + + current(key: string): Tracker | undefined { + return this.#trackers.get(key); + } + + initial(_context: Context): JsonObject { + throw new ChordError({ + code: "released", + reason: "settled", + message: `${this.#definition.name}: the run's document is retired`, + }); + } + + adopt(key: string, tracker: Tracker, prepared: Prepared): void { + tracker.adopt(prepared); + this.#changed(key, prepared.ops); + } + + /** Retires run `key`'s document (its record's deletion). */ + retire(key: string): void { + this.#trackers.delete(key); + } + + /** Creates run `id`'s document with `value` (its start's commit). */ + create(id: string, value: JsonObject): void { + this.#trackers.set(id, track(value)); + } + + /** Drops every document (the object ended). */ + drop(): void { + this.#trackers.clear(); + } +} diff --git a/packages/chord/src/next/jobs/run.ts b/packages/chord/src/next/jobs/run.ts new file mode 100644 index 000000000..688e0efa1 --- /dev/null +++ b/packages/chord/src/next/jobs/run.ts @@ -0,0 +1,137 @@ +// One run of a job (§15): its record (id, version, key, args, phase, outcome), its abort signal, the fence of its +// invocation (its seal token), its `running` entry and the `Run` handle its callers wait on. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { ChordError } from "../data/index.ts"; +import { SealToken } from "../documents/internal.ts"; +import type { Outcome } from "./definition.ts"; + +/** A started run as its caller holds it: its id and a wait for its outcome. */ +export interface Run { + readonly id: string; + /** + * Resolves with the outcome once `finish` committed (also without a `finish`); a blocked run keeps waiting, a dropped + * one rejects `aborted`, the object's end rejects with its error; `context` ends only this wait, which then keeps + * nothing of it. A rejection's error keeps its stack's receivers (the transaction that ended the run) until its + * `stack` is read (V8). + */ + outcome(context: Context): Promise; +} + +/** A run's outcome, as Chord keeps it whatever the job's types. */ +export type AnyOutcome = Outcome; + +/** + * Where a run's record stands. `running` from its start's commit (its `run` starts after it), `finishing` once its + * outcome is recorded, `blocked` after five failed finishes. + */ +type Phase = "running" | "finishing" | "blocked"; + +/** Where a run's invocation stands: `ctx.state.change` is admitted only while it runs (the invocation fence). */ +type Invocation = "pending" | "running" | "settled"; + +/** How a run settled for its waiters. */ +type Answer = + | { readonly kind: "done"; readonly outcome: AnyOutcome } + | { readonly kind: "failed"; readonly error: ChordError }; + +/** One wait for a run's outcome, removed when its context aborts. */ +type Waiter = { readonly settle: (settled: Answer) => void }; + +/** One run: changed only by its job, inside its transactions' commit steps (records) and after them (actions). */ +export class JobRun { + readonly id: string; + readonly version: number; + readonly key: string | undefined; + readonly args: JsonValue; + /** `run`'s abort signal: `abort(id)`, the object's end and `host.close` fire it; an early abort pre-marks it. */ + readonly abort = new AbortController(); + readonly handle: Run; + /** The seal token of its invocation: sealed when the invocation is cut off by its attempt's seal (`ctx.state.change`). */ + readonly token = new SealToken(); + phase: Phase = "running"; + invocation: Invocation = "pending"; + /** What it ended with, once recorded (finishing and blocked). */ + outcome: AnyOutcome | undefined; + /** Its entry in `running`, while its record exists; replaced when its record or state document changes. */ + entry: JsonValue | undefined; + /** Its waiters: apart from the run, so a handle kept by a caller keeps only how the run settled. */ + readonly #waiters = new Waiters(); + + constructor(id: string, version: number, key: string | undefined, args: JsonValue) { + this.id = id; + this.version = version; + this.key = key; + this.args = args; + const waiters = this.#waiters; + this.handle = { id, outcome: (context) => waiters.wait(context) }; + } + + /** Its `running` entry for its current record and `state` (the job's state document, `undefined` without one). */ + entryFor(state: JsonValue | undefined): JsonValue { + const { id, version, key, args, phase } = this; + if (phase === "blocked") return { id, version, phase, reason: "finish_failed" }; + return { + id, + version, + ...(key === undefined ? {} : { key }), + args, + ...(state === undefined ? {} : { state }), + phase, + }; + } + + /** Its waiters get the outcome. */ + resolve(outcome: AnyOutcome): void { + this.#waiters.settle({ kind: "done", outcome }); + } + + /** Its waiters get `error`. */ + reject(error: ChordError): void { + this.#waiters.settle({ kind: "failed", error }); + } +} + +/** A run's waits for its outcome: each removed when its context aborts; late ones answered at once. */ +class Waiters { + /** How the run settled, once it did. */ + #settled: Answer | undefined; + readonly #waiting = new Set(); + + /** The run settled (once): every wait gets it. */ + settle(settled: Answer): void { + if (this.#settled !== undefined) return; + this.#settled = settled; + for (const waiter of this.#waiting) waiter.settle(settled); + this.#waiting.clear(); + } + + /** A wait for the outcome: at once once settled; otherwise until it settles or `context` aborts. */ + wait(context: Context): Promise { + const settled = this.#settled; + if (settled !== undefined) return answer(settled); + const signal = context.abortSignal; + if (signal?.aborted) return Promise.reject(signal.reason); + const { promise, resolve, reject } = Promise.withResolvers(); + const abort = (): void => { + this.#waiting.delete(waiter); + reject(signal?.reason); + }; + const waiter: Waiter = { + settle: (each) => { + signal?.removeEventListener("abort", abort); + if (each.kind === "done") resolve(each.outcome); + else reject(each.error); + }, + }; + this.#waiting.add(waiter); + signal?.addEventListener("abort", abort, { once: true }); + return promise; + } +} + +/** A settled run's answer to a wait. */ +function answer(settled: Answer): Promise { + return settled.kind === "done" ? Promise.resolve(settled.outcome) : Promise.reject(settled.error); +} diff --git a/packages/chord/src/next/jobs/running.ts b/packages/chord/src/next/jobs/running.ts new file mode 100644 index 000000000..3f24ccd24 --- /dev/null +++ b/packages/chord/src/next/jobs/running.ts @@ -0,0 +1,143 @@ +// A job's `running` (§15): a JSON watchable of its unsettled runs, one entry per run in start order, tagged by phase. +// It is a source of its own (keyed state documents publish nothing): the job tells it each change in the commit step, +// and a frame carries exactly the ops of those changes, a run's state ops re-pathed under its entry. + +import type { Context } from "../../context/index.ts"; +import type { NonEmptyPath, Op, Path } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { ChordError } from "../data/index.ts"; +import { type Batch, Source } from "../state/internal.ts"; + +/** + * One entry of `running`: a running or finishing run with its args (and state, when the job has a state document), or + * a blocked run, which carries neither (it may be of another version than the code). + */ +export type RunningEntry = + | ({ + readonly id: string; + readonly version: number; + readonly key?: string; + readonly args: A; + readonly phase: "running" | "finishing"; + } & ([S] extends [undefined] ? unknown : { readonly state: S })) + | { readonly id: string; readonly version: number; readonly phase: "blocked"; readonly reason: "finish_failed" }; + +const EMPTY: readonly JsonValue[] = []; + +/** + * The list behind `running`. Every change copies the list (O(unsettled runs)) and, while it is watched, records its + * ops; the batch delivers them as one frame. Unwatched, no op is built. + */ +export class Running extends Source { + /** The job in errors. */ + readonly #label: string; + /** `undefined` once its object ended: reads throw `released`, and no ended run's args or state are kept. */ + #list: readonly JsonValue[] | undefined = EMPTY; + /** The ops of the current batch's changes, kept only while watched. */ + #ops: Op[] = []; + /** Whether the current batch listed this source for a frame. */ + #listed = false; + + constructor(batch: Batch, label: string) { + super(batch); + this.#label = label; + } + + get value(): JsonValue { + // A list of JSON entries, shared and never mutated once published. + return this.#current() as JsonValue; + } + + /** Appends the entry of a new run. */ + add(entry: JsonValue): void { + const list = [...this.#current(), entry]; + this.#list = list; + if (this.#mark()) this.#ops.push(["p", [], list.length - 1, 0, [entry]]); + } + + /** Replaces a run's entry `before` by `after` (its record changed). */ + replace(before: JsonValue, after: JsonValue): void { + const index = this.#replaced(before, after); + if (this.#mark()) this.#ops.push(["s", [index], after]); + } + + /** Replaces a run's entry `before` by `after`, whose state document changed by `ops`. */ + stateChanged(before: JsonValue, after: JsonValue, ops: readonly Op[]): void { + const index = this.#replaced(before, after); + if (this.#mark()) for (const op of ops) this.#ops.push(under([index, "state"], op)); + } + + /** Removes a run's entry. */ + remove(entry: JsonValue): void { + const list = this.#current(); + const index = list.indexOf(entry); + this.#list = list.toSpliced(index, 1); + if (this.#mark()) this.#ops.push(["p", [], index, 1, []]); + } + + deliver(context: Context): void { + const ops = this.#ops; + this.#ops = []; + this.#listed = false; + this.publish(this.#current() as JsonValue, ops, false, context); + } + + /** Its object ended: every watch ends, and the list goes. */ + end(): void { + this.#list = undefined; + this.#ops = []; + this.endWatches(); + } + + #current(): readonly JsonValue[] { + if (this.#list === undefined) { + throw new ChordError({ code: "released", reason: "ended", message: `${this.#label}: ended` }); + } + return this.#list; + } + + /** Copies the list with `after` in the place of `before`; returns that place. */ + #replaced(before: JsonValue, after: JsonValue): number { + const list = this.#current(); + const index = list.indexOf(before); + this.#list = list.with(index, after); + return index; + } + + /** + * Records that the list changed, in the batch running now: derives over it are marked; while watched it is listed + * once per batch for a frame. Returns whether the change's ops are to be recorded (only while watched). + */ + #mark(): boolean { + if (!this.watched) { + this.changed(); + return false; + } + if (!this.#listed) { + this.#listed = true; + this.changed(); + } + return true; + } +} + +/** `op` of a value at `prefix` inside a larger one: a replacement becomes a set there. */ +function under(prefix: readonly [number, string], op: Op): Op { + const at = (path: Path): NonEmptyPath => [prefix[0], prefix[1], ...path]; + switch (op[0]) { + case "r": + return ["s", prefix, op[1]]; + case "s": + return ["s", at(op[1]), op[2]]; + case "d": + return ["d", at(op[1])]; + case "a": + return ["a", at(op[1]), op[2]]; + case "t": + return ["t", at(op[1]), op[2]]; + case "p": + return ["p", at(op[1]), op[2], op[3], op[4]]; + case "m": + return ["m", at(op[1]), op[2]]; + } +} diff --git a/packages/chord/src/next/link/README.md b/packages/chord/src/next/link/README.md new file mode 100644 index 000000000..381fea216 --- /dev/null +++ b/packages/chord/src/next/link/README.md @@ -0,0 +1,141 @@ +# link + +Links between hosts and the protocol on them (§3, §22, slices 3a 2b, 4a and 4b): one `Message` type checked after +every decode (skipped on trusted links), the codecs that turn it into bytes, opaque parts with their limits and +compression at the origin, the handshake, the heartbeat, in-order handling of messages waiting for an asynchronous +inflate, the IPC and WebSocket transports, values with raw byte segments, and errors on the wire. + +## For apps (`index.ts`) + +```ts +import { memoryLinkPair, textCodec } from "./index.ts"; + +const [up, down] = memoryLinkPair(); +server.accept(up, { principal: { kind: "guest", session: "S1" } }, ctx); // the entry point authenticated it +client.connect(down, { principal: { kind: "server" } }, ctx); // both ends: binaryCodec by default +// For a readable wire while debugging: { principal, codec: textCodec } on both ends. + +// A forked worker over IPC (the parent end; the child uses ipcLink(process)), trusted: no message schema check. +const child = fork("worker.ts", [], { serialization: "advanced" }); +server.accept(ipcLink(child), { principal: { kind: "worker" }, trusted: true }, ctx); +// A browser tab over a WebSocket (`ws` on the server, the browser's WebSocket in the tab). +wss.on("connection", (ws, req) => server.accept(webSocketLink(ws), { principal: authenticate(req) }, ctx)); +tab.connect(webSocketLink(new WebSocket(url)), { principal: { kind: "server" } }, ctx); +``` + +- `Link`: an ordered, bidirectional channel of `Uint8Array` messages: `send(message)` (a no-op once closed), + `onMessage(listener)` (messages that arrived earlier are delivered to it first; not kept on a closed link), + `close()` (both ends; undelivered messages are dropped), `onClose(listener)` (runs at once on a closed link). + Adapters only move whole messages, in order, and close both ends. +- `memoryLinkPair()`: two ends in one process; each message arrives a microtask later (`queueMicrotask`), in order. + Closing one end closes it at once and the other a microtask later. +- `ipcLink(port)`: over a Node IPC channel, a `ChildProcess` forked with `serialization: "advanced"` or `process` in + the child (`IpcPort`, typed structurally: Chord imports nothing from Node). Each message is one `Uint8Array`; other + messages on the channel (the app's own, such as a start message) are skipped, and messages that arrive before the + link's listener is set are queued, so a child makes its link before awaiting its start message. `send` returning + `false` is backpressure, not a failure; a thrown `send`, an error in its callback or `disconnect` closes the link once + and removes its listeners. +- `webSocketLink(socket)`: over a WebSocket, the browser's or `ws`'s (`WebSocketLike`, typed structurally): binary + frames, `binaryType = "arraybuffer"`; messages sent while it connects wait for `open`; a socket closing or closed + already gives a closed link; `close`, `error` and a text frame close the link once. The transport's `maxPayload` + is its hard cap (a larger frame closes the socket, code 1009); the limits per part are the host's (`host/`). +- `Compression` (`{ deflate(bytes), inflate(bytes, limit) }`, synchronous): what `createHost({ compression })` takes. + `deflate` is raw deflate at level 1; `inflate` gives a tagged `Inflated`: `{ kind: "bytes", bytes }` (at most + `limit` bytes; `Infinity`: none; 0 allowed), `{ kind: "too_large" }` when the output would be larger, `{ kind: + "corrupt", error }` for data that is no raw deflate stream. Chord ships `nodeCompression` + (zlib's `deflateRawSync`/`inflateRawSync` with `maxOutputLength`) in `src/next-node/compression.ts`, outside the + portable code. `PartLimits` (`{ part?, decoded? }`, bytes) is what a link's `limits` take. +- `Codec { name, encode(message), decode(bytes) }`, chosen per link (`codec` in `accept`/`connect` and a route's + `resolve` result, the same at both ends; no negotiation). Two links of one relay may use different codecs: + - `binaryCodec` (the default): a kind byte (its high bit set when an `origin` follows as a varint), then the kind's + fields: unsigned varints (ids, indexes, counts, + lengths), length-prefixed UTF-8 strings, the incarnation as 16 and the fingerprint as 8 raw bytes, parts as + length-prefixed bytes (decoded as views of the received message, never parsed), other JSON fields (a hold's + address and principal, errors) as length-prefixed JSON; a hold's optional fields behind one flags byte; a payload + marked as carrying bytes adds its segments (count, lengths, raw bytes); `failed`'s `retryIn` as one more than it + (0: absent). A part's length prefix is its length doubled, plus one when the part is deflated. A hold's `chain` + as a count and 16 raw bytes per instance id, its `limit` as a varint. No outer length. + - `textCodec`, for debugging: one JSON tuple per message with a short kind tag and positional fields (`["R", hold]`, + `["c", call, hold, method, args?]`, `["b", [[hold, variant, [[member, value]…], [[member, error]…]?]…]]`, `["s", + [[hold, [[member, ops]…], [[member, error]…]?]…]]`, `["f", hold, "t" | "b", error, retryIn?]`, `["T", hold]` for + `retry`, `["l", hold]` for `loading`, `["a"]` ack, `["C"]` closed, `["p"]` ping, `["P"]` pong, …; a message with an origin wrapped as + `["@", origin, tuple]`); parts are + spliced into the tuple as their JSON text and re-serialized at decode (one parse per part per hop, accepted for a + debugging codec); a payload marked as carrying bytes ends with its segments' lengths, the segments following the + tuple after a newline (JSON text has no raw one). Every part is tagged with its codec, so no JSON value is + reserved: `[0, json]`, or `[1, base64]` for a part its origin deflated (the text codec itself never compresses); + `["V", revocation]` is `revoke`. + - `describeMessage(message)`: the text tuple form of any message, for logs and tests. +- `accept`, `connect` and a route's `resolve` take `trusted` (default `false`): a trusted link skips the `MESSAGE` + schema check (framing, the codec's decode and the owners' checks still run). For links to the app's own processes; + trust never grants a hold's principal fields (they are read only on a parent link, `remote/`). + +## For Chord's layers (`internal.ts`) + +- `Message` (`message.ts`): the protocol's one tagged union, from the TypeBox schema `MESSAGE`: `hello { protocol, + incarnation, calls }` (`calls`: how many concurrent calls this side accepts per origin, its call window), + `hold { hold, type, version, fingerprint, params, address?, principal?, incarnation? }` (`address`: `{ kind, params }` + the caller's placement computed, it routes the hold; the provider places it by its params; absent for a type without placement; `principal` and `incarnation`: whom it acts for, read only + on a parent link; `chain?`: the instance ids of the activations waiting for the hold, at most 32, read only on a + trusted link; `limit?`: the decoded limit per part of the link a relay admitted it on, read only on a parent link), `release`, `call { call, hold, method, args?, segments? }`, `abort`, `bound { holds: [{ hold, + variant, members: [{ member, value }], failed }] }` (one message may bind several holds: first bindings, re-holds), + `state { holds: [{ hold, members: [{ member, ops }], failed }] }`, `ack {}` (one per applied `bound` or `state` of + its origin), `closed { origin }` (a relay's downstream origin ended: its holds and calls go), `ping {}`, `pong {}`, + `revoke { revocation }` (from the parent, applied with the child's own `revokedBy`; no origin), `failed { hold, cause, + retryIn? }` (`retryIn`: the milliseconds left of the provider's backoff window, absent for a broken failure), + `loading { hold }` (the hold's object, or a relay's upstream, starts over), `retry { hold }` (the consumer asks for an + attempt now), `result { call, value?, segments? }`, `error`, `gone`. `error`, `gone` and a `failed` cause carry the + wire error format (`WireError`: a ChordError's JSON, or a plugin's code and message). `args` absent is a method without arguments, + `value` absent is `null`; `segments` present marks a payload with byte segments (its method's schema has a `bytes()` + leaf, `Snapshot.bytes`). `member`, `method` and `variant` are indexes into the hold's bound version's names, each + variant's sorted (`variantNames`, `variantOrder` in contracts). Every kind but `hello`, `revoke`, `ping` and `pong` may carry + `origin` (an integer from 1; absent: the sender's own work): a relay's number for one of its downstream origins, + keying holds, calls, windows and acks with the ids (`OriginMessage`; decision "Relay flow control per origin"). + `PROTOCOL_VERSION` is 1. +- `Part = string | Uint8Array | DeflatedPart` (`params`, `args`, `value`, `bound` member values, `state` member ops, + `part.ts`): a value's JSON text at its origin (and after a text decode), its UTF-8 bytes once a binary codec + received it, or those bytes deflated by an origin that compresses. A part is opaque on the way and opened and parsed + only where it is owned, while its message is handled (a malformed part is still a protocol error that closes the + link): the provider opens `params` and `args`, the consumer results, snapshots and ops. A relay forwards parts + unparsed and still deflated; a message naming an unknown id is ignored before its parts are parsed (Q1). +- Parts and their limits (`part.ts`): `originPart(json, compression, ctx)` (a JSON part at its origin, deflated above + 16 KiB of UTF-8, `COMPRESS_ABOVE`, when the host compresses and only if it got smaller, else the bytes encoded for + the attempt; counted `chord.link.deflates`), + `oversized(part, limit)` (the `too_large/part` error of a part past a wire limit: one comparison), + `openPart(part, limits, compression, ctx)` (the JSON its owner parses: the wire size checked, then the decoded size, + a deflated part inflated within it; counted `chord.link.inflates`), `inflateAsync(part, limit, signal, ctx)` (the + `DecompressionStream` fallback, counting output bytes and stopping past the limit; it leaves its tagged `Inflated` + on the part for `openPart`; `signal`, the link's close, cancels it at once), `linkLimits(trusted, limits?)` (none on + trusted links, 16 MiB on the wire and 64 MiB decoded otherwise). Sizes are UTF-8 bytes (a text part's counted + without encoding it); byte segments are not parts and no part limit bounds them (the transport's cap does). +- `Channel`: a host's end of a link. It sends `hello { protocol, incarnation, calls }` first and queues its own messages + until the peer's arrived. Every received message is decoded by the codec and, unless the link is trusted, checked + with `checkMessage` (the compiled `MESSAGE` schema, counted in `chord.link.checks`); a first message that is no + `hello` of this codec ("the first message is no binary message"), another protocol version, a second `hello`, a + failed check, or a throw while its `ChannelHandler` handles a message, is a protocol error: reported + (`protocol_error`), and the link closes. `peer` is the peer's incarnation, `peerCalls` its call window. A handler + that returns a promise (a message waiting for an asynchronous inflate) holds every later message, `ping` included, + until it settles; they are then handled in order, and dropped when the link closes meanwhile; a rejection is a + protocol error. +- The heartbeat, in `Channel`: every received message stamps `lastReceived` (sends do not count); the host's one + interval timer calls `tick(now, ping, close)` on every link: a link silent for `close` ms (30 s by default), the + handshake included, closes; an open one silent for `ping` ms (15 s) sends one `ping`. A `ping` is answered with + `pong` at once, never queued behind the window; neither reaches the handler. It detects a silent transport, not a + peer that answers pings but withholds acks (the window and the pending cap bound that). +- Who checks what (layers 1-3 failing close the link): + 1. the codec's `decode`: lengths and varints fit, the bytes consumed exactly, a known kind and arity, strict UTF-8, + each JSON value parses, segment references valid and only in payloads marked as carrying bytes; + 2. `checkMessage`: the envelope (kinds, ids and indexes, lists, error shapes), not the parts; skipped on a trusted + link; + 3. the remote side (`remote/`): ids not reused while live (unknown ids ignored), indexes within the bound version, + `bound` before `state`, at most N concurrent calls; + 4. owners, where a part is owned: `checkParams`, `checkArgs`, a route's address check, Delta's decoder and + `applyImmutable` before adoption, `chordErrorFromJson`; snapshots and results are never validated. +- Parts (`values.ts`): `jsonPart(value)` (a value's JSON text), `payloadPart(value, bytes)` (a payload's JSON, with + segments and `{"$bytes": i}` references when marked: an own key `$bytes` there fails `invalid/args`; elsewhere it is + plain data), `parsePart(part)` and `parsePayload(part, segments)` (strict UTF-8 JSON; references read back as views + of the message, no copy). +- Errors: `wireError(thrown)` is `{ kind: "chord", error }` for a ChordError, `{ kind: "plugin", code, message, + details? }` for a PluginError, and `{ kind: "plugin", code: "internal", message }` for anything else; never a stack + or a cause. `errorFromWire` rebuilds a ChordError (L1's envelope check; a malformed one throws, a protocol error) or + a PluginError. Failure causes and `gone` carry ChordError JSON directly. diff --git a/packages/chord/src/next/link/binary.ts b/packages/chord/src/next/link/binary.ts new file mode 100644 index 000000000..2786e0601 --- /dev/null +++ b/packages/chord/src/next/link/binary.ts @@ -0,0 +1,565 @@ +// The binary codec (2b): a kind byte (its high bit set when an origin follows as a varint), then the kind's fields as +// unsigned varints, length-prefixed UTF-8 strings, raw incarnation (16 bytes) and fingerprint (8 bytes), parts as +// length-prefixed bytes whose prefix also says whether the part is deflated (decoded as views of the message, never +// parsed or inflated here), and byte segments only beside payloads marked as carrying bytes. No outer length: links +// deliver whole messages. + +import { + type BoundEntry, + type Codec, + type FailedMember, + type Message, + NO_RETIRED, + type StateEntry, +} from "./message.ts"; +import { DeflatedPart, type Part } from "./part.ts"; +import { parseJson, readUtf8 } from "./values.ts"; + +const UTF8 = new TextEncoder(); + +/** Kinds by their byte. */ +const KINDS = [ + "hello", + "hold", + "release", + "call", + "abort", + "bound", + "state", + "failed", + "result", + "error", + "gone", + "retry", + "loading", + "ack", + "closed", + "ping", + "pong", + "revoke", +] as const; + +const KIND_BYTES = new Map(KINDS.map((kind, byte) => [kind, byte])); + +/** The kind byte's flag for an origin. */ +const ORIGIN = 0x80; + +/** A payload's flags: present, and carrying byte segments. */ +const PRESENT = 1; + +const SEGMENTS = 2; + +/** A hold's optional fields, one bit each. */ +const ADDRESS = 1; + +const PRINCIPAL = 2; + +const INCARNATION = 4; + +const CHAIN = 8; + +const LIMIT = 16; + +/** A part's length prefix is its length doubled, plus one when the part is deflated. */ +const DEFLATED = 1; + +/** Messages as bytes; the default codec of every link. */ +export const binaryCodec: Codec = { + name: "binary", + encode(message) { + const writer = new Writer(); + const origin = "origin" in message ? message.origin : undefined; + if (origin === undefined) { + writer.byte(KIND_BYTES.get(message.kind)!); + } else { + writer.byte(KIND_BYTES.get(message.kind)! | ORIGIN); + writer.varint(origin); + } + encodeFields(message, writer); + return writer.finish(); + }, + decode(bytes) { + const reader = new Reader(bytes); + const byte = reader.byte(); + const origin = byte & ORIGIN ? reader.varint() : undefined; + const kind = KINDS[byte & ~ORIGIN]; + if (kind === undefined) throw new Error("binary: an unknown kind"); + const message = decodeFields(kind, reader); + reader.end(); + // A fresh object; the message schema refuses an origin on a kind without one. + if (origin !== undefined) (message as { origin?: number }).origin = origin; + return message; + }, +}; + +function encodeFields(message: Message, writer: Writer): void { + switch (message.kind) { + case "hello": + writer.varint(message.protocol); + writer.raw(hex(message.incarnation.replaceAll("-", ""), 16)); + writer.varint(message.calls); + return; + case "hold": { + writer.varint(message.hold); + writer.string(message.type); + writer.varint(message.version); + writer.raw(hex(message.fingerprint, 8)); + writer.part(message.params); + const { address, principal, incarnation, chain, limit } = message; + writer.byte( + (address === undefined ? 0 : ADDRESS) | + (principal === undefined ? 0 : PRINCIPAL) | + (incarnation === undefined ? 0 : INCARNATION) | + (chain === undefined ? 0 : CHAIN) | + (limit === undefined ? 0 : LIMIT), + ); + if (address !== undefined) { + writer.string(address.kind); + writer.json(address.params); + } + if (principal !== undefined) writer.json(principal); + if (incarnation !== undefined) writer.raw(hex(incarnation.replaceAll("-", ""), 16)); + if (chain !== undefined) { + writer.varint(chain.length); + for (const instance of chain) writer.raw(hex(instance.replaceAll("-", ""), 16)); + } + if (limit !== undefined) writer.varint(limit); + return; + } + case "release": + case "retry": + case "loading": + writer.varint(message.hold); + return; + case "call": + writer.varint(message.call); + writer.varint(message.hold); + writer.varint(message.method); + writer.payload(message.args, message.segments); + return; + case "abort": + writer.varint(message.call); + return; + case "bound": + writer.varint(message.holds.length); + for (const { hold, variant, members, failed } of message.holds) { + writer.varint(hold); + writer.varint(variant); + writer.varint(members.length); + for (const { member, value } of members) { + writer.varint(member); + writer.part(value); + } + writer.failed(failed); + } + return; + case "state": + writer.varint(message.holds.length); + for (const { hold, members, failed, retired } of message.holds) { + writer.varint(hold); + writer.varint(members.length); + for (const { member, ops } of members) { + writer.varint(member); + writer.part(ops); + } + writer.failedAndRetired(failed, retired); + } + return; + case "failed": + writer.varint(message.hold); + writer.byte(message.cause.kind === "transient" ? 0 : 1); + writer.json(message.cause.error); + // 0 when absent, else one more than the delay. + writer.varint(message.retryIn === undefined ? 0 : message.retryIn + 1); + return; + case "result": + writer.varint(message.call); + writer.payload(message.value, message.segments); + return; + case "error": + writer.varint(message.call); + writer.json(message.error); + return; + case "gone": + writer.varint(message.hold); + writer.json(message.error); + return; + case "revoke": + writer.json(message.revocation); + return; + case "ack": + case "closed": + case "ping": + case "pong": + return; + } +} + +/** The kind's fields, read structurally; the envelope's types are checked by the message schema afterwards. */ +function decodeFields(kind: (typeof KINDS)[number], reader: Reader): Message { + // The message schema establishes the types of what is read here (JSON fields are checked by it). + switch (kind) { + case "hello": + return { kind, protocol: reader.varint(), incarnation: uuid(reader.raw(16)), calls: reader.varint() }; + case "hold": { + const hold = reader.varint(); + const type = reader.string(); + const version = reader.varint(); + const fingerprint = hexText(reader.raw(8)); + const params = reader.part(); + const flags = reader.byte(); + if (flags > (ADDRESS | PRINCIPAL | INCARNATION | CHAIN | LIMIT)) throw new Error("binary: unknown hold flags"); + const address = flags & ADDRESS ? { address: { kind: reader.string(), params: reader.json() } } : {}; + const principal = flags & PRINCIPAL ? { principal: reader.json() as { kind: string } } : {}; + const incarnation = flags & INCARNATION ? { incarnation: uuid(reader.raw(16)) } : {}; + const chain = flags & CHAIN ? { chain: reader.list(() => uuid(reader.raw(16))) } : {}; + const limit = flags & LIMIT ? { limit: reader.varint() } : {}; + return { + kind, + hold, + type, + version, + fingerprint, + params, + ...address, + ...principal, + ...incarnation, + ...chain, + ...limit, + }; + } + case "release": + case "retry": + case "loading": + return { kind, hold: reader.varint() }; + case "call": { + const [call, hold, method] = [reader.varint(), reader.varint(), reader.varint()]; + return { kind, call, hold, method, ...reader.payload("args") }; + } + case "abort": + return { kind, call: reader.varint() }; + case "bound": { + const holds = reader.list((): BoundEntry => { + const [hold, variant] = [reader.varint(), reader.varint()]; + const members = reader.list(() => ({ member: reader.varint(), value: reader.part() })); + return { hold, variant, members, failed: reader.failed() }; + }); + return { kind, holds }; + } + case "state": { + const holds = reader.list((): StateEntry => { + const hold = reader.varint(); + const members = reader.list(() => ({ member: reader.varint(), ops: reader.part() })); + // The failure count times two, plus one when retirements follow. + const flags = reader.varint(); + const failed = reader.failedEntries(Math.floor(flags / 2)); + const retired = flags % 2 === 0 ? NO_RETIRED : reader.list(() => reader.varint()); + return { hold, members, failed, retired }; + }); + return { kind, holds }; + } + case "failed": { + const hold = reader.varint(); + const cause = reader.byte(); + if (cause > 1) throw new Error("binary: an unknown failure kind"); + // The schema check after decoding establishes the wire error. + const error = reader.json() as Extract["error"]; + const retryIn = reader.varint(); + return { + kind, + hold, + cause: { kind: cause === 0 ? "transient" : "broken", error }, + ...(retryIn === 0 ? {} : { retryIn: retryIn - 1 }), + }; + } + case "result": + return { kind, call: reader.varint(), ...reader.payload("value") }; + case "error": + return { kind, call: reader.varint(), error: reader.json() as Extract["error"] }; + case "gone": + return { kind, hold: reader.varint(), error: reader.json() as Extract["error"] }; + case "revoke": + return { kind, revocation: reader.json() }; + case "ack": + case "ping": + case "pong": + return { kind }; + case "closed": + // Its origin is the kind byte's; 0 without one, which the message schema refuses. + return { kind, origin: 0 }; + } +} + +/** Up to this many UTF-16 units a string takes at most 126 bytes, so its length is one varint byte. */ +const SHORT = 42; + +/** From this many UTF-16 units a string is encoded in place (below it, encoding on its own and copying is cheaper). */ +const LONG = 4096; + +/** The fixed length prefix of a long string: five varint bytes, up to 32 GiB (16 GiB for a part). */ +const LENGTH_SLOT = 5; + +/** The room left after a large chunk for the fields that follow it. */ +const SPARE = 4096; + +/** Writes a message into one growing buffer, right-sized at the end. */ +class Writer { + /** Large enough for most small messages (a state change, a call), small enough to zero cheaply. */ + #buffer = new Uint8Array(256); + #length = 0; + + byte(byte: number): void { + this.#reserve(1); + this.#buffer[this.#length++] = byte; + } + + /** An unsigned LEB128 varint of a safe integer. */ + varint(value: number): void { + this.#reserve(8); + let rest = value; + while (rest >= 0x80) { + this.#buffer[this.#length++] = (rest % 0x80) | 0x80; + rest = Math.floor(rest / 0x80); + } + this.#buffer[this.#length++] = rest; + } + + raw(bytes: Uint8Array): void { + this.#reserve(bytes.byteLength); + this.#buffer.set(bytes, this.#length); + this.#length += bytes.byteLength; + } + + /** A length-prefixed UTF-8 string (`#text`). */ + string(text: string): void { + this.#text(text, 1); + } + + json(value: unknown): void { + this.string(JSON.stringify(value) ?? "null"); + } + + /** + * A part: its text encoded in place, or its bytes (a part received) copied as they are, behind its length doubled, + * plus one when deflated. + */ + part(part: Part): void { + if (typeof part === "string") { + this.#text(part, 2); + return; + } + const bytes = part instanceof DeflatedPart ? part.bytes : part; + this.varint(bytes.byteLength * 2 + (part instanceof DeflatedPart ? DEFLATED : 0)); + this.raw(bytes); + } + + /** + * Text behind its UTF-8 length times `scale` (2 for a part's prefix), encoded straight into the buffer where the + * prefix is known in advance: a short text (its prefix fits one byte: at most 126 bytes, 42 UTF-16 units, divided + * by `scale`) and a long one (a snapshot's JSON, behind a fixed five-byte prefix, a padded varint, which readers + * accept). Others are encoded on their own and copied in. No text is encoded twice, and the largest value of a + * message is never copied. + */ + #text(text: string, scale: 1 | 2): void { + if (text.length <= SHORT / scale) { + this.#reserve(1 + text.length * 3); + const { written } = UTF8.encodeInto(text, this.#buffer.subarray(this.#length + 1)); + this.#buffer[this.#length] = written * scale; + this.#length += 1 + written; + return; + } + if (text.length < LONG) { + const bytes = UTF8.encode(text); + this.varint(bytes.byteLength * scale); + this.raw(bytes); + return; + } + const slot = this.#length; + // At least one byte per UTF-16 unit; more room is made when the text is not all ASCII. + this.#reserve(LENGTH_SLOT + text.length); + this.#length += LENGTH_SLOT; + let read = 0; + while (read < text.length) { + const rest = read === 0 ? text : text.slice(read); + const done = UTF8.encodeInto(rest, this.#buffer.subarray(this.#length)); + read += done.read; + this.#length += done.written; + if (read < text.length) this.#reserve(Math.max((text.length - read) * 2, 4)); + } + let length = (this.#length - slot - LENGTH_SLOT) * scale; + for (let index = 0; index < LENGTH_SLOT; index++) { + this.#buffer[slot + index] = (length % 0x80) | (index < LENGTH_SLOT - 1 ? 0x80 : 0); + length = Math.floor(length / 0x80); + } + } + + /** A payload: its flags, then its part and, when marked, its segments (count, lengths, raw bytes). */ + payload(part: Part | undefined, segments: readonly Uint8Array[] | undefined): void { + if (part === undefined) { + this.varint(0); + return; + } + this.varint(segments === undefined ? PRESENT : PRESENT | SEGMENTS); + this.part(part); + if (segments === undefined) return; + this.varint(segments.length); + for (const segment of segments) this.varint(segment.byteLength); + for (const segment of segments) this.raw(segment); + } + + failed(failed: readonly FailedMember[]): void { + this.varint(failed.length); + this.#failedEntries(failed); + } + + /** + * A state entry's failures and retirements: the failure count times two, plus one when retirements follow, so an + * entry without retirements costs no byte more than its failures. + */ + failedAndRetired(failed: readonly FailedMember[], retired: readonly number[]): void { + this.varint(failed.length * 2 + (retired.length > 0 ? 1 : 0)); + this.#failedEntries(failed); + if (retired.length === 0) return; + this.varint(retired.length); + for (const member of retired) this.varint(member); + } + + #failedEntries(failed: readonly FailedMember[]): void { + for (const { member, error } of failed) { + this.varint(member); + this.json(error); + } + } + + /** + * The message: a view of the buffer, or a right-sized copy when the spare room is meaningful (at least 64 bytes and + * more than an eighth of the message), so a kept message keeps at most that much beyond itself. A large message + * ends within its 4 KiB of room and is not copied. + */ + finish(): Uint8Array { + const message = this.#buffer.subarray(0, this.#length); + const spare = this.#buffer.byteLength - this.#length; + return spare >= 64 && spare * 8 > this.#length ? message.slice() : message; + } + + /** Grows the buffer to fit `bytes` more: doubled, or for a large chunk to its end plus room for the small fields after it. */ + #reserve(bytes: number): void { + if (this.#length + bytes <= this.#buffer.byteLength) return; + const grown = new Uint8Array(Math.max(this.#buffer.byteLength * 2, this.#length + bytes + SPARE)); + grown.set(this.#buffer.subarray(0, this.#length)); + this.#buffer = grown; + } +} + +/** Reads a message's bytes; every read is bounds-checked, segments are views of the message. */ +class Reader { + readonly #bytes: Uint8Array; + #offset = 0; + + constructor(bytes: Uint8Array) { + this.#bytes = bytes; + } + + byte(): number { + const byte = this.#bytes[this.#offset]; + if (byte === undefined) throw new Error("binary: the message ends early"); + this.#offset++; + return byte; + } + + /** An unsigned LEB128 varint of at most 8 bytes, at most a safe integer. */ + varint(): number { + let value = 0; + let scale = 1; + for (let read = 1; ; read++) { + if (read > 8) throw new Error("binary: a varint does not fit"); + const byte = this.byte(); + value += (byte & 0x7f) * scale; + if (value > Number.MAX_SAFE_INTEGER) throw new Error("binary: a varint does not fit"); + if (byte < 0x80) return value; + scale *= 0x80; + } + } + + raw(length: number): Uint8Array { + if (length > this.#bytes.byteLength - this.#offset) throw new Error("binary: the message ends early"); + const bytes = this.#bytes.subarray(this.#offset, this.#offset + length); + this.#offset += length; + return bytes; + } + + string(): string { + return readUtf8(this.raw(this.varint())); + } + + json(): unknown { + return parseJson(this.raw(this.varint())); + } + + /** A part as a view of the message, unparsed and not inflated. */ + part(): Uint8Array | DeflatedPart { + const prefix = this.varint(); + const bytes = this.raw(Math.floor(prefix / 2)); + return prefix % 2 === DEFLATED ? new DeflatedPart(bytes) : bytes; + } + + /** A list: its count (each element takes at least a byte, so it cannot pass the message's end), then its elements. */ + list(read: () => T): T[] { + const count = this.varint(); + if (count > this.#bytes.byteLength - this.#offset) throw new Error("binary: a count does not fit"); + return Array.from({ length: count }, read); + } + + /** A payload part under `key`, absent when not present, with its segments when marked as carrying them. */ + payload(key: K): { [P in K]?: Part } & { segments?: Uint8Array[] } { + const flags = this.varint(); + if (flags === 0) return {}; + if (flags !== PRESENT && flags !== (PRESENT | SEGMENTS)) throw new Error("binary: unknown payload flags"); + const part = this.part(); + if (flags === PRESENT) return { [key]: part } as { [P in K]?: Part }; + const lengths = this.list(() => this.varint()); + const segments = lengths.map((length) => this.raw(length)); + return { [key]: part, segments } as { [P in K]?: Part } & { segments: Uint8Array[] }; + } + + failed(): FailedMember[] { + return this.failedEntries(this.varint()); + } + + /** `count` failed members, as `Writer.failed` writes them after their count. */ + failedEntries(count: number): FailedMember[] { + if (count > this.#bytes.byteLength - this.#offset) throw new Error("binary: a count does not fit"); + // The message schema checks each error's shape. + return Array.from({ length: count }, () => ({ + member: this.varint(), + error: this.json() as Extract["error"], + })); + } + + /** The message is consumed exactly. */ + end(): void { + if (this.#offset !== this.#bytes.byteLength) throw new Error("binary: bytes after the message"); + } +} + +/** Lowercase hex of 8 and 16 bytes: a fingerprint, an incarnation without its dashes. */ +const HEX = new Map([ + [8, /^[0-9a-f]{16}$/], + [16, /^[0-9a-f]{32}$/], +]); + +/** `length` bytes from lowercase hex (an incarnation without dashes, a fingerprint). */ +function hex(text: string, length: number): Uint8Array { + if (!HEX.get(length)!.test(text)) throw new Error(`binary: not ${length} bytes of hex`); + return Uint8Array.from({ length }, (_, index) => Number.parseInt(text.slice(index * 2, index * 2 + 2), 16)); +} + +function hexText(bytes: Uint8Array): string { + return Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join(""); +} + +/** 16 bytes as a UUID's text. */ +function uuid(bytes: Uint8Array): string { + const text = hexText(bytes); + return `${text.slice(0, 8)}-${text.slice(8, 12)}-${text.slice(12, 16)}-${text.slice(16, 20)}-${text.slice(20)}`; +} diff --git a/packages/chord/src/next/link/channel.ts b/packages/chord/src/next/link/channel.ts new file mode 100644 index 000000000..14a43c216 --- /dev/null +++ b/packages/chord/src/next/link/channel.ts @@ -0,0 +1,243 @@ +// A host's end of a link (§3, §22, 2b): the handshake, the link's codec, the checks of every message (skipped on a +// trusted link), the heartbeat, and messages handled in order while one waits for an asynchronous inflate; a protocol +// error closes the link. + +import type { Context } from "../../context/index.ts"; +import { PROTOCOL_VERSION } from "../data/internal.ts"; +import type { Link } from "./link.ts"; +import { type Codec, checkMessage, type Message } from "./message.ts"; + +/** What the owner of a channel handles. */ +export interface ChannelHandler { + /** The peer's `hello` arrived: the link is open. */ + opened(): void; + /** + * A checked message after the handshake (never `hello`, `ping` or `pong`). A throw is a protocol error. A promise + * (the message waits for an asynchronous inflate) holds every later message until it settles; a rejection is a + * protocol error too. + */ + message(message: Message): void | Promise; + /** The link closed, from either end; called once. */ + closed(): void; +} + +/** What a channel is made with besides its link and handler. */ +type ChannelOptions = { + readonly codec: Codec; + /** This host's incarnation, sent in `hello`. */ + readonly incarnation: string; + /** How many concurrent calls this side accepts, sent in `hello`. */ + readonly calls: number; + /** Skips the `MESSAGE` check (framing, the codec's decode and the owners' checks still run). */ + readonly trusted: boolean; + /** Where protocol errors go. */ + readonly report: (error: unknown) => void; +}; + +/** + * Where a channel stands: `greeting` until the peer's `hello`, with what this end sent meanwhile; `open` with the + * peer's incarnation and call window. + */ +type ChannelState = + | { readonly kind: "greeting"; readonly queued: Uint8Array[] } + | { readonly kind: "open"; readonly peer: string; readonly calls: number } + | { readonly kind: "closed" }; + +/** + * One end of a link. It sends `hello { protocol, incarnation, calls }` first and nothing else until the peer's arrived + * (its own messages wait). Every message is decoded by the link's codec and, unless the link is trusted, checked + * against the message schema; a first message that is no `hello` of this codec, a second `hello`, another protocol + * version, or a message failing a check closes the link and is reported. The heartbeat: every received message + * stamps `lastReceived`; the host's timer calls `tick`, which pings a silent open link and closes a link silent for + * too long (the handshake included); a `ping` is answered with `pong` at once. A message the handler handles + * asynchronously holds the later ones, `ping` included, until it settles; they are handled in order then, and dropped + * when the link closes meanwhile. + */ +export class Channel { + readonly #link: Link; + readonly #codec: Codec; + readonly #handler: ChannelHandler; + readonly #report: (error: unknown) => void; + readonly #trusted: boolean; + readonly #context: Context; + #state: ChannelState = { kind: "greeting", queued: [] }; + /** When the last message arrived (`Date.now()`), or the channel was made. */ + #lastReceived = Date.now(); + /** Whether a `ping` went out since the last received message. */ + #pinged = false; + /** + * Messages received while one is handled asynchronously, in order, and the index of the next to handle; `undefined` + * while none waits. + */ + #held: { queue: Uint8Array[]; next: number } | undefined; + /** Whether a message is being handled asynchronously: the held ones wait. */ + #waiting = false; + readonly #hello: Uint8Array; + + constructor(link: Link, handler: ChannelHandler, options: ChannelOptions, context: Context) { + this.#link = link; + this.#codec = options.codec; + this.#handler = handler; + this.#report = options.report; + this.#trusted = options.trusted; + this.#context = context; + this.#hello = this.#codec.encode({ + kind: "hello", + protocol: PROTOCOL_VERSION, + incarnation: options.incarnation, + calls: options.calls, + }); + } + + /** + * Sends `hello` and starts listening: messages that arrived already are handled now, and a link closed already + * calls `closed` now; so the handler is complete before it starts. + */ + start(): void { + this.#link.send(this.#hello); + this.#link.onMessage((bytes) => this.#receive(bytes)); + this.#link.onClose(() => this.#closed()); + } + + /** The peer's incarnation, from its `hello`; empty before it arrived (nothing else can arrive before it). */ + get peer(): string { + return this.#state.kind === "open" ? this.#state.peer : ""; + } + + /** How many concurrent calls the peer accepts, from its `hello`; 0 before it arrived. */ + get peerCalls(): number { + return this.#state.kind === "open" ? this.#state.calls : 0; + } + + get opened(): boolean { + return this.#state.kind === "open"; + } + + get closed(): boolean { + return this.#state.kind === "closed"; + } + + /** Sends one message; before the handshake it waits, after the close it is dropped. */ + send(message: Message): void { + const state = this.#state; + if (state.kind === "closed") return; + const bytes = this.#codec.encode(message); + if (state.kind === "greeting") state.queued.push(bytes); + else this.#link.send(bytes); + } + + close(): void { + this.#link.close(); + } + + /** + * The heartbeat, at `now`: a link silent for `close` ms (the handshake included) closes; an open one silent for + * `ping` ms sends one `ping`. + */ + tick(now: number, ping: number, close: number): void { + const silent = now - this.#lastReceived; + if (silent >= close) this.close(); + else if (silent >= ping && !this.#pinged && this.#state.kind === "open") { + this.#pinged = true; + this.#link.send(this.#codec.encode({ kind: "ping" })); + } + } + + #receive(bytes: Uint8Array): void { + if (this.#state.kind === "closed") return; + this.#lastReceived = Date.now(); + this.#pinged = false; + if (this.#held === undefined) this.#handle(bytes); + else this.#held.queue.push(bytes); + } + + /** Handles one received message; one the handler handles asynchronously holds the next until it settles. */ + #handle(bytes: Uint8Array): void { + const state = this.#state; + try { + if (state.kind === "greeting") { + this.#greet(bytes, state.queued); + return; + } + const message = this.#decode(bytes); + if (message.kind === "hello") throw new Error("handshake: a second hello"); + // Control of the link itself, never queued behind the window. + if (message.kind === "ping") this.#link.send(this.#codec.encode({ kind: "pong" })); + else if (message.kind !== "pong") { + const handling = this.#handler.message(message); + if (handling !== undefined) this.#hold(handling); + } + } catch (error) { + this.#failed(error); + } + } + + /** Holds later messages until `handling` settles, then handles them in order (each may hold again). */ + #hold(handling: Promise): void { + this.#held ??= { queue: [], next: 0 }; + this.#waiting = true; + handling.then( + () => { + this.#waiting = false; + this.#release(); + }, + (error: unknown) => this.#failed(error), + ); + } + + /** + * The held message was handled: the ones behind it are handled in order, until one holds again (the handled ones are + * dropped from the queue then). + */ + #release(): void { + const held = this.#held; + if (held === undefined) return; + while (held.next < held.queue.length) { + if (this.#state.kind !== "open") return; + this.#handle(held.queue[held.next++]!); + if (this.#waiting) { + held.queue.splice(0, held.next); + held.next = 0; + return; + } + } + this.#held = undefined; + } + + /** A protocol error: reported, and the link closes. */ + #failed(error: unknown): void { + if (this.#state.kind === "closed") return; + this.#report(error); + this.close(); + } + + /** A message from its bytes, checked against the message schema unless the link is trusted. */ + #decode(bytes: Uint8Array): Message { + const message = this.#codec.decode(bytes); + return this.#trusted ? message : checkMessage(message, this.#context); + } + + /** The first message: the peer's `hello` in this link's codec and protocol version. */ + #greet(bytes: Uint8Array, queued: readonly Uint8Array[]): void { + let message: Message; + try { + message = this.#decode(bytes); + } catch (error) { + throw new Error(`handshake: the first message is no ${this.#codec.name} message`, { cause: error }); + } + if (message.kind !== "hello") throw new Error(`handshake: ${message.kind} before hello`); + if (message.protocol !== PROTOCOL_VERSION) { + throw new Error(`handshake: protocol ${message.protocol}, this host speaks ${PROTOCOL_VERSION}`); + } + this.#state = { kind: "open", peer: message.incarnation, calls: message.calls }; + for (const bytes of queued) this.#link.send(bytes); + this.#handler.opened(); + } + + #closed(): void { + if (this.#state.kind === "closed") return; + this.#state = { kind: "closed" }; + this.#held = undefined; + this.#handler.closed(); + } +} diff --git a/packages/chord/src/next/link/errors.ts b/packages/chord/src/next/link/errors.ts new file mode 100644 index 000000000..cb650549e --- /dev/null +++ b/packages/chord/src/next/link/errors.ts @@ -0,0 +1,23 @@ +// Errors on the wire: a ChordError's JSON or a plugin's code, message and details; never a stack or a cause. + +import type { JsonValue } from "../../json.ts"; +import { type ChordError, isChordError, PluginError } from "../data/index.ts"; +import { chordErrorFromJson } from "../data/internal.ts"; +import type { WireError } from "./message.ts"; + +/** What crosses for a thrown value: a ChordError, a PluginError, or anything else as the plugin code `internal`. */ +export function wireError(error: unknown): WireError { + if (isChordError(error)) return { kind: "chord", error: error.toJSON() }; + if (error instanceof PluginError) { + const { code, message, details } = error; + return { kind: "plugin", code, message, ...(details === undefined ? {} : { details }) }; + } + return { kind: "plugin", code: "internal", message: error instanceof Error ? error.message : String(error) }; +} + +/** The error a consumer rethrows. A malformed ChordError envelope throws (a protocol error). */ +export function errorFromWire(error: WireError): ChordError | PluginError { + if (error.kind === "chord") return chordErrorFromJson(error.error); + // Fresh JSON from the link: details are JSON. + return new PluginError(error.code, error.message, error.details as JsonValue | undefined); +} diff --git a/packages/chord/src/next/link/index.ts b/packages/chord/src/next/link/index.ts new file mode 100644 index 000000000..e2dac1d96 --- /dev/null +++ b/packages/chord/src/next/link/index.ts @@ -0,0 +1,10 @@ +// Links for apps: the link interface adapters implement, the in-memory pair, IPC and WebSocket links, the codecs a link +// may use, its limits per part and the compressor a host may bring. + +export { binaryCodec } from "./binary.ts"; +export { type IpcPort, ipcLink } from "./ipc.ts"; +export { type Link, memoryLinkPair } from "./link.ts"; +export type { Codec, Message } from "./message.ts"; +export type { Compression, Inflated, PartLimits } from "./part.ts"; +export { describeMessage, textCodec } from "./text.ts"; +export { type WebSocketLike, webSocketLink } from "./websocket.ts"; diff --git a/packages/chord/src/next/link/internal.ts b/packages/chord/src/next/link/internal.ts new file mode 100644 index 000000000..e3026f2a8 --- /dev/null +++ b/packages/chord/src/next/link/internal.ts @@ -0,0 +1,35 @@ +// Links for Chord's layers: a host's end of a link (handshake, codec, checks, heartbeat and in-order handling), the +// message type, parts with their limits and compression, and errors in messages. + +export { binaryCodec } from "./binary.ts"; +export { Channel, type ChannelHandler } from "./channel.ts"; +export { errorFromWire, wireError } from "./errors.ts"; +export type { Link } from "./link.ts"; +export { + type BoundEntry, + CHAIN, + type Codec, + type ConsumerMessage, + checkMessage, + type FailedMember, + type Message, + NO_RETIRED, + type OriginMessage, + type ProviderMessage, + type StateEntry, + type WireAddress, +} from "./message.ts"; +export { + type Compression, + DeflatedPart, + inflateAsync, + type Limits, + linkLimits, + openPart, + originPart, + oversized, + type Part, + type PartLimits, +} from "./part.ts"; +export { describeMessage, textCodec } from "./text.ts"; +export { jsonPart, parsePart, parsePayload, payloadPart } from "./values.ts"; diff --git a/packages/chord/src/next/link/ipc.ts b/packages/chord/src/next/link/ipc.ts new file mode 100644 index 000000000..1c5f1137b --- /dev/null +++ b/packages/chord/src/next/link/ipc.ts @@ -0,0 +1,78 @@ +// IPC links (§3): a link over a Node IPC channel, a forked child (`fork` with `serialization: "advanced"`) or +// `process` in the child, typed structurally so Chord imports nothing from Node. + +import { type Link, LinkEnd } from "./link.ts"; + +/** + * The part of a Node IPC channel a link uses: a `ChildProcess` or the child's `process`. `send` returning `false` is + * backpressure (the message is queued by Node), not a failure. + */ +export interface IpcPort { + send(message: Uint8Array, callback: (error: Error | null) => void): boolean; + on(event: "message", listener: (message: unknown) => void): unknown; + on(event: "disconnect", listener: () => void): unknown; + removeListener(event: "message", listener: (message: unknown) => void): unknown; + removeListener(event: "disconnect", listener: () => void): unknown; + disconnect(): void; + readonly connected: boolean; +} + +/** + * A link over `port`: each message one `Uint8Array` (V8 serialization keeps it one). It listens from the moment it is + * made and queues messages until its listener is set, so a child makes it before awaiting its start message; messages + * that are no `Uint8Array` (the app's own, such as that start message) are not the link's and are skipped. A thrown + * `send`, an error in its callback or `disconnect` closes the link once; its listeners are removed then. + */ +export function ipcLink(port: IpcPort): Link { + return new IpcLink(port); +} + +class IpcLink extends LinkEnd { + /** The port while the link is open. */ + #port: IpcPort | undefined; + readonly #onMessage = (message: unknown): void => { + if (message instanceof Uint8Array) this.deliver(message); + }; + readonly #onDisconnect = (): void => this.#end(); + readonly #onSent = (error: Error | null): void => { + if (error !== null) this.#end(); + }; + + constructor(port: IpcPort) { + super(); + this.#port = port; + if (!port.connected) { + this.#end(); + return; + } + port.on("message", this.#onMessage); + port.on("disconnect", this.#onDisconnect); + } + + send(message: Uint8Array): void { + const port = this.#port; + if (port === undefined) return; + try { + port.send(message, this.#onSent); + } catch { + this.#end(); + } + } + + close(): void { + const port = this.#port; + if (port === undefined) return; + this.#end(); + // Disconnecting a channel that is gone already would emit an error. + if (port.connected) port.disconnect(); + } + + /** Detaches from the port and closes this end. */ + #end(): void { + const port = this.#port; + this.#port = undefined; + port?.removeListener("message", this.#onMessage); + port?.removeListener("disconnect", this.#onDisconnect); + this.ended(); + } +} diff --git a/packages/chord/src/next/link/link.ts b/packages/chord/src/next/link/link.ts new file mode 100644 index 000000000..0678cdca1 --- /dev/null +++ b/packages/chord/src/next/link/link.ts @@ -0,0 +1,106 @@ +// Links (§3): an ordered, bidirectional channel of messages between two hosts, and the in-memory pair. + +/** An ordered, bidirectional channel of `Uint8Array` messages between two hosts. Adapters only move bytes. */ +export interface Link { + /** Sends one message; a no-op once closed. */ + send(message: Uint8Array): void; + /** + * Sets the one listener for incoming messages; messages that arrived earlier are delivered to it first. On a closed + * link it is not kept. + */ + onMessage(listener: (message: Uint8Array) => void): void; + /** Closes both ends; messages not delivered yet are dropped. */ + close(): void; + /** Sets the one listener for the close of either end; it runs at once when the link is closed already. */ + onClose(listener: () => void): void; +} + +/** Two ends of one link: what one sends, the other receives asynchronously (a microtask later), in order. */ +export function memoryLinkPair(): [Link, Link] { + const a = new MemoryLink(); + const b = new MemoryLink(); + a.peer = b; + b.peer = a; + return [a, b]; +} + +/** + * What every link end shares: the one message listener (messages that arrived before it are queued for it), the one + * close listener, and closing once. Subclasses move the bytes: `deliver` what arrived, `ended` when the transport + * closed, and implement `send` and `close`. + */ +export abstract class LinkEnd implements Link { + #closed = false; + #listener: ((message: Uint8Array) => void) | undefined; + /** Messages that arrived before the listener was set. */ + #queued: Uint8Array[] = []; + #closeListener: (() => void) | undefined; + + abstract send(message: Uint8Array): void; + + abstract close(): void; + + get closed(): boolean { + return this.#closed; + } + + onMessage(listener: (message: Uint8Array) => void): void { + // A closed end delivers nothing: keeping the listener would keep its owner. + if (this.#closed) return; + this.#listener = listener; + // A listener may close the link: what is still queued then is dropped. + for (const message of this.#queued.splice(0)) { + if (this.#closed) return; + listener(message); + } + } + + onClose(listener: () => void): void { + if (this.#closed) listener(); + else this.#closeListener = listener; + } + + /** A message arrived; dropped once closed. */ + protected deliver(message: Uint8Array): void { + if (this.#closed) return; + if (this.#listener === undefined) this.#queued.push(message); + else this.#listener(message); + } + + /** Closes this end (once): nothing is delivered any more, and nothing the listeners reach is kept. */ + protected ended(): void { + if (this.#closed) return; + this.#closed = true; + const listener = this.#closeListener; + this.#listener = undefined; + this.#closeListener = undefined; + this.#queued = []; + listener?.(); + } +} + +/** + * One end of a memory link. Closing it closes this end at once and the peer a microtask later, as a transport would. + * Portable: `queueMicrotask` only. + */ +class MemoryLink extends LinkEnd { + peer: MemoryLink | undefined; + + send(message: Uint8Array): void { + if (this.closed) return; + const peer = this.peer!; + queueMicrotask(() => peer.deliver(message)); + } + + close(): void { + if (this.closed) return; + const peer = this.peer!; + this.#end(); + queueMicrotask(() => peer.#end()); + } + + #end(): void { + this.peer = undefined; + this.ended(); + } +} diff --git a/packages/chord/src/next/link/message.ts b/packages/chord/src/next/link/message.ts new file mode 100644 index 000000000..a8c022571 --- /dev/null +++ b/packages/chord/src/next/link/message.ts @@ -0,0 +1,182 @@ +// Messages (§22, R1, R4, 2b): the one `Message` type every link carries, defined as a TypeBox schema and checked after +// every decode, and the codecs that turn it into bytes. + +import Type from "typebox"; +import type { Context } from "../../context/index.ts"; +import { arr, int, lit, obj, opt, type Static, str, union, unknown } from "../contracts/index.ts"; +import { validatorFor } from "../data/internal.ts"; +import type { Part } from "./part.ts"; + +/** A link-local id, chosen by its origin: a counter, unique per link, direction and origin while in use. */ +const id = int({ min: 0 }); + +/** + * Whose work a message is (decision "Relay flow control per origin"): a relay's number for one downstream link (and + * the origin that link's messages carried), never reused on a link; absent for the sender's own work. Holds, calls, + * windows and acks are keyed by (origin, id), and replies carry the origin of what they answer. + */ +const origin = int({ min: 1 }); + +/** A name as an index into the hold's bound version (each variant's names sorted). */ +const index = int({ min: 0 }); + +/** An error that may come from plugin code: a ChordError's JSON (checked by L1), or a plugin's code and message. */ +const wireError = union([ + obj({ kind: lit("chord"), error: unknown() }), + obj({ kind: lit("plugin"), code: str(), message: str(), details: opt(unknown()) }), +]); + +/** The members of a hold that failed with this message (W1). */ +const failed = arr(obj({ member: index, error: wireError })); + +/** A part's envelope: the codec built it, so any value passes (its content is the owner's to parse). */ +const part = Type.Unsafe({}); + +/** + * The byte segments of a payload part whose method's schema has a `bytes()` leaf, referenced from its JSON by + * `{"$bytes": i}`; absent for every other payload. Built by the codec, like the part. + */ +const segments = opt(Type.Unsafe({})); + +/** An address as a hold names it: its host kind and params, checked against the kind where it is routed. */ +const address = obj({ kind: str(), params: unknown() }); + +/** A principal on a hold: JSON with a `kind`, honoured only on a parent link. */ +const principal = Type.Object({ kind: str() }); + +/** A uuidv7 as text. */ +const incarnation = str({ pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$" }); + +/** The longest activation chain a hold carries (Q5): a longer one keeps its last entries. */ +export const CHAIN = 32; + +/** The protocol's messages, one tagged union. Parts are opaque here (the owners parse them). */ +const MESSAGE = union([ + // `calls`: how many concurrent calls this side accepts (the call window). + obj({ kind: lit("hello"), protocol: int({ min: 0 }), incarnation, calls: int({ min: 1 }) }), + obj({ + kind: lit("hold"), + origin: opt(origin), + hold: id, + type: str(), + version: int({ min: 1 }), + fingerprint: str({ pattern: "^[0-9a-f]{16}$" }), + params: part, + // The address the caller computed (absent for a type without placement). + address: opt(address), + // Whom the hold acts for and the origin's incarnation; read only on a parent link. + principal: opt(principal), + incarnation: opt(incarnation), + // The instance ids of the activations waiting for this hold (Q5); read only on a trusted link. + chain: opt(Type.Array(incarnation, { maxItems: CHAIN })), + // The decoded limit per part of the link the hold was admitted on (Q2); read only on a parent link. + limit: opt(int({ min: 0 })), + }), + obj({ kind: lit("release"), origin: opt(origin), hold: id }), + obj({ kind: lit("call"), origin: opt(origin), call: id, hold: id, method: index, args: opt(part), segments }), + obj({ kind: lit("abort"), origin: opt(origin), call: id }), + // Asks the provider for an attempt of a failed hold's object now (`ref.retry()`). + obj({ kind: lit("retry"), origin: opt(origin), hold: id }), + // Binds several holds at once (first bindings, re-holds), applied in one batch. + obj({ + kind: lit("bound"), + origin: opt(origin), + holds: arr(obj({ hold: id, variant: index, members: arr(obj({ member: index, value: part })), failed })), + }), + obj({ + kind: lit("state"), + origin: opt(origin), + // `retired`: the members whose document was retired since the hold's last message (decision "One retirement + // mechanism"); a `bound` snapshot carries no flag. + holds: arr(obj({ hold: id, members: arr(obj({ member: index, ops: part })), failed, retired: arr(index) })), + }), + // One per applied `bound` or `state` message of its origin (the window). + obj({ kind: lit("ack"), origin: opt(origin) }), + // A relay's downstream link closed: the origin's holds are released and its calls aborted. + obj({ kind: lit("closed"), origin }), + // A revocation from the parent, applied with this host's own `revokedBy` (no origin). + obj({ kind: lit("revoke"), revocation: unknown() }), + obj({ kind: lit("ping") }), + obj({ kind: lit("pong") }), + obj({ + kind: lit("failed"), + origin: opt(origin), + hold: id, + // A failure cause's error in the wire error format (a ChordError, checked by L1's envelope check, or a plugin's). + cause: obj({ kind: union([lit("transient"), lit("broken")]), error: wireError }), + // The rest of the object's backoff window in milliseconds: when it retries by itself (absent: broken). + retryIn: opt(int({ min: 0 })), + }), + // The failed object of a hold started a new attempt. + obj({ kind: lit("loading"), origin: opt(origin), hold: id }), + obj({ kind: lit("result"), origin: opt(origin), call: id, value: opt(part), segments }), + obj({ kind: lit("error"), origin: opt(origin), call: id, error: wireError }), + obj({ kind: lit("gone"), origin: opt(origin), hold: id, error: wireError }), +]); + +/** + * A protocol message. `args` absent: a method without arguments; `value` absent: `null`. A `state` entry's `retired` + * list is read-only (the schema's type is mutable): entries without retirements share `NO_RETIRED`. + */ +export type Message = + | Exclude + | (Omit & { readonly holds: readonly StateEntry[] }); + +type WireMessage = Static; + +type WireState = Extract; + +/** An error as it crosses a link. */ +export type WireError = Static; + +/** A message that may carry an origin (every kind but `hello`, `revoke`, `ping` and `pong`). */ +export type OriginMessage = Exclude; + +/** A message for a provider side: the direction from the consumer (`closed` is the link's). */ +export type ProviderMessage = Extract< + Message, + { readonly kind: "hold" | "release" | "call" | "abort" | "retry" | "ack" } +>; + +/** A message for a consumer side: the direction from the provider. */ +export type ConsumerMessage = Extract< + Message, + { readonly kind: "bound" | "state" | "failed" | "loading" | "result" | "error" | "gone" } +>; + +/** One hold's entry of a `bound` message. */ +export type BoundEntry = Extract["holds"][number]; + +/** One hold's entry of a `state` message. */ +export type StateEntry = Omit & { readonly retired: readonly number[] }; + +/** The `retired` list of a state entry without retirements, shared (no allocation per message). */ +export const NO_RETIRED: readonly number[] = []; + +/** An address as a `hold` carries it. */ +export type WireAddress = Static; + +/** A member of a hold that failed, by index, as `bound` and `state` list it. */ +export type FailedMember = Static[number]; + +/** How a link turns messages into bytes and back; both ends of a link use the same one. */ +export interface Codec { + readonly name: string; + encode(message: Message): Uint8Array; + /** Throws on bytes that are no message of this codec (a protocol error); the envelope is checked afterwards. */ + decode(bytes: Uint8Array): Message; +} + +const MESSAGE_SCHEMA = { schema: MESSAGE }; + +/** + * Checks a decoded message's envelope with the compiled schema (the wire is a trust boundary, after any codec): kinds, + * ids and indexes, lists and error shapes; not the parts. Throws on a mismatch (a protocol error). Counted in + * `chord.link.checks`. + */ +export function checkMessage(message: unknown, context: Context): Message { + context.telemetry.metrics?.counter("chord.link.checks").add(1); + if (!validatorFor(MESSAGE_SCHEMA, context).Check(message)) throw new Error("message: fails the message schema"); + // The check above establishes the type. + return message as Message; +} diff --git a/packages/chord/src/next/link/part.ts b/packages/chord/src/next/link/part.ts new file mode 100644 index 000000000..d0daeb45a --- /dev/null +++ b/packages/chord/src/next/link/part.ts @@ -0,0 +1,208 @@ +// Parts on links (R1, R2, Q2, §22): a value as it crosses a link (JSON text at its origin, its bytes once received, or +// deflated by its origin), the size limits per part, compression at the origin, and opening a part where it is owned: +// its size checked before anything is parsed or inflated, a deflated part inflated within the decoded limit. + +import type { Context } from "../../context/index.ts"; +import { ChordError } from "../data/index.ts"; + +/** + * A part (R1): a value as JSON text at its origin (and after a text decode), its UTF-8 bytes once a binary codec + * received it (or at an origin that encoded it already), or those bytes deflated by an origin that compresses. Opaque + * on the way: only the end that owns it opens and parses it, while handling its message (a malformed part is a protocol + * error then), so a relay forwards it unparsed and still deflated. + */ +export type Part = string | Uint8Array | DeflatedPart; + +/** What inflating a deflated part gave: its JSON bytes, more than the decoded limit, or data that is no raw deflate. */ +export type Inflated = + | { readonly kind: "bytes"; readonly bytes: Uint8Array } + | { readonly kind: "too_large" } + | { readonly kind: "corrupt"; readonly error: unknown }; + +/** + * A JSON part deflated (raw deflate) by its origin. A host without a synchronous compressor inflates it before its + * owner handles the message (`inflateAsync`) and keeps the outcome here for the owner to open. + */ +export class DeflatedPart { + readonly bytes: Uint8Array; + /** The asynchronous inflate's outcome; `undefined` until one ran. */ + inflated: Inflated | undefined; + + constructor(bytes: Uint8Array) { + this.bytes = bytes; + } +} + +/** + * A synchronous compressor (`createHost({ compression })`; `nodeCompression` for Node). `deflate` is raw deflate at + * level 1. `inflate` gives at most `limit` bytes (`Infinity`: no limit; 0 allowed), `too_large` when the output would + * be larger, `corrupt` for data that is no raw deflate stream. + */ +export interface Compression { + deflate(bytes: Uint8Array): Uint8Array; + inflate(bytes: Uint8Array, limit: number): Inflated; +} + +/** A link's limits per part as `accept`, `connect` and `resolve` take them, in bytes. */ +export type PartLimits = { + /** A part's size on the wire. */ + readonly part?: number; + /** A part's size after inflating it. */ + readonly decoded?: number; +}; + +/** A link's limits in force; `Infinity` for none. */ +export type Limits = { readonly part: number; readonly decoded: number }; + +const MiB = 1024 * 1024; + +/** The limits of untrusted links unless set (Q2); trusted links (IPC) have none. */ +const UNTRUSTED: Limits = { part: 16 * MiB, decoded: 64 * MiB }; + +const NONE: Limits = { part: Number.POSITIVE_INFINITY, decoded: Number.POSITIVE_INFINITY }; + +/** The limits of a link: the ones set, the defaults of its trust for the others. */ +export function linkLimits(trusted: boolean, limits: PartLimits = {}): Limits { + const defaults = trusted ? NONE : UNTRUSTED; + return { part: limits.part ?? defaults.part, decoded: limits.decoded ?? defaults.decoded }; +} + +/** JSON parts above this many UTF-8 bytes are compressed at their origin (a protocol constant). */ +export const COMPRESS_ABOVE = 16 * 1024; + +const UTF8 = new TextEncoder(); + +/** + * A JSON part at its origin: deflated when the host compresses, its UTF-8 bytes are above `COMPRESS_ABOVE` and + * deflating made it smaller; otherwise the text, or its bytes once they were encoded (never encoded twice). Counts one + * deflate (`chord.link.deflates`). + */ +export function originPart(json: string, compression: Compression | undefined, context: Context): Part { + // A UTF-16 unit is at most three UTF-8 bytes: a text this short is within the threshold without encoding it. + if (compression === undefined || json.length * 3 <= COMPRESS_ABOVE) return json; + const bytes = UTF8.encode(json); + if (bytes.byteLength <= COMPRESS_ABOVE) return bytes; + context.telemetry.metrics?.counter("chord.link.deflates").add(1); + const deflated = compression.deflate(bytes); + return deflated.byteLength < bytes.byteLength ? new DeflatedPart(deflated) : bytes; +} + +/** + * The `too_large/part` error of a part whose size on the wire (its UTF-8 bytes; a deflated part's deflated bytes) is + * above `limit`; `undefined` within it. Nothing is parsed or inflated. + */ +export function oversized(part: Part, limit: number): ChordError | undefined { + if (limit === Number.POSITIVE_INFINITY) return undefined; + const size = sizeOf(part); + return size > limit ? tooLarge(size, limit, "wire") : undefined; +} + +/** + * The JSON of a part its owner is about to parse: its size checked against `limits.part` first, then its decoded size + * against `limits.decoded`, a deflated part inflated within it (synchronously with `compression`, or by the + * asynchronous inflate that ran ahead of its owner). Throws `too_large/part` past a limit; throws a plain error for a + * deflated part nothing inflated or that is no raw deflate stream (a protocol error). Counts one synchronous inflate + * (`chord.link.inflates`). + */ +export function openPart( + part: Part, + limits: Limits, + compression: Compression | undefined, + context: Context, +): string | Uint8Array { + const wire = oversized(part, limits.part); + if (wire !== undefined) throw wire; + if (!(part instanceof DeflatedPart)) { + // Not deflated: its decoded size is its size. + const size = limits.decoded === Number.POSITIVE_INFINITY ? 0 : sizeOf(part); + if (size > limits.decoded) throw tooLarge(size, limits.decoded, "decoded"); + return part; + } + let inflated = part.inflated; + if (inflated === undefined) { + if (compression === undefined) throw new Error("part: a deflated part this host did not inflate"); + context.telemetry.metrics?.counter("chord.link.inflates").add(1); + inflated = compression.inflate(part.bytes, limits.decoded); + } + switch (inflated.kind) { + case "bytes": + return inflated.bytes; + case "too_large": + throw tooLarge(undefined, limits.decoded, "decoded"); + case "corrupt": + throw new Error("part: no raw deflate stream", { cause: inflated.error }); + } +} + +/** + * Inflates `part` with the web `DecompressionStream` (a host without a synchronous compressor), counting output bytes + * and stopping past `limit`; the outcome is kept on the part. `signal` (the link's close) cancels the stream at once, + * leaving no outcome. Counts one inflate (`chord.link.inflates`). + */ +export async function inflateAsync( + part: DeflatedPart, + limit: number, + signal: AbortSignal, + context: Context, +): Promise { + context.telemetry.metrics?.counter("chord.link.inflates").add(1); + const reader = new Blob([part.bytes as Uint8Array]) + .stream() + .pipeThrough(new DecompressionStream("deflate-raw")) + .getReader(); + // A pending read settles once cancelled: nothing of the stream outlives the link. + const cancel = (): void => void reader.cancel().catch(() => {}); + signal.addEventListener("abort", cancel, { once: true }); + const chunks: Uint8Array[] = []; + let length = 0; + try { + for (;;) { + const { done, value } = await reader.read(); + if (done || signal.aborted) break; + length += value.byteLength; + if (length > limit) { + cancel(); + part.inflated = { kind: "too_large" }; + return; + } + chunks.push(value); + } + } catch (error) { + if (!signal.aborted) part.inflated = { kind: "corrupt", error }; + return; + } finally { + signal.removeEventListener("abort", cancel); + } + if (signal.aborted) return; + const bytes = new Uint8Array(length); + let offset = 0; + for (const chunk of chunks) { + bytes.set(chunk, offset); + offset += chunk.byteLength; + } + part.inflated = { kind: "bytes", bytes }; +} + +/** A part's size on the wire: its UTF-8 bytes (counted for text, without encoding it). */ +function sizeOf(part: Part): number { + if (typeof part !== "string") return part instanceof DeflatedPart ? part.bytes.byteLength : part.byteLength; + let size = part.length; + for (let index = 0; index < part.length; index++) { + const unit = part.charCodeAt(index); + if (unit < 0x80) continue; + if (unit < 0x800) size += 1; + else if (unit >= 0xd800 && unit < 0xdc00 && (part.charCodeAt(index + 1) & 0xfc00) === 0xdc00) { + // A surrogate pair: four bytes for two units. + size += 2; + index++; + } else size += 2; + } + return size; +} + +/** The error of a part past a limit (`size` unknown while inflating). */ +function tooLarge(size: number | undefined, limit: number, measure: "wire" | "decoded"): ChordError { + const what = size === undefined ? "a part" : `a part of ${size} bytes`; + const message = `${what} is over the ${measure} limit of ${limit} bytes`; + return new ChordError({ code: "too_large", reason: "part", message }); +} diff --git a/packages/chord/src/next/link/text.ts b/packages/chord/src/next/link/text.ts new file mode 100644 index 000000000..c9b7b8e0d --- /dev/null +++ b/packages/chord/src/next/link/text.ts @@ -0,0 +1,356 @@ +// The text codec (2b): one JSON tuple per message, Delta style (a short kind tag, positional fields, no field names), +// and `describeMessage`, which prints any message in that form. A part's JSON text is spliced into the tuple as it is; +// decoding parses the tuple and writes each part back as text (one parse per part per hop: a debugging codec). Byte +// segments of a payload marked as carrying bytes follow the tuple after a newline (JSON text never contains a raw +// one), their lengths the tuple's last field. A message with an origin is wrapped: `["@", origin, tuple]`. Every part +// is tagged with its codec, so no JSON value is reserved: `[0, json]`, or `[1, base64]` for a part its origin deflated +// (the text codec itself never compresses). + +import { + type BoundEntry, + type Codec, + type FailedMember, + type Message, + NO_RETIRED, + type StateEntry, +} from "./message.ts"; +import type { Part } from "./part.ts"; +import { parseJson, partText, textPart } from "./values.ts"; + +const UTF8 = new TextEncoder(); + +/** Messages as JSON tuples: readable on the wire, for debugging links. */ +export const textCodec: Codec = { + name: "text", + encode(message) { + const json = UTF8.encode(tuple(message)); + const segments = segmentsOf(message); + if (segments === undefined) return json; + let length = json.byteLength + 1; + for (const segment of segments) length += segment.byteLength; + const bytes = new Uint8Array(length); + bytes.set(json, 0); + bytes[json.byteLength] = NEWLINE; + let offset = json.byteLength + 1; + for (const segment of segments) { + bytes.set(segment, offset); + offset += segment.byteLength; + } + return bytes; + }, + decode(bytes) { + const newline = bytes.indexOf(NEWLINE); + const json = newline === -1 ? bytes : bytes.subarray(0, newline); + const rest = newline === -1 ? bytes.subarray(bytes.byteLength) : bytes.subarray(newline + 1); + const fields = parseJson(json); + if (!Array.isArray(fields)) throw new Error("text: a message is a tuple"); + return fromTuple(fields, rest); + }, +}; + +/** A message in the text codec's tuple form, for logs and tests (byte segments show as `{"$bytes": i}`). */ +export function describeMessage(message: Message): string { + return tuple(message); +} + +const NEWLINE = 0x0a; + +/** The segments following a payload's tuple; `undefined` without a payload marked as carrying bytes. */ +function segmentsOf(message: Message): readonly Uint8Array[] | undefined { + return message.kind === "call" || message.kind === "result" ? message.segments : undefined; +} + +/** The tuple's JSON text, parts spliced in; wrapped with its origin when it has one. */ +function tuple(message: Message): string { + const origin = "origin" in message ? message.origin : undefined; + return origin === undefined ? ownTuple(message) : `["@",${origin},${ownTuple(message)}]`; +} + +/** The tuple of a message's own fields. */ +function ownTuple(message: Message): string { + switch (message.kind) { + case "hello": + return json(["h", message.protocol, message.incarnation, message.calls]); + case "hold": { + const { address, principal, incarnation, chain, limit } = message; + const head = json(["H", message.hold, message.type, message.version, message.fingerprint]).slice(0, -1); + // Optional trailing fields: an absent one in the middle is null, absent ones at the end are left out. + const optional = [ + address === undefined ? null : [address.kind, address.params], + principal ?? null, + incarnation ?? null, + chain ?? null, + limit ?? null, + ]; + while (optional.length > 0 && optional.at(-1) === null) optional.pop(); + const rest = optional.map((field) => `,${json(field)}`).join(""); + return `${head},${partText(message.params)}${rest}]`; + } + case "release": + return json(["R", message.hold]); + case "call": + return withPayload(json(["c", message.call, message.hold, message.method]), message.args, message.segments); + case "abort": + return json(["x", message.call]); + case "bound": + return `["b",[${message.holds.map(boundEntry).join(",")}]]`; + case "state": + return `["s",[${message.holds.map(stateEntry).join(",")}]]`; + case "failed": { + const kind = message.cause.kind === "transient" ? "t" : "b"; + const retryIn = message.retryIn === undefined ? [] : [message.retryIn]; + return json(["f", message.hold, kind, message.cause.error, ...retryIn]); + } + case "result": + return withPayload(json(["r", message.call]), message.value, message.segments); + case "error": + return json(["e", message.call, message.error]); + case "gone": + return json(["g", message.hold, message.error]); + case "revoke": + return json(["V", message.revocation]); + case "retry": + return json(["T", message.hold]); + case "loading": + return json(["l", message.hold]); + case "ack": + return json(["a"]); + case "closed": + return json(["C"]); + case "ping": + return json(["p"]); + case "pong": + return json(["P"]); + } +} + +function json(value: unknown): string { + return JSON.stringify(value); +} + +/** `[hold, variant, [[member, value]…], failed?]`. */ +function boundEntry({ hold, variant, members, failed }: BoundEntry): string { + const pairs = members.map(({ member, value }) => `[${member},${partText(value)}]`).join(","); + return `[${hold},${variant},[${pairs}]${failedField(failed)}]`; +} + +/** `[hold, [[member, ops]…], failed?, retired?]`; `failed` is `[]` when only `retired` follows. */ +function stateEntry({ hold, members, failed, retired }: StateEntry): string { + const pairs = members.map(({ member, ops }) => `[${member},${partText(ops)}]`).join(","); + if (retired.length === 0) return `[${hold},[${pairs}]${failedField(failed)}]`; + return `[${hold},[${pairs}],${json(failed.map(({ member, error }) => [member, error]))},${json(retired)}]`; +} + +/** The failures as an entry's optional last field: left out when there are none. */ +function failedField(failed: readonly FailedMember[]): string { + return failed.length === 0 ? "" : `,${json(failed.map(({ member, error }) => [member, error]))}`; +} + +/** + * A tuple (its closed JSON `head`) ending in a payload part; one marked as carrying bytes adds the segments' lengths + * as the last field. + */ +function withPayload(head: string, part: Part | undefined, segments: readonly Uint8Array[] | undefined): string { + if (part === undefined) return head; + const prefix = `${head.slice(0, -1)},${partText(part)}`; + if (segments === undefined) return `${prefix}]`; + return `${prefix},${json(segments.map((segment) => segment.byteLength))}]`; +} + +/** + * A message from its tuple: the tag and arity checked, lists of pairs checked as lists, parts written back as text; + * the field types are the message schema's to check. `rest` holds the segments, which only a payload marked as + * carrying bytes may have. + */ +function fromTuple(fields: unknown[], rest: Uint8Array): Message { + if (fields[0] !== "@") return fromFields(fields, rest); + const inner = fields.length === 3 ? list(fields[2]) : []; + if (inner[0] === "@" || inner.length === 0) throw new Error("text: an origin wraps one message"); + const message = fromFields(inner, rest); + // A fresh object; the message schema checks the origin and refuses one on a kind without it. + (message as { origin?: unknown }).origin = fields[1]; + return message; +} + +/** A message from its own fields' tuple (see `fromTuple`). */ +function fromFields(fields: unknown[], rest: Uint8Array): Message { + const [tag, ...f] = fields; + const arity = (...allowed: number[]): void => { + if (!allowed.includes(f.length)) throw new Error(`text: a "${String(tag)}" message with ${f.length} fields`); + }; + if (tag !== "c" && tag !== "r" && rest.byteLength > 0) { + throw new Error("text: byte segments without a payload that carries them"); + } + // The message schema establishes the types of the fields taken here. + const field = (index: number): T => f[index] as T; + switch (tag) { + case "h": + arity(3); + return { kind: "hello", protocol: field(0), incarnation: field(1), calls: field(2) }; + case "H": { + arity(5, 6, 7, 8, 9, 10); + const [address, principal, incarnation, chain, limit] = [5, 6, 7, 8, 9].map((index) => f[index] ?? null); + let wireAddress = {}; + if (address !== null) { + const pair = list(address); + if (pair.length !== 2) throw new Error("text: an address is a pair"); + wireAddress = { address: { kind: pair[0], params: pair[1] } }; + } + return { + kind: "hold", + hold: field(0), + type: field(1), + version: field(2), + fingerprint: field(3), + params: textPart(f[4]), + ...wireAddress, + ...(principal === null ? {} : { principal: principal as { kind: string } }), + ...(incarnation === null ? {} : { incarnation: incarnation as string }), + ...(chain === null ? {} : { chain: chain as string[] }), + ...(limit === null ? {} : { limit: limit as number }), + }; + } + case "R": + arity(1); + return { kind: "release", hold: field(0) }; + case "c": + arity(3, 4, 5); + return { + kind: "call", + call: field(0), + hold: field(1), + method: field(2), + ...payload("args", f.slice(3), rest), + }; + case "x": + arity(1); + return { kind: "abort", call: field(0) }; + case "b": + arity(1); + return { + kind: "bound", + holds: list(f[0]).map((entry): BoundEntry => { + const fields = list(entry); + if (fields.length !== 3 && fields.length !== 4) { + throw new Error(`text: a bound entry with ${fields.length} fields`); + } + const [hold, variant, members, failed] = fields; + return { + hold: hold as number, + variant: variant as number, + members: pairs(members).map(([member, value]) => ({ + member: member as number, + value: textPart(value), + })), + failed: failures(failed), + }; + }), + }; + case "s": + arity(1); + return { + kind: "state", + holds: list(f[0]).map((entry): StateEntry => { + const fields = list(entry); + if (fields.length < 2 || fields.length > 4) { + throw new Error(`text: a state entry with ${fields.length} fields`); + } + const [hold, members, failed, retired] = fields; + return { + hold: hold as number, + members: pairs(members).map(([member, ops]) => ({ member: member as number, ops: textPart(ops) })), + failed: failures(failed), + // The message schema checks each index. + retired: retired === undefined ? NO_RETIRED : (list(retired) as number[]), + }; + }), + }; + case "f": { + arity(3, 4); + if (f[1] !== "t" && f[1] !== "b") throw new Error("text: an unknown failure kind"); + // The schema check after decoding establishes the wire error. + const error = f[2] as Extract["error"]; + const cause = { kind: f[1] === "t" ? "transient" : "broken", error } as const; + return { kind: "failed", hold: field(0), cause, ...(f.length === 4 ? { retryIn: field(3) } : {}) }; + } + case "r": + arity(1, 2, 3); + return { kind: "result", call: field(0), ...payload("value", f.slice(1), rest) }; + case "e": + arity(2); + return { kind: "error", call: field(0), error: field(1) }; + case "g": + arity(2); + return { kind: "gone", hold: field(0), error: field(1) }; + case "V": + arity(1); + return { kind: "revoke", revocation: f[0] }; + case "T": + arity(1); + return { kind: "retry", hold: field(0) }; + case "l": + arity(1); + return { kind: "loading", hold: field(0) }; + case "a": + arity(0); + return { kind: "ack" }; + case "C": + arity(0); + // Its origin is the wrapper's; 0 without one, which the message schema refuses. + return { kind: "closed", origin: 0 }; + case "p": + arity(0); + return { kind: "ping" }; + case "P": + arity(0); + return { kind: "pong" }; + default: + throw new Error("text: an unknown kind"); + } +} + +/** + * A payload part from its fields: none, the value, or the value and its segments' lengths (marked as carrying bytes, + * the segments views of `rest`). + */ +function payload( + key: K, + fields: unknown[], + rest: Uint8Array, +): { [P in K]?: Part } & { segments?: Uint8Array[] } { + if (fields.length < 2) { + if (rest.byteLength > 0) throw new Error("text: byte segments without a payload that carries them"); + return (fields.length === 0 ? {} : { [key]: textPart(fields[0]) }) as { [P in K]?: Part }; + } + const lengths = list(fields[1]); + let offset = 0; + const segments = lengths.map((length) => { + if (!Number.isSafeInteger(length) || (length as number) < 0) throw new Error("text: a segment length"); + const segment = rest.subarray(offset, offset + (length as number)); + offset += length as number; + return segment; + }); + if (offset !== rest.byteLength) throw new Error("text: the segments do not fill the message"); + return { [key]: textPart(fields[0]), segments } as { [P in K]?: Part } & { segments: Uint8Array[] }; +} + +function list(value: unknown): unknown[] { + if (!Array.isArray(value)) throw new Error("text: expected a list"); + return value; +} + +/** A list of `[index, value]` pairs. */ +function pairs(value: unknown): [unknown, unknown][] { + return list(value).map((pair) => { + const entry = list(pair); + if (entry.length !== 2) throw new Error("text: expected a pair"); + return entry as [unknown, unknown]; + }); +} + +/** The optional failures field; absent means none. */ +function failures(value: unknown): FailedMember[] { + if (value === undefined) return []; + // The message schema checks each error's shape. + return pairs(value).map(([member, error]) => ({ member, error }) as FailedMember); +} diff --git a/packages/chord/src/next/link/values.ts b/packages/chord/src/next/link/values.ts new file mode 100644 index 000000000..858866766 --- /dev/null +++ b/packages/chord/src/next/link/values.ts @@ -0,0 +1,126 @@ +// Values in messages (§3 values, §22): JSON text, and for payloads marked as carrying bytes raw byte segments, a +// `Uint8Array` leaf written as `{"$bytes": i}` and read back as a view of the received message (no copy, never +// base64); a part as the text codec splices it. Shared by the codecs. + +import { invalid } from "../data/internal.ts"; +import { DeflatedPart, type Part } from "./part.ts"; + +/** Strict, and a leading U+FEFF is data, not a byte order mark to strip. */ +const STRICT_UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }); + +/** A JSON part at its origin: the value's JSON text (state and params are JSON only). */ +export function jsonPart(value: unknown): string { + return JSON.stringify(value) ?? "null"; +} + +/** + * A payload part at its origin (arguments, a result): its JSON and, when `bytes` (its method's schema has a `bytes()` + * leaf), its byte segments: each `Uint8Array` leaf becomes a segment, and an own key `$bytes`, reserved there, fails + * `invalid/args`; otherwise plain JSON, where `$bytes` is ordinary data. `undefined` (a handler returning nothing) is + * `null`. + */ +export function payloadPart( + value: unknown, + bytes: boolean, +): { readonly json: string; readonly segments: Uint8Array[] | undefined } { + if (!bytes) return { json: jsonPart(value), segments: undefined }; + const segments: Uint8Array[] = []; + const json = JSON.stringify(value, (_key, leaf: unknown) => { + if (leaf instanceof Uint8Array) { + segments.push(leaf); + return { $bytes: segments.length - 1 }; + } + // The replacer does not see the markers it returns, only their fields. + if (typeof leaf === "object" && leaf !== null && Object.hasOwn(leaf, "$bytes")) { + throw invalid("args", '"$bytes" is reserved in values crossing a link'); + } + return leaf; + }); + return { json: json ?? "null", segments }; +} + +/** + * Parses a part's JSON where it is owned, once opened (`openPart`). Throws on a part that is no strict UTF-8 JSON (a + * protocol error). + */ +export function parsePart(json: string | Uint8Array): unknown { + return typeof json === "string" ? JSON.parse(json) : parseJson(json); +} + +/** + * Parses a payload part's JSON where it is owned, once opened, its `{"$bytes": i}` references resolved to views of + * `segments` when it has them. Throws on a malformed part (a protocol error). + */ +export function parsePayload(json: string | Uint8Array, segments: readonly Uint8Array[] | undefined): unknown { + const value = parsePart(json); + return segments === undefined ? value : resolveSegments(value, segments); +} + +/** + * A part as the text codec splices it into its tuple, tagged so that no JSON value is reserved: `[0, json]` for a plain + * part, `[1, base64]` for a deflated one. Throws on invalid UTF-8. + */ +export function partText(part: Part): string { + if (part instanceof DeflatedPart) return `[1,"${base64(part.bytes)}"]`; + return `[0,${typeof part === "string" ? part : readUtf8(part)}]`; +} + +/** A part from its place in a text tuple (`partText` reversed). Throws on anything else (a protocol error). */ +export function textPart(value: unknown): Part { + // Trust boundary: fresh JSON from the text decoder. + if (!Array.isArray(value) || value.length !== 2) throw new Error("text: a part is no [codec, payload] pair"); + const [codec, payload] = value as [unknown, unknown]; + if (codec === 0) return JSON.stringify(payload); + if (codec === 1 && typeof payload === "string") return new DeflatedPart(fromBase64(payload)); + throw new Error("text: a part of an unknown codec"); +} + +/** Strict UTF-8 JSON. Throws on anything else (a protocol error). */ +export function parseJson(bytes: Uint8Array): unknown { + return JSON.parse(STRICT_UTF8.decode(bytes)); +} + +/** Strict UTF-8 text. Throws on invalid UTF-8 (a protocol error). */ +export function readUtf8(bytes: Uint8Array): string { + return STRICT_UTF8.decode(bytes); +} + +/** + * Replaces every `{"$bytes": i}` in a freshly decoded payload with segment `i`, in place. Throws on a malformed + * reference (a protocol error). + */ +export function resolveSegments(value: unknown, segments: readonly Uint8Array[]): unknown { + if (typeof value !== "object" || value === null) return value; + // Trust boundary: fresh JSON from the link, owned by this decode. + const node = value as Record; + if (!Array.isArray(node) && Object.hasOwn(node, "$bytes")) { + const index = node.$bytes; + if (Object.keys(node).length !== 1 || typeof index !== "number" || segments[index] === undefined) { + throw new Error("payload: a malformed byte segment reference"); + } + return segments[index]; + } + for (const key of Object.keys(node)) { + const child = node[key]; + const resolved = resolveSegments(child, segments); + // Defined, not assigned: a key `__proto__` is an own property of parsed JSON. + if (resolved !== child) + Object.defineProperty(node, key, { value: resolved, writable: true, enumerable: true, configurable: true }); + } + return node; +} + +/** Bytes as base64 (the text codec's deflated parts; portable: `btoa` over a binary string, in chunks). */ +function base64(bytes: Uint8Array): string { + let binary = ""; + for (let offset = 0; offset < bytes.byteLength; offset += 0x8000) { + binary += String.fromCharCode(...bytes.subarray(offset, offset + 0x8000)); + } + return btoa(binary); +} + +/** Bytes from base64. Throws on text that is no base64 (a protocol error). */ +function fromBase64(text: string): Uint8Array { + const binary = atob(text); + return Uint8Array.from(binary, (char) => char.charCodeAt(0)); +} diff --git a/packages/chord/src/next/link/websocket.ts b/packages/chord/src/next/link/websocket.ts new file mode 100644 index 000000000..52fe1b2f5 --- /dev/null +++ b/packages/chord/src/next/link/websocket.ts @@ -0,0 +1,106 @@ +// WebSocket links (§3): a link over a WebSocket, the browser's or `ws` in Node, typed structurally; binary frames, +// messages sent while it still connects buffered until it opens. + +import { type Link, LinkEnd } from "./link.ts"; + +/** The part of a WebSocket a link uses (the browser's `WebSocket`, `ws`'s `WebSocket`). */ +export interface WebSocketLike { + /** 0 connecting, 1 open, 2 closing, 3 closed. */ + readonly readyState: number; + binaryType: string; + /** Typed as the browser's `WebSocket` takes bytes: on an `ArrayBuffer`, never shared memory. */ + send(data: Uint8Array): void; + close(): void; + addEventListener(type: "open" | "close" | "error", listener: () => void): void; + addEventListener(type: "message", listener: (event: { readonly data: unknown }) => void): void; + removeEventListener(type: "open" | "close" | "error", listener: () => void): void; + removeEventListener(type: "message", listener: (event: { readonly data: unknown }) => void): void; +} + +const CONNECTING = 0; + +const OPEN = 1; + +/** + * A link over `socket`, one binary frame per message (`binaryType = "arraybuffer"`). Messages sent while it connects + * wait for `open`; a socket closing or closed already gives a closed link; `close` and `error` close the link once, and + * so does a text frame (no message of this protocol). A frame above the transport's `maxPayload` closes the socket + * (code 1009), which closes the link. + */ +export function webSocketLink(socket: WebSocketLike): Link { + return new WebSocketLink(socket); +} + +class WebSocketLink extends LinkEnd { + /** The socket while the link is open. */ + #socket: WebSocketLike | undefined; + /** Messages sent while the socket connects; `undefined` once it is open. */ + #connecting: Uint8Array[] | undefined; + readonly #onOpen = (): void => { + const queued = this.#connecting ?? []; + this.#connecting = undefined; + for (const message of queued) this.send(message); + }; + readonly #onMessage = (event: { readonly data: unknown }): void => { + if (event.data instanceof ArrayBuffer) this.deliver(new Uint8Array(event.data)); + else this.close(); + }; + readonly #onError = (): void => this.#end(); + + constructor(socket: WebSocketLike) { + super(); + if (socket.readyState !== CONNECTING && socket.readyState !== OPEN) { + this.ended(); + return; + } + this.#socket = socket; + socket.binaryType = "arraybuffer"; + if (socket.readyState === CONNECTING) { + this.#connecting = []; + socket.addEventListener("open", this.#onOpen); + } + socket.addEventListener("message", this.#onMessage); + // `error` and `close` stay handled until the socket's own close, after this end closed: `ws` emits an `error` + // for a socket closed while it connects, and an unhandled one would end the process. + const onClose = (): void => { + socket.removeEventListener("close", onClose); + socket.removeEventListener("error", this.#onError); + this.#end(); + }; + socket.addEventListener("close", onClose); + socket.addEventListener("error", this.#onError); + } + + send(message: Uint8Array): void { + const socket = this.#socket; + if (socket === undefined) return; + if (this.#connecting !== undefined) { + this.#connecting.push(message); + return; + } + try { + // A link's messages are codec output or received frames, both on plain `ArrayBuffer`s. + socket.send(message as Uint8Array); + } catch { + this.close(); + } + } + + close(): void { + const socket = this.#socket; + if (socket === undefined) return; + this.#end(); + socket.close(); + } + + /** Stops listening for frames, drops what waits to be sent and closes this end. */ + #end(): void { + const socket = this.#socket; + if (socket === undefined) return; + this.#socket = undefined; + this.#connecting = undefined; + socket.removeEventListener("open", this.#onOpen); + socket.removeEventListener("message", this.#onMessage); + this.ended(); + } +} diff --git a/packages/chord/src/next/remote/README.md b/packages/chord/src/next/remote/README.md new file mode 100644 index 000000000..cd27471a5 --- /dev/null +++ b/packages/chord/src/next/remote/README.md @@ -0,0 +1,172 @@ +# remote + +Remote objects over links (§3 hosts, links, routes, flow control and recovery, §4, §7, §22, W1; decision "Relay flow +control per origin"): both sides of every link of a host, the paths its upstream holds take across links, relayed +holds, the window of `bound` and `state` messages per origin, the call window, and (slice 4b) parts made and opened +within each link's limits, compression at the origin, activation chains and revocation from the parent. Apps use them through `host.accept`, `host.connect`, +`host.route` and references; nothing here is exported for apps. + +## For Chord's layers (`internal.ts`) + +- `HostLink` (one per link: `accept`, `connect` or a route's `resolve`): one `Channel` (handshake, codec, checks, + heartbeat), a role (`parent`, `child`, `route`), the peer's principal as this host stamps it, a provider side per + **origin** and one consumer side. Message kinds are directional, so each goes to its side: `hold`, `release`, + `call`, `abort`, `retry` and `ack` to the provider side of their `origin` (0 when absent; a `hold` makes it, other + messages for an origin without one are ignored), `closed { origin }` ends that provider side; `bound`, `state`, + `failed`, `loading`, `result`, `error` and `gone` to the consumer side; `revoke` to the host, from a parent link only + (`host.revoke` with this host's own `revokedBy`; anywhere else it is ignored). `origin` is honoured on every link. + Closing it, from either end, ends both sides: the provider sides release the peer's holds and abort their calls, the + consumer side's calls reject and its path tells the holders. It makes the parts this host originates (`part`: + deflated above 16 KiB with the host's compressor) and opens the parts it owns (`openPart`: within the limits its + owner gives, inflated in place with the host's compressor). On a host without a compressor the owner decides first + what it owns (`inflates`: a call's arguments on a bound hold served here, the values of its own known holds' `bound` + and `state`, the result of a call still waiting; never a part it relays, never one for an unknown id, Q1, never one + past the wire limit), then inflates those asynchronously one after another within their decoded limit (`inflate`) + and handles the message again; the channel holds the link's later messages until then, and the link's close cancels + the inflate at once. +- **Whom a hold acts for** (`ProviderSide`): on a parent link, the hold's `principal` and `incarnation` fields when + present, else the link's principal and the peer's incarnation; on child and route links always the link's (fields + ignored: a tab cannot claim to be the owner). Calls carry no principal: a call runs as its hold's caller + (`ctx.principal`, `ctx.incarnation`). Trust never grants the fields. + + What a host sends in those fields when it holds over a route link (`actingFor`, for `RemoteObject` and + `RelayedHold`); on a parent link it sends none (the parent ignores them from a child: upward is a boundary, a worker + that acts for a user towards the server passes the user as data). A relay never passes on the fields it received + (`HoldRequest` has none): it decides whom the hold acts for once, when it arrives: + + | The reference was made by | `principal` | `incarnation` | + | --- | --- | --- | + | `host.get` / `host.hold` | the host's own | this host's | + | `ctx.get` in a handler | the handler's `ctx.principal` (the caller's) | the handler's `ctx.incarnation` | + | `self.get` in a remote object | that object's principal | this host's | + | `self.get` in a local object or a facet | the host's own | this host's | + | a relayed hold | the arriving hold's principal | the arriving hold's incarnation | + +- **Origins.** An origin is whose work a message is: 0 for the sender's own, or a relay's number for one of its + downstream origins (a downstream link, and the origin its messages carry), one counter per host (`host.relays`), + never reused. Messages carry it as `origin` (absent for 0) and answers carry it back. +- `ProviderSide` (one origin of a link: the holds and calls the peer sent for it, by their ids): a message naming an id + it does not know is ignored (Q1), one reusing a live id is a protocol error, and so are more running calls than this + side announced in `hello` (N = 256 per origin; a call on an unknown hold is ignored before it is counted). Its table + has two kinds of entries: `Hold` (served here) and `RelayedHold` (relayed, never hydrated). A call on a pending hold + is answered `not_ready`; an aborted, released or gone call gets no reply. While the host closes, its holds are only + dropped (the host ends the objects itself). It sends with its origin. A hold's `chain` is kept only from a trusted + link and its stamped `limit` only from a parent link; its decoded limit is the smaller of the link's and the stamped + one, which a relayed hold carries upstream as `limit`. A call on a relayed hold whose arguments are past the link's + wire limit, or on a served hold whose arguments are past its limits, is answered `too_large/part` and never + admitted. Compressed params (on any link) and compressed arguments from a child are protocol errors: params are never + compressed, and clients never send compressed parts. Holds of a principal being revoked that still wait for `load` + end with `gone` (`revokeWaiting`). +- **The window** (K = 2, `ProviderSide`): at most two of this host's own `bound` and `state` messages unacknowledged + per origin; every other message is control and never waits. The consumer sends `ack` with the origin after applying + each. The side keeps the sender of every message in flight in order (`local`, or the upstream link a relayed one came + from); each `ack` answers the oldest: its own frees a slot, a relayed one goes to that upstream link as `ack` with + the origin's number there. One with nothing in flight is ignored. + - At the end of a batch, with a slot free and nothing waiting, one `state` message carries an entry per hold the + batch touched. Otherwise each touched hold's ops join its members' pending queues (`PendingOps` from `delta/`: raw + ops, folded by the leaf index, past 10,000 only "replace with the current value") and the hold waits. + - A hold that goes live binds now when a slot is free and nothing waits, else waits for a slot; its snapshot and + subscription are taken in one synchronous step when its `bound` is sent (race 2). + - When an `ack` frees a slot: waiting state goes out as one `state` message (each waiting member's pending ops encoded + once, or `[["r", value]]` after an overflow), waiting bindings as one `bound` with every waiting hold; when both + wait they take turns, state first, so neither a hold that keeps changing nor a stream of new holds starves the + other. + - A live object stays live until it ends, so a hold waiting for its slot never goes back to pending; slice 7's + reload adds that. +- **At a relay** nothing is folded, dropped or reset: each upstream `bound`/`state` for an origin is forwarded at once + and acknowledged upstream when the downstream acknowledges it. A slow tab fills only its own origin's window at each + worker, whose pending queues fold its commits; other tabs are other origins. A tab served by W workers may have 2W + messages in flight, plus its relay's own 2. + + Trace (tab D through relay R, worker U, a derive over A and B on the tab): U's window for D's origin is full; a + commit touching A and B joins U's pending queues. D acks one: R returns the `ack` to U, whose freed slot carries one + `state` with A's and B's entries: the derive sees both at once, never A new with B old. +- `Hold` (one per `hold` served here, a `Holder` of its object): takes its object once `load` resolved; the hold's + `params` part is parsed then (malformed: a protocol error). A refused type, a mismatch, invalid params, a wrong + placement or a snapshot that throws is a broken `failed` that ends it (the provider forgets the id). While pending + it follows its object's `state`: each failure, transient or broken, is sent as `failed` with `retryIn`, the rest of + the object's backoff window (absent when broken), the current one at once when the object is failed already; the + next attempt is announced with `loading`; the hold stays, so a `retry` (from any holder) renews even a broken object + for every holder. Live, it binds through the window (`boundEntry`: the variant and each member's value as a part, + names as indexes into the bound version). Its members' frames arrive as `Subscriber` calls inside the batch; at its + end the hold's entry lists the members that changed with their ops, each encoded with that member's own Delta + encoder (`entry`), or queues them (`queue`, later `pendingEntry`). Each entry counts one `chord.link.encodes`. A + call's method is an index into the bound variant's methods; one outside them is a protocol error, its arguments are + parsed when it arrives. A member that fails is listed in the header of the message that carries the failure, once + per failure episode; while the window is full a member keeps one pending failure at most (a later one replaces it, + dropping the member's pending ops; a value after it replaces it), and a retirement of a failed member waits for its + next value (never listed beside its failure). Its + object's end sends `gone` with its error (`forbidden/revoked` for a revocation; its running calls aborted with it). A + released pending hold stops its wait for `load` at once. Params past the link's wire limit, or its decoded one, fail + the hold broken `too_large/part`; a cycle its chain closes, broken `invalid/cycle` (`host.serving`). It counts as one + of the host's `references` (`host.activity`) while it keeps its object. +- `RelayedHold` (a hold this host neither serves nor parses, `host.place` decided it): id `h` of its downstream + origin, held with the same id on its path's current link under that origin's number, as the arriving hold's caller, + from its kept `hold` message (params copied once out of the received message, so no view keeps the buffer). `call`, + `abort`, `retry` and `release` go upstream at once; a call with no upstream link is answered `unavailable`. What + comes back with the origin reaches its downstream `ProviderSide` (`forward`), which sends it whole with its own + origin, ids and parts untouched. A link answers only for what was relayed over it: an entry, notice or reply whose + hold (or call) is not relayed over the link it came from is dropped (Q1), a `bound`/`state` left empty is + acknowledged to that link at once, and a link the origin relayed nothing to is ignored whole. Answers travel + unchecked and unparsed: the end that parses a part checks it against its own link's limits. A revocation ends a + relayed hold of the principal still waiting for its path's link (`revoked`: `gone` downstream), since the link it + would be held on next gets no past `revoke`. Two downstream holds are two upstream holds with their own encoders + (R1). + - Upstream drop (its path tells it): relayed calls in flight `unavailable` (never resent), `loading` downstream; the + path's next link holds it again with the same id and origin, and its `bound` is forwarded. + - Downstream drop: the origin's provider side forgets its relayed holds and sends one `closed { origin }` to each + upstream link it relayed to (no `release` or `abort`). + - `gone` from upstream is forwarded and the relayed hold forgotten. +- `ConsumerSide` (the holds and calls this host's own references sent on a link, by its own counters from 0): + `attach(holder)` holds a `RemoteObject` with a new id. A message with an origin goes to that origin's provider side + (`host.relays`; an unknown origin is ignored). A `bound` or `state` message is prepared for every entry first (one + that does not fit, an index outside the bound version or a member twice included, closes the link before anything + changed), then adopted in one batch of this host, then one `ack` is sent. A malformed reply rejects its call + `unavailable` before the link closes. When it closes, calls in flight and waiting reject `unavailable` (never + resent) and its tables are dropped: nothing of a closed link reaches the next. +- `CallWindow` (one per consumer side): at most the peer's N calls (from its `hello`; none before it) in flight; further + calls wait in one FIFO queue, sent in order as replies free slots; it does not preempt running calls. A relay + forwards its downstream calls at once, so only this host's own calls wait here. A slot is freed once per call: its + `result` or `error`, its `abort` (when sent), its hold's `release` or `gone`, or the link's close. A call still + waiting when its hold gets `loading` is rejected `unavailable` (it may name a method of the old variant). +- `Path` (the parent, or one route address): its current link and its holders (`RemoteObject`s and `RelayedHold`s), + held on that link and again on every next one; when the link drops, every holder is told (`dropped`). `changed` + tells its owner (a route) when it needs a link or lost its last holder. +- `RemoteObject` (one per remote reference, owned by its host across the links of its path): its `state` (`loading` + until a link's hold is bound and again on every drop or `loading`, `failed` from `failed` with `retryAt = now + + retryIn`, `live` on `bound`, `released`), `settled` (pending while loading, resolved while live, rejected while + failed and by a drop or a relay's `loading` while loading: a dependent's activation waits for it and fails at once + then, R18, and `bound` retries the dependent), `lastFailure`, `variant` (set in `bound`'s batch, before derives recompute) and + `member` (the last binding's, kept while loading or failed; outside the variant `invalid/member`), `call` (only while + live; outside the variant `invalid/member`), `retry` (sends `retry` while failed on an open link; after a part past + the limits, placed anew with `host.pathOf` and held with a new id, now or on that path's next link). Its holds made + while the object that declared it activates carry that object's chain (`activationChain`). A snapshot or state part + past the link's limits fails only its hold (`tooLarge`): released at the provider, the reference broken + `too_large/part` and moved to the host's `unplaced` path, so no reconnect holds it and no route resolves for it, + until `retry`; the message's other entries apply. Its call arguments go up a parent link + uncompressed (the parent may have accepted it as a child). It keeps its + binding's caller: two references with different principals are two remote objects. Its holds carry the params as a + part and the address its placement computed. A `bound` (a first binding, a re-hold; also on a live hold) + replaces the replicas in the message's batch: members it had get the snapshot as one frame `[["r", value]]` and a + fresh decoder, members outside a new variant vanish (reads throw `invalid/member`, watches get one `failed` call and + stay open, derives over them fail; a later binding with the member brings it back as `[["r", value]]`, as on the + provider's host), new members are made; then the reference goes live (its parked calls are sent). Its release sends + `release` on the current link and rejects its calls `released`; `gone` and the host's close end the reference with + their error. +- `Replica`: a state member on the consumer, a `Source` with its own Delta decoder: built from the snapshot, then + `decode(wire)` (the new value by Delta's `applyImmutable`, structure shared, nothing validated) and `adopt(value, + ops)` inside the batch, `fail(error)` (a failure call: the watches stay open; the provider's next value for it is + `[["r", value]]`, which `decode` alone accepts while failed); inside a re-hold's batch `reset(snapshot)` and + `vanish(error)`; `end(error)` outside a batch when its reference ends. + +Rules: + +- One encoder and one decoder per member of a hold: two references from one host are two holds, and so are two + downstream holds a relay forwards. +- A host serves over all its links (a server holds a worker's types over a route link and serves the worker's holds of + server types on the same link) and holds over the parent link and its route links. Who reconnects a parent link is + the app (its adapter calls `connect` again); route links are resolved again by their route. Chord only re-holds. + Calls on a loading reference park in its `Binding` (`host/`), so links see only calls on live holds. +- Costs per hop: a relay decodes the envelope, swaps the origin (one shallow copy) and encodes the envelope, copying + parts once (never parsing them), one FIFO entry per `bound`/`state`; a provider encodes once per commit per touched + member, or once per waiting member per freed slot; a consumer adds one `ack` per `bound`/`state` message. diff --git a/packages/chord/src/next/remote/calls.ts b/packages/chord/src/next/remote/calls.ts new file mode 100644 index 000000000..dc415da51 --- /dev/null +++ b/packages/chord/src/next/remote/calls.ts @@ -0,0 +1,70 @@ +// The call window (§3, decision log "Flow control: an ack-clocked window"): at most the peer's N calls in flight on a +// link from this host; further calls wait in one FIFO queue, sent in order as replies free slots. + +/** A call waiting for a slot: its hold (a rebind or release takes it out) and what sends it. */ +export type QueuedCall = { readonly hold: number; readonly send: () => void }; + +/** + * The calls of one link's consumer side. `limit` is the peer's N from its `hello` (0 before it, so every call waits). + * A relay forwards its downstream calls at once, so only this host's own calls are counted here. + */ +export class CallWindow { + #limit = 0; + #inFlight = 0; + readonly #waiting = new Set(); + + /** Sends `call` now when a slot is free and nothing waits, otherwise queues it behind the earlier calls. */ + submit(call: C): void { + if (this.#waiting.size === 0 && this.#inFlight < this.#limit) { + this.#inFlight++; + call.send(); + return; + } + this.#waiting.add(call); + } + + /** The peer's window arrived with its `hello`: waiting calls go out. */ + opened(limit: number): void { + this.#limit = limit; + this.#drain(); + } + + /** A call in flight ended (its reply, its abort sent, its hold released or gone): its slot is free. */ + freed(): void { + this.#inFlight--; + this.#drain(); + } + + /** Takes a waiting call out (its caller cancelled); nothing happens for one that is not waiting. */ + remove(call: C): void { + this.#waiting.delete(call); + } + + /** Takes every waiting call of `hold` out and returns them (its hold rebinds or was released). */ + removeHold(hold: number): C[] { + const removed: C[] = []; + for (const call of this.#waiting) { + if (call.hold !== hold) continue; + this.#waiting.delete(call); + removed.push(call); + } + return removed; + } + + /** Takes every waiting call out and returns them (the link closed). */ + clear(): C[] { + const removed = [...this.#waiting]; + this.#waiting.clear(); + return removed; + } + + /** Sends waiting calls in order while slots are free. */ + #drain(): void { + for (const call of this.#waiting) { + if (this.#inFlight >= this.#limit) return; + this.#waiting.delete(call); + this.#inFlight++; + call.send(); + } + } +} diff --git a/packages/chord/src/next/remote/consumer.ts b/packages/chord/src/next/remote/consumer.ts new file mode 100644 index 000000000..8d654e291 --- /dev/null +++ b/packages/chord/src/next/remote/consumer.ts @@ -0,0 +1,355 @@ +// The consumer side of a link (§7, §22, W1): the holds this host's own remote references sent on it, by this side's +// ids, the calls through them within the peer's call window, and each `bound` and `state` message applied to their +// replicas as one batch, then acknowledged. Messages with an origin answer holds this host relays: they go to that +// origin's provider side. On a host without a compressor, the deflated parts of this host's own holds and calls are +// inflated asynchronously first, and the message is handled once they are. + +import type { Context } from "../../context/index.ts"; +import { type MethodData, takesNoArgs } from "../contracts/internal.ts"; +import { ChordError, isChordError } from "../data/index.ts"; +import { asChordError } from "../data/internal.ts"; +import { + type BoundEntry, + type ConsumerMessage, + type DeflatedPart, + errorFromWire, + type Message, + type Part, + parsePayload, + payloadPart, + type StateEntry, +} from "../link/internal.ts"; +import { CallWindow } from "./calls.ts"; +import { actingFor } from "./hold.ts"; +import type { HostLink } from "./host-link.ts"; +import type { RemoteObject } from "./remote-object.ts"; + +/** A reply to a call. */ +type Reply = Extract; + +/** + * A call through this side: waiting for a slot of the call window (no id yet), then in flight under its id until its + * reply, its abort, its hold's release or `gone`, or the link's close. + */ +type OutgoingCall = { + readonly hold: number; + /** The method's index in the bound variant and its arguments' part. */ + readonly method: number; + readonly args: Part | undefined; + readonly segments: readonly Uint8Array[] | undefined; + readonly resolve: (value: unknown) => void; + readonly reject: (error: unknown) => void; + id: number | undefined; + send(): void; +}; + +/** + * The consumer side of one link. Ids are its own counters, from 0; a reply naming an id this side gave up is ignored + * (Q1). When the link closes, every call rejects `unavailable` (never resent) and its path tells every holder. + */ +export class ConsumerSide { + readonly #link: HostLink; + readonly #entries = new Map(); + readonly #calls = new Map(); + readonly #window = new CallWindow(); + #nextHold = 0; + #nextCall = 0; + + constructor(link: HostLink) { + this.#link = link; + } + + /** + * Holds `holder` on this link and returns its new id: its first hold, or a re-hold on a new link; `chain`: the + * activation chain it is held for, if any (Q5). + */ + attach(holder: RemoteObject, chain: string[] | undefined): number { + const id = this.#nextHold++; + this.#entries.set(id, holder); + const acting = actingFor(this.#link, holder.caller); + this.#link.send({ + kind: "hold", + hold: id, + ...holder.request, + ...acting, + ...(chain === undefined ? {} : { chain }), + }); + return id; + } + + /** + * Calls method `method` (an index into the bound variant) on hold `hold` with `args`: left out for a method without + * arguments, with byte segments when its argument schema has a `bytes()` leaf, deflated above the threshold except on + * the parent link. The result is parsed here. The caller's cancellation rejects it at once and sends `abort` when it + * is in flight. + */ + call(hold: number, method: number, data: MethodData, args: unknown, context: Context): Promise { + const signal = context.abortSignal; + signal?.throwIfAborted(); + const payload = takesNoArgs(data) ? undefined : payloadPart(args, data.args.bytes); + // Up a parent link they travel uncompressed: the parent may have accepted this host as a child, and child links + // refuse compressed parts (§22). + const link = this.#link; + const part = + payload === undefined ? undefined : link.role === "parent" ? payload.json : link.part(payload.json, context); + const { promise, resolve, reject } = Promise.withResolvers(); + const cancel = (): void => { + this.#cancel(call); + reject(signal!.reason); + }; + const done = (): void => signal?.removeEventListener("abort", cancel); + const call: OutgoingCall = { + hold, + method, + args: part, + segments: payload?.segments, + resolve: (value) => { + done(); + resolve(value); + }, + reject: (error) => { + done(); + reject(error); + }, + id: undefined, + send: () => this.#send(call), + }; + signal?.addEventListener("abort", cancel, { once: true }); + this.#window.submit(call); + return promise; + } + + /** Hold `hold` was released: the provider is told, its calls fail with `error`. */ + release(hold: number, error: ChordError): void { + this.#link.send({ kind: "release", hold }); + this.forget(hold, error); + } + + /** `ref.retry()` on a failed hold: the provider is asked for an attempt now. */ + retry(hold: number): void { + this.#link.send({ kind: "retry", hold }); + } + + /** Drops hold `hold` from the tables; its calls, in flight and waiting, fail with `error` and free their slots. */ + forget(hold: number, error: ChordError): void { + this.#entries.delete(hold); + this.#failCalls(hold, error); + } + + /** The peer's `hello` arrived: calls go out within its window. */ + opened(calls: number): void { + this.#window.opened(calls); + } + + /** + * Handles one message from the provider: this host's own, or (with an origin) one for a relayed hold. A `bound`, + * `state` or `result` with deflated parts to inflate asynchronously returns that promise and is handled after it. + */ + message(message: ConsumerMessage): void | Promise { + if (message.origin !== undefined) { + this.#link.host.relays.get(message.origin)?.forward(message, this.#link); + return; + } + switch (message.kind) { + case "bound": + case "state": { + const deflated = this.#deflated(message.holds); + if (deflated !== undefined) return this.#link.inflate(deflated, () => this.message(message)); + if (message.kind === "bound") this.#bound(message.holds); + else this.#state(message.holds); + break; + } + case "failed": + this.#entries.get(message.hold)?.failed(message); + break; + case "loading": + this.#loading(message.hold); + break; + case "result": + case "error": + return this.#reply(message); + case "gone": + this.#gone(message); + break; + } + } + + /** The link closed: calls in flight and waiting reject `unavailable` (never resent, §7); the tables are dropped. */ + closed(): void { + const error = new ChordError({ code: "unavailable", message: "the link closed" }); + const calls = [...this.#calls.values(), ...this.#window.clear()]; + this.#calls.clear(); + this.#entries.clear(); + for (const call of calls) call.reject(error); + } + + /** Sends a call that got its slot, with the next id. */ + #send(call: OutgoingCall): void { + const id = this.#nextCall++; + call.id = id; + this.#calls.set(id, call); + const { hold, method, args, segments } = call; + this.#link.send({ + kind: "call", + call: id, + hold, + method, + ...(args === undefined ? {} : { args }), + ...(segments === undefined ? {} : { segments }), + }); + } + + /** + * Cancels `call`: a waiting one leaves the window and nothing is sent; one in flight sends `abort` and frees its + * slot. Settling the caller is the caller's. + */ + #cancel(call: OutgoingCall): void { + if (call.id === undefined) { + this.#window.remove(call); + return; + } + if (!this.#calls.delete(call.id)) return; + this.#link.send({ kind: "abort", call: call.id }); + this.#window.freed(); + } + + /** + * The deflated parts of this side's own holds in `entries` that wait for an asynchronous inflate, each with the + * link's decoded limit; `undefined` when none does (nothing allocated). Entries of holds this side gave up are not + * inflated (Q1). + */ + #deflated(entries: readonly (BoundEntry | StateEntry)[]): [DeflatedPart, number][] | undefined { + const { limits } = this.#link; + let deflated: [DeflatedPart, number][] | undefined; + for (const entry of entries) { + if (!this.#entries.has(entry.hold)) continue; + for (const member of entry.members) { + const part = "value" in member ? member.value : member.ops; + if (!this.#link.inflates(part, limits)) continue; + deflated ??= []; + deflated.push([part, limits.decoded]); + } + } + return deflated; + } + + /** + * One `bound` message: every entry prepared first (one that does not fit closes the link before anything changed) + * and applied in one batch; then the references go live, and the message is acknowledged. An entry with a part past + * the link's limits fails only its hold (`too_large/part`); the others apply. + */ + #bound(holds: readonly BoundEntry[]): void { + const adoptions: (() => void)[] = []; + const bound: RemoteObject[] = []; + let oversized: [RemoteObject, ChordError][] | undefined; + for (const entry of holds) { + const holder = this.#entries.get(entry.hold); + if (holder === undefined) continue; + try { + adoptions.push(holder.prepareBound(entry, this.#link)); + } catch (error) { + if (!isChordError(error, "too_large")) throw error; + oversized ??= []; + oversized.push([holder, error]); + continue; + } + bound.push(holder); + } + this.#adopt(adoptions); + // After the batch: the references go live with every replica of the message in place. + for (const holder of bound) holder.wentLive(); + for (const [holder, error] of oversized ?? []) holder.tooLarge(error); + this.#link.send({ kind: "ack" }); + } + + /** One `state` message, as `#bound` does it. */ + #state(holds: readonly StateEntry[]): void { + const adoptions: (() => void)[] = []; + let oversized: [RemoteObject, ChordError][] | undefined; + for (const entry of holds) { + const holder = this.#entries.get(entry.hold); + if (holder === undefined) continue; + try { + adoptions.push(holder.prepare(entry, this.#link)); + } catch (error) { + if (!isChordError(error, "too_large")) throw error; + oversized ??= []; + oversized.push([holder, error]); + } + } + this.#adopt(adoptions); + for (const [holder, error] of oversized ?? []) holder.tooLarge(error); + this.#link.send({ kind: "ack" }); + } + + /** Applies prepared entries in one batch of this host, so no watch or derive sees half a provider commit (W1). */ + #adopt(adoptions: readonly (() => void)[]): void { + if (adoptions.length === 0) return; + this.#link.host.batch.commit(() => { + for (const adopt of adoptions) adopt(); + }, this.#link.context); + } + + /** + * `loading`: the hold's object, or its relay's upstream, starts over; its calls still waiting for a slot reject + * `unavailable` (they may name a method of the old variant). + */ + #loading(hold: number): void { + const holder = this.#entries.get(hold); + if (holder === undefined) return; + const error = new ChordError({ code: "unavailable", message: `hold ${hold}: rebinding` }); + for (const call of this.#window.removeHold(hold)) call.reject(error); + holder.loading(); + } + + /** + * `gone`: one for a hold this side gave up is ignored (Q1); the holder ends, which forgets its hold here and fails + * its calls. + */ + #gone(message: Extract): void { + const holder = this.#entries.get(message.hold); + if (holder === undefined) return; + holder.lost(asChordError(errorFromWire(message.error), `hold ${message.hold}`)); + } + + /** + * A reply: its call settles with the parsed result or the error. A result past the link's limits rejects the call + * `too_large/part`. A malformed part rejects the call `unavailable` and throws (a protocol error: the link closes). + * A result to inflate asynchronously is inflated first (for a call still waiting), then handled again. + */ + #reply(reply: Reply): void | Promise { + const call = this.#calls.get(reply.call); + if (call === undefined) return; + const link = this.#link; + if (reply.kind === "result" && link.inflates(reply.value, link.limits)) { + return link.inflate([[reply.value, link.limits.decoded]], () => this.#reply(reply)); + } + this.#calls.delete(reply.call); + this.#window.freed(); + try { + if (reply.kind === "error") call.reject(errorFromWire(reply.error)); + else if (reply.value === undefined) call.resolve(null); + else call.resolve(parsePayload(link.openPart(reply.value, link.limits, link.context), reply.segments)); + } catch (error) { + if (isChordError(error, "too_large")) { + call.reject(error); + return; + } + call.reject(new ChordError({ code: "unavailable", message: `call ${reply.call}: a malformed reply` })); + throw error; + } + } + + /** Fails `hold`'s calls in flight and waiting with `error`; each in-flight one frees its slot. */ + #failCalls(hold: number, error: ChordError): void { + // Waiting ones first: a freed slot must not send one of them. + const failed = this.#window.removeHold(hold); + for (const [id, call] of this.#calls) { + if (call.hold !== hold) continue; + this.#calls.delete(id); + this.#window.freed(); + failed.push(call); + } + for (const call of failed) call.reject(error); + } +} diff --git a/packages/chord/src/next/remote/hold.ts b/packages/chord/src/next/remote/hold.ts new file mode 100644 index 000000000..a87c4aaa8 --- /dev/null +++ b/packages/chord/src/next/remote/hold.ts @@ -0,0 +1,434 @@ +// A hold arriving on a link and served here (§7, §22, W1): where it goes once `load` resolved, the object it keeps, +// the notices of its object's failures and new attempts, the snapshot and subscription taken in one synchronous step +// when a slot of the window is free, and the ops of its members per batch (sent at once, or pending while the window +// is full), each member with its own Delta encoder. + +import type { Context } from "../../context/index.ts"; +import { type Encoder, encoder, type Op, PendingOps } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { + currentVariants, + type VariantData, + type VariantNames, + variantNames, + variantOrder, +} from "../contracts/internal.ts"; +import { type ChordError, isChordError } from "../data/index.ts"; +import { asChordError } from "../data/internal.ts"; +import type { Caller, ChordObject, Holder } from "../host/internal.ts"; +import { + type BoundEntry, + type FailedMember, + jsonPart, + type Limits, + type Message, + NO_RETIRED, + oversized, + parsePart, + type StateEntry, + wireError, +} from "../link/internal.ts"; +import { Source, type Subscriber } from "../state/internal.ts"; +import type { HostLink } from "./host-link.ts"; +import type { ProviderSide } from "./provider.ts"; +import { RelayedHold } from "./relay.ts"; + +/** A `hold` message. */ +type HoldMessage = Extract; + +/** + * What a `hold` message asks for. Whom it acts for is not part of it: a host takes that from the message once, as its + * link allows (`ProviderSide`), and names it again only on a route link (`actingFor`). Its `chain` and `limit` are the + * ones this host honours, sent on as they are when it relays the hold. + */ +export type HoldRequest = Omit; + +/** + * A state member a bound hold follows: its index in the bound version, its source and subscription (which feeds this + * record), its own encoder (R1, path interning) and the ops waiting for a slot of the window. + */ +type Followed = { + readonly index: number; + readonly source: Source; + readonly subscriber: Subscriber; + readonly encoder: Encoder; + readonly pending: PendingOps; +}; + +/** + * Where a hold stands: `pending` until its object is live (waiting for `load`, the activation and its retries; + * `toldFailed` once the consumer was told of a failure, so the next attempt is announced with `loading`); `binding` + * while its object is live and its `bound` waits for a slot; `bound` with the object's variant and the members it + * follows. + */ +type HoldState = + | { readonly kind: "pending"; readonly toldFailed: boolean } + | { readonly kind: "binding" } + | { + readonly kind: "bound"; + readonly variant: VariantData; + /** The bound version's names of the variant, which the wire's indexes point into. */ + readonly names: VariantNames; + /** The members it follows, in the order they were subscribed. */ + readonly followed: Followed[]; + } + | { readonly kind: "released" }; + +/** A bound hold's state. */ +type Bound = Extract; + +const PENDING: HoldState = { kind: "pending", toldFailed: false }; + +const BINDING: HoldState = { kind: "binding" }; + +/** + * One hold served here: a reference to its object acting as its caller (the hold's principal). While pending, each + * failure of its object is sent as `failed` (with the rest of its backoff window) and the next attempt as `loading`; a + * hold that cannot be taken (a refused type, a mismatch, invalid params or placement) is a broken `failed` that ends + * it. Once live it binds when its link's window has a slot, then sends a part per batch that changed its members + * (or keeps their ops pending while the window is full), and `gone` when its object ends. + */ +export class Hold implements Holder { + readonly kind = "served"; + readonly id: number; + /** A hold is declared by no object on this host. */ + readonly dependent = undefined; + /** Whom the hold acts for: its calls run as it, with its origin's incarnation. */ + readonly caller: Caller; + /** The limits per part of its params and its calls' arguments: its link's, the decoded one smaller when stamped (Q2). */ + readonly limits: Limits; + readonly #provider: ProviderSide; + #state: HoldState = PENDING; + /** The held object, from its binding until the hold is released. */ + #object: ChordObject | undefined; + /** + * Resolved on release: a pending hold stops waiting for `load`. A plain resolution, not an abort: an aborted wait + * rejects with an error whose stack keeps this hold, and the waited promise keeps that rejection. + */ + readonly #released = Promise.withResolvers(); + /** The members the current batch changed, with their ops, in order. */ + #changes: (readonly [Followed, readonly Op[]])[] = []; + /** The members that failed since the last message that carried this hold. */ + #failed: FailedMember[] = []; + /** The members whose document was retired since the last message that carried this hold (ORed while it waits). */ + #retired: readonly number[] = NO_RETIRED; + #touched = false; + /** Whether the hold counts as one of the host's references (`host.activity`): while it keeps its object. */ + #counted = false; + + constructor(id: number, provider: ProviderSide, caller: Caller, limits: Limits) { + this.id = id; + this.#provider = provider; + this.caller = caller; + this.limits = limits; + } + + /** Whether the hold is live: calls on it reach its object. */ + get bound(): boolean { + return this.#state.kind === "bound"; + } + + /** Whether the hold waits for `load`: admitted, nothing taken yet (revocation ends it, §4). */ + get waiting(): boolean { + return this.#state.kind === "pending" && this.#object === undefined; + } + + /** + * Admits the hold once `load` resolved: its params within the link's part limit (`too_large/part`), then relayed + * (it is replaced by a relayed hold), or served here, its params opened and parsed (a malformed part is a protocol + * error), its object taken and followed (`moved`). A hold that cannot go anywhere, a type that does not match, + * invalid params, a wrong placement or a cycle (Q5) is a broken `failed`, which ends it. + */ + async start(request: HoldRequest): Promise { + const { link } = this.#provider; + // Waits only for a `load` still running (it never rejects): once loaded, the reference is taken in the turn the + // hold arrived. + const pendingLoad = link.host.pendingLoad; + if (pendingLoad !== undefined) await Promise.race([pendingLoad, this.#released.promise]); + // Released meanwhile (the wait may have settled first): nothing is taken. + if (this.#state.kind !== "pending") return; + let object: ChordObject; + try { + const oversize = oversized(request.params, link.limits.part); + if (oversize !== undefined) throw oversize; + const placed = link.host.place(request, link, this.caller, link.context); + if (placed.kind === "path") { + this.#state = { kind: "released" }; + new RelayedHold(this.id, this.#provider, request, this.caller, placed.path).start(); + return; + } + let params: unknown; + try { + params = parsePart(link.openPart(request.params, this.limits, link.context)); + } catch (error) { + // Past its limits: the hold's \`too_large/part\`. No JSON: a protocol error, reported, and the link closes. + if (isChordError(error, "too_large")) throw error; + link.host.report({ kind: "protocol_error", error }); + link.close(); + return; + } + object = link.host.serving(request, params, this.caller.principal!, link.context); + } catch (error) { + return this.#refuse(asChordError(error, request.type)); + } + // Counted before the object lists it in its presence: a derive over that may close this link synchronously. + this.#object = object; + this.#counted = true; + link.host.count("references", 1); + object.hold(this); + this.moved(); + } + + /** + * The object's state changed, or the hold took it: a pending hold binds once it is live (when a slot is free), + * sends each failure, and announces the attempt after a failure with `loading`. A live object stays live until it + * ends, so a hold waiting for its slot or bound has nothing to do (slice 7's reload changes that). + */ + moved(): void { + const current = this.#state; + if (current.kind !== "pending") return; + const state = this.#object!.state; + if (state.kind === "live") { + this.#state = BINDING; + this.#provider.bind(this, this.#provider.link.context); + } else if (state.kind === "failed") { + const { cause, retryAt } = state; + const retryIn = retryAt === undefined ? {} : { retryIn: Math.max(0, retryAt - Date.now()) }; + const failed = { kind: cause.kind, error: wireError(cause.error) }; + this.#provider.send({ kind: "failed", hold: this.id, cause: failed, ...retryIn }); + this.#state = { kind: "pending", toldFailed: true }; + } else if (state.kind === "loading" && current.toldFailed) { + this.#provider.send({ kind: "loading", hold: this.id }); + this.#state = PENDING; + } + } + + /** A `retry` message: the object is asked for an attempt now (it ignores this unless failed). */ + retry(): void { + this.#object?.retry(); + } + + /** Calls a method of the held object as the hold's caller. */ + call(method: string, args: unknown, context: Context): Promise { + return this.#object!.call(method, args, context, this.caller); + } + + /** + * The method at `index` of the bound variant and whether its result carries byte segments (its schema has a + * `bytes()` leaf). Throws on an index outside the variant (a protocol error); a hold that is not bound has none. + */ + method(index: number): { readonly name: string; readonly resultBytes: boolean } { + const state = this.#state; + const name = state.kind === "bound" ? state.names.methods[index] : undefined; + if (name === undefined) throw new Error(`hold ${this.id}: no method ${index}`); + // The bound state holds the variant whose names these are. + const { variant } = state as Bound; + return { name, resultBytes: variant.methods.get(name)!.result.bytes }; + } + + /** + * The hold's binding, sent when a slot is free. The snapshot of every member, as a value or failed, and the + * subscription are taken in this synchronous step (race 2), so each member's next ops continue exactly from it. + * Names travel as indexes into the bound version (2b). Counts one encode (`chord.link.encodes`). + */ + boundEntry(context: Context): BoundEntry { + const object = this.#object!; + const { contract } = object.served; + // A live remote object has its variant. + const variant = object.variant!; + const state = this.#bindTo(object); + const members: BoundEntry["members"][number][] = []; + const failed: FailedMember[] = []; + for (const [member, source] of object.stateMembers) { + const index = state.names.state.indexOf(member); + let value: JsonValue; + try { + value = source.value; + } catch (error) { + // A failed member (a derive whose compute throws) is followed: it recovers with a root replacement. + failed.push({ member: index, error: wireError(error) }); + if (source instanceof Source) this.#follow(state, index, source); + continue; + } + try { + members.push({ member: index, value: this.#provider.link.part(jsonPart(value), context) }); + } catch (error) { + // A value past the link's limits: failed for this hold, which takes no more of it. + failed.push({ member: index, error: wireError(error) }); + continue; + } + // A constant never changes: nothing to follow. + if (source instanceof Source) this.#follow(state, index, source); + } + this.#changes = []; + this.#failed = []; + this.#retired = NO_RETIRED; + context.telemetry.metrics?.counter("chord.link.encodes").add(1); + return { hold: this.id, variant: variantOrder(contract).indexOf(variant), members, failed }; + } + + /** + * This batch's entry of the `state` message: the members it changed with their ops, each encoded once with its + * member's encoder, and the members that failed; resets both. Counts one encode per hold (`chord.link.encodes`). + */ + entry(context: Context): StateEntry { + const { link } = this.#provider; + const members = this.#take().map(([{ index, encoder }, ops]) => ({ + member: index, + ops: link.part(jsonPart(encoder.encode(ops)), context), + })); + return this.#entry(members, context); + } + + /** The window is full: this batch's ops join the members' pending queues (raw, never mutated). */ + queue(): void { + for (const [{ pending }, ops] of this.#take()) pending.push(ops); + } + + /** + * The entry of a freed slot's `state` message: each member's pending ops encoded once, or `[["r", value]]` after an + * overflow, all reflecting the latest commit; and the members that failed meanwhile. Counts one encode per hold. + */ + pendingEntry(context: Context): StateEntry { + const { link } = this.#provider; + const members: StateEntry["members"][number][] = []; + // Only a bound hold has pending ops. + for (const { index, source, encoder, pending } of (this.#state as Bound).followed) { + if (pending.size === 0) continue; + const taken = pending.take(); + const ops = taken.kind === "replace" ? [["r", source.value] as const] : taken.ops; + if (ops.length > 0) members.push({ member: index, ops: link.part(jsonPart(encoder.encode(ops)), context) }); + } + return this.#entry(members, context); + } + + /** The consumer released the hold, or its link closed: its waits stop and its object loses this reference. */ + release(): void { + this.drop(); + this.#object?.unhold(this); + this.#object = undefined; + } + + /** Ends the hold without releasing its object (the host closes and ends its objects itself). */ + drop(): void { + if (this.#state.kind === "bound") { + for (const { source, subscriber } of this.#state.followed) source.unsubscribe(subscriber); + } + this.#state = { kind: "released" }; + this.#released.resolve(); + if (this.#counted) this.#provider.link.host.count("references", -1); + this.#counted = false; + } + + /** + * The object ended while held (revocation; a reload, slice 7), or the hold was revoked while it waited for `load`: + * the hold sends `gone` with `error`. + */ + end(error: ChordError): void { + if (this.#state.kind === "released") return; + this.#provider.forget(this, error); + this.drop(); + this.#object = undefined; + this.#provider.send({ kind: "gone", hold: this.id, error: wireError(error) }); + } + + /** The binding: its variant and names, with no member followed yet. */ + #bindTo(object: ChordObject): Bound { + const data = currentVariants(object.served.contract).get(object.variant!)!; + const state: Bound = { kind: "bound", variant: data, names: variantNames(data), followed: [] }; + this.#state = state; + return state; + } + + /** + * Follows member `index` from its snapshot on (a failed one too: it may recover in this hold): its own encoder and + * pending queue, and a subscription feeding its calls into this hold's batch. Failures fold per member: a failure + * drops what was pending for it and replaces a failure not sent yet (one at most per member, however long the + * consumer stalls); the value after it (a root replacement) replaces the failure. A retirement waits for a value. + */ + #follow(state: Bound, index: number, source: Source): void { + const followed: Followed = { + index, + source, + subscriber: { + advance: (call) => { + this.#touch(); + if (call.kind === "failed") { + followed.pending.take(); + const failure = { member: index, error: wireError(call.error) }; + const at = this.#failed.findIndex(({ member }) => member === index); + // A new list when one is replaced (rare): a list handed to an entry is never written to. + if (at === -1) this.#failed.push(failure); + else this.#failed = this.#failed.map((pending, i) => (i === at ? failure : pending)); + return; + } + // Rare: a failure of this member still waits to be sent. + if (this.#failed.length > 0) this.#failed = this.#failed.filter(({ member }) => member !== index); + const { ops, retired } = call; + // A retired derive's frame may carry no ops (rule 21): only the flag travels. + if (ops.length > 0) this.#changes.push([followed, ops]); + // A new list per retirement (rare): the shared empty one is never written to. + if (retired && !this.#retired.includes(index)) this.#retired = [...this.#retired, index]; + }, + // The member's source ended with its object, which ends this hold. + end: () => {}, + }, + encoder: encoder(), + pending: new PendingOps(), + }; + source.subscribe(followed.subscriber); + state.followed.push(followed); + } + + /** Takes the current batch's changes; the hold is untouched again. */ + #take(): (readonly [Followed, readonly Op[]])[] { + const changes = this.#changes; + this.#changes = []; + this.#touched = false; + return changes; + } + + /** A state entry with `members` and the failures and retirements since the last one; counts one encode. */ + #entry(members: StateEntry["members"], context: Context): StateEntry { + const failed = this.#failed; + let retired = this.#retired; + this.#failed = []; + this.#retired = NO_RETIRED; + if (retired.length > 0 && failed.length > 0) { + // A failed member's retirement travels with its next value, never with its failure. + const kept = retired.filter((index) => failed.some(({ member }) => member === index)); + if (kept.length > 0) { + this.#retired = kept; + retired = retired.filter((index) => !kept.includes(index)); + } + } + context.telemetry.metrics?.counter("chord.link.encodes").add(1); + return { hold: this.id, members, failed, retired }; + } + + #touch(): void { + if (this.#touched) return; + this.#touched = true; + this.#provider.touched(this); + } + + /** Refuses the hold: a broken `failed`, final; the provider forgets it. */ + #refuse(error: ChordError): void { + this.#provider.forget(this); + this.#provider.send({ kind: "failed", hold: this.id, cause: { kind: "broken", error: wireError(error) } }); + this.release(); + } +} + +/** A `hold` message's fields naming whom it acts for. */ +type ActingFor = Pick; + +const NOBODY: ActingFor = {}; + +/** + * The fields of a `hold` message naming whom it acts for (`caller`): sent only on a route link, whose child honours + * them (§4); none elsewhere. + */ +export function actingFor(link: HostLink, { principal, incarnation }: Caller): ActingFor { + return link.role === "route" && principal !== undefined ? { principal, incarnation } : NOBODY; +} diff --git a/packages/chord/src/next/remote/host-link.ts b/packages/chord/src/next/remote/host-link.ts new file mode 100644 index 000000000..f0da9e79f --- /dev/null +++ b/packages/chord/src/next/remote/host-link.ts @@ -0,0 +1,219 @@ +// One link of a host (§3, §22): its channel, its role, the peer's principal, its limits per part, and both sides: a +// provider side per origin (holds and calls the peer sent, its own or a relay's for one of its downstream links) and +// the consumer side (holds and calls this host sent). Message kinds are directional, so each goes to its side; a +// revocation is honoured from the parent only. Parts are made and opened here, with the host's compressor, or inflated +// asynchronously for their owner when the host has none. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { ChordError, Principal } from "../data/index.ts"; +import type { AnyHost } from "../host/internal.ts"; +import { + Channel, + type ChannelHandler, + type Codec, + DeflatedPart, + inflateAsync, + type Limits, + type Link, + type Message, + openPart, + originPart, + type Part, +} from "../link/internal.ts"; +import { ConsumerSide } from "./consumer.ts"; +import { ProviderSide } from "./provider.ts"; + +/** + * How a link was made: `parent` by `connect` (holds arriving on it may name whom they act for), `child` by `accept`, + * `route` by a route's `resolve`. + */ +export type LinkRole = "parent" | "child" | "route"; + +/** What a link is made with. */ +type HostLinkOptions = { + readonly role: LinkRole; + /** The peer, as this host stamps it. */ + readonly principal: Principal; + /** Skips the message schema check (framing and the owners' checks still run). */ + readonly trusted: boolean; + readonly codec: Codec; + readonly limits: Limits; +}; + +/** How many concurrent calls a host accepts on each of its links, announced in `hello`. */ +const CALLS = 256; + +/** + * A link of a host, both directions over one channel. Closing it, from either end, ends both sides: the provider sides + * release the peer's holds, the consumer side's holds go loading until their path's next link. + */ +export class HostLink implements ChannelHandler { + readonly host: AnyHost; + readonly role: LinkRole; + readonly principal: Principal; + /** Whether the link is trusted: activation chains on holds arriving on it are honoured (Q5). */ + readonly trusted: boolean; + /** The limits per part of what arrives on this link (Q2). */ + readonly limits: Limits; + /** The parent of the work arriving on this link. */ + readonly context: Context; + readonly consumer: ConsumerSide; + /** + * The provider side of each origin the peer sent holds for, made by its first; `origin` is honoured on every link + * (decision "Relay flow control per origin"). + */ + readonly #providers = new Map(); + readonly #channel: Channel; + /** Aborted when the link closes: stops its asynchronous inflates at once. */ + readonly #closing = new AbortController(); + + constructor(host: AnyHost, link: Link, options: HostLinkOptions, context: Context) { + this.host = host; + this.role = options.role; + this.principal = options.principal; + this.trusted = options.trusted; + this.limits = options.limits; + this.context = context; + this.consumer = new ConsumerSide(this); + const { codec, trusted } = options; + const report = (error: unknown): void => host.report({ kind: "protocol_error", error }); + const channel = { codec, incarnation: host.incarnation, calls: CALLS, trusted, report }; + this.#channel = new Channel(link, this, channel, context); + // Last: a link closed already calls `closed` at once. + this.#channel.start(); + } + + /** The peer's incarnation, from its `hello`. */ + get peer(): string { + return this.#channel.peer; + } + + /** Whether the peer's `hello` arrived (it may have arrived while the link was made). */ + get greeted(): boolean { + return this.#channel.opened; + } + + /** Whether the link is open; one closed already when made is not kept. */ + get open(): boolean { + return !this.#channel.closed; + } + + send(message: Message): void { + this.#channel.send(message); + } + + close(): void { + this.#channel.close(); + } + + /** The heartbeat (see `Channel.tick`). */ + tick(now: number, ping: number, close: number): void { + this.#channel.tick(now, ping, close); + } + + opened(): void { + this.consumer.opened(this.#channel.peerCalls); + this.host.linkOpened(this); + } + + /** A JSON part this host originates on this link: deflated above the threshold when the host compresses. */ + part(json: string, context: Context): Part { + return originPart(json, this.host.compression, context); + } + + /** The JSON of a part arriving on this link, where it is owned, within `limits`. Throws as `openPart` does. */ + openPart(part: Part, limits: Limits, context: Context): string | Uint8Array { + return openPart(part, limits, this.host.compression, context); + } + + /** + * Whether `part` waits for an asynchronous inflate before its owner can open it: deflated, not inflated yet, on a + * host without a synchronous compressor, and within the wire limit (past it, opening refuses it uninflated). + */ + inflates(part: Part | undefined, limits: Limits): part is DeflatedPart { + return ( + part instanceof DeflatedPart && + part.inflated === undefined && + this.host.compression === undefined && + part.bytes.byteLength <= limits.part + ); + } + + /** + * Inflates `parts` (deflated parts an owner admitted, each with its decoded limit) one after another with the web + * `DecompressionStream`, then lets the owner handle its message again (`handle`); the channel holds the link's later + * messages until then. A link closed meanwhile stops the inflate at once and drops the message (race 13). + */ + async inflate(parts: readonly (readonly [DeflatedPart, number])[], handle: () => void): Promise { + const { signal } = this.#closing; + for (const [part, limit] of parts) { + await inflateAsync(part, limit, signal, this.context); + if (signal.aborted) return; + } + handle(); + } + + /** + * A message from the peer, to its side. One whose owner first waits for an asynchronous inflate returns its promise: + * the channel holds later messages until it settled. + */ + message(message: Message): void | Promise { + switch (message.kind) { + case "hold": + this.#hold(message); + break; + case "release": + case "call": + case "abort": + case "retry": + case "ack": + // An origin this link has no holds for: its ids are unknown (Q1). + return this.#providers.get(message.origin ?? 0)?.message(message); + case "closed": + this.#providers.get(message.origin)?.closed(); + this.#providers.delete(message.origin); + break; + case "bound": + case "state": + case "failed": + case "loading": + case "result": + case "error": + case "gone": + return this.consumer.message(message); + case "revoke": + // Only the parent revokes; a child or a route cannot (race 6). + // JSON from the parent, the app's revocation type there as here (the same app code). + if (this.role === "parent") this.host.revoke(message.revocation as JsonValue, this.context); + break; + default: + // `hello`, `ping` and `pong` are the channel's. + break; + } + } + + /** Ends the holds on this link still waiting for `load` that act for a principal `revoked` names (§4). */ + revokeWaiting(revoked: (principal: Principal) => boolean, error: ChordError): void { + for (const provider of this.#providers.values()) provider.revokeWaiting(revoked, error); + } + + closed(): void { + this.#closing.abort(); + for (const provider of this.#providers.values()) provider.closed(); + this.#providers.clear(); + this.consumer.closed(); + this.host.unlink(this); + } + + /** A `hold` goes to its origin's provider side, made for the origin's first. */ + #hold(message: Extract): void { + const origin = message.origin ?? 0; + let provider = this.#providers.get(origin); + if (provider === undefined) { + provider = new ProviderSide(this, origin, CALLS); + this.#providers.set(origin, provider); + } + provider.message(message); + } +} diff --git a/packages/chord/src/next/remote/internal.ts b/packages/chord/src/next/remote/internal.ts new file mode 100644 index 000000000..92ae4ad8c --- /dev/null +++ b/packages/chord/src/next/remote/internal.ts @@ -0,0 +1,8 @@ +// Remote objects for Chord's layers: a host's links (both sides of each), the paths its upstream holds take, and the +// objects of its remote references. + +export type { HoldRequest } from "./hold.ts"; +export { HostLink, type LinkRole } from "./host-link.ts"; +export { Path, type Upstream } from "./path.ts"; +export type { ProviderSide } from "./provider.ts"; +export { RemoteObject } from "./remote-object.ts"; diff --git a/packages/chord/src/next/remote/path.ts b/packages/chord/src/next/remote/path.ts new file mode 100644 index 000000000..b1050ec7d --- /dev/null +++ b/packages/chord/src/next/remote/path.ts @@ -0,0 +1,68 @@ +// Paths (§3 recovery): where this host's upstream holds go, the parent link or a route address's link, across links. +// Each new link of a path holds every holder of the path again; when it drops, every holder is told. + +import { ChordError } from "../data/index.ts"; +import type { HostLink } from "./host-link.ts"; +import type { RelayedHold } from "./relay.ts"; +import type { RemoteObject } from "./remote-object.ts"; + +/** What a path holds on its links: one of this host's remote references, or a hold it relays. */ +export type Upstream = RemoteObject | RelayedHold; + +/** + * The parent link or one route address: its current link and its holders, held on that link and again on every next + * one. `changed` runs when it gets a holder without a link, loses its link or its last holder (a route decides whether + * to resolve). + */ +export class Path { + readonly #holders = new Set(); + readonly #changed: () => void; + #link: HostLink | undefined; + + constructor(changed: () => void) { + this.#changed = changed; + } + + /** The current link while one is open. */ + get link(): HostLink | undefined { + return this.#link; + } + + /** Whether anything is held over this path. */ + get held(): boolean { + return this.#holders.size > 0; + } + + /** Holds `holder` over this path: on the current link now, and on every next one. */ + attach(holder: Upstream): void { + this.#holders.add(holder); + if (this.#link === undefined) this.#changed(); + else holder.attach(this.#link); + } + + /** `holder` was released or ended; no later link holds it. */ + detach(holder: Upstream): void { + this.#holders.delete(holder); + if (this.#holders.size === 0) this.#changed(); + } + + /** `link` is the path's link now: every holder is held on it. */ + connected(link: HostLink): void { + this.#link = link; + for (const holder of this.#holders) holder.attach(link); + } + + /** `link` closed (its sides ended their calls already): every holder goes loading until the next link. */ + disconnected(link: HostLink): void { + if (this.#link !== link) return; + this.#link = undefined; + const error = new ChordError({ code: "unavailable", message: "the link closed" }); + for (const holder of this.#holders) holder.dropped(error); + this.#changed(); + } + + /** Every holder, for the host's close. */ + holders(): Upstream[] { + return [...this.#holders]; + } +} diff --git a/packages/chord/src/next/remote/provider.ts b/packages/chord/src/next/remote/provider.ts new file mode 100644 index 000000000..4325f7c12 --- /dev/null +++ b/packages/chord/src/next/remote/provider.ts @@ -0,0 +1,510 @@ +// The provider side of one origin on a link (§3, §22, W1; decision "Relay flow control per origin"): the holds the +// peer sent for that origin, served here or relayed, by their ids; the calls on them, at most this side's N at once; +// and the window: at most K unacknowledged `bound` and `state` messages of this host's own, with what waits meanwhile, +// and the sender of every message in flight, so each `ack` reaches the host that sent its message. + +import { type Context, withAbortSignal } from "../../context/index.ts"; +import { ChordError, isChordError, type Principal } from "../data/index.ts"; +import type { Caller } from "../host/internal.ts"; +import { + type BoundEntry, + type ConsumerMessage, + DeflatedPart, + type Message, + type OriginMessage, + oversized, + type ProviderMessage, + parsePayload, + payloadPart, + type StateEntry, + wireError, +} from "../link/internal.ts"; +import { Hold } from "./hold.ts"; +import type { HostLink } from "./host-link.ts"; +import type { RelayedHold } from "./relay.ts"; + +/** + * A call being handled: on a hold served here, with whether its result carries byte segments and the abort of its + * handler's context; or relayed, its reply forwarded from upstream. + */ +type RunningCall = + | { readonly kind: "served"; readonly hold: Hold; readonly bytes: boolean; readonly abort: AbortController } + | { readonly kind: "relayed"; readonly hold: RelayedHold }; + +/** Who sent a `bound` or `state` message in flight: this host, or the upstream link a relay forwarded it from. */ +type Sender = "local" | HostLink; + +/** A relaying origin: its number on upstream links, and the open upstream links it relayed holds to. */ +type Relay = { readonly origin: number; readonly links: Set }; + +/** At most this many of this host's own `bound` and `state` messages unacknowledged per origin. */ +const WINDOW = 2; + +/** + * The provider side of one origin on a link. Ids are the origin's; a message naming an id this side does not know is + * ignored (Q1), one reusing a live id is a protocol error, and so are more concurrent calls than this side announced. + * This host's own `bound` and `state` messages go through the window: locally served holds wait while it is full + * (their ops in their members' pending queues, their bindings for the next slot). A relay forwards upstream messages + * at once (each upstream host keeps its own window for the origin it sees), so a slow consumer stalls only its origin. + */ +export class ProviderSide { + readonly link: HostLink; + /** Whose work this side serves: 0 for the peer's own, a relay's number for one of its downstream links. */ + readonly origin: number; + readonly #calls = new Map(); + readonly #entries = new Map(); + readonly #limit: number; + /** The sender of each `bound` and `state` message in flight, in order: the consumer acknowledges in order. */ + readonly #sent: Sender[] = []; + /** This host's own messages in flight. */ + #inFlight = 0; + /** The holds the current batch touched, in order; its flush sends or queues them. */ + #touched: Hold[] = []; + /** Holds whose ops wait in their pending queues for a slot. */ + readonly #waiting = new Set(); + /** Holds waiting for a slot to send their first binding. */ + readonly #binding = new Set(); + /** The waiting bindings take the next slot: the last one went to waiting state while they waited. */ + #bindingTurn = false; + /** Set once a hold of this origin is relayed. */ + #relay: Relay | undefined; + + constructor(link: HostLink, origin: number, limit: number) { + this.link = link; + this.origin = origin; + this.#limit = limit; + } + + /** Handles one message of this origin; a call whose arguments wait for an asynchronous inflate returns its promise. */ + message(message: ProviderMessage): void | Promise { + switch (message.kind) { + case "hold": + this.#hold(message); + break; + case "release": + this.#release(message.hold); + break; + case "call": + return this.#call(message); + case "abort": + this.#abort(message.call); + break; + case "retry": + this.#entries.get(message.hold)?.retry(); + break; + case "ack": + this.#ack(); + break; + } + } + + /** + * The link closed, or the origin did (`closed`): every hold is released (relayed ones by one `closed` to each + * upstream link) and every call aborted. + */ + closed(): void { + const closing = this.link.host.closing; + const entries = [...this.#entries.values()]; + this.#entries.clear(); + const closed = new ChordError({ code: "unavailable", message: "the link closed" }); + for (const call of this.#calls.values()) if (call.kind === "served") call.abort.abort(closed); + this.#calls.clear(); + // While the host closes, it ends its objects itself. + for (const entry of entries) { + if (entry.kind === "served" && !closing) entry.release(); + else entry.drop(); + } + // A batch flushing later finds nothing of this side. + this.#touched = []; + const relay = this.#relay; + if (relay === undefined) return; + this.link.host.relays.delete(relay.origin); + for (const link of relay.links) link.send({ kind: "closed", origin: relay.origin }); + } + + /** Sends `message` on the link with this origin. */ + send(message: OriginMessage): void { + this.link.send(this.origin === 0 ? message : { ...message, origin: this.origin }); + } + + /** + * Ends with `error` every hold of this origin that acts for a principal `revoked` names and still waits: for `load` + * (served here), or for its route's link (relayed). + */ + revokeWaiting(revoked: (principal: Principal) => boolean, error: ChordError): void { + for (const hold of [...this.#entries.values()]) { + // A hold arriving on a link acts for a principal: the link's, or one its parent named. + if (!hold.waiting || !revoked(hold.caller.principal!)) continue; + if (hold.kind === "served") hold.end(error); + else hold.revoked(error); + } + } + + /** A pending hold turned out to be relayed: it takes the hold's place in the table. */ + relayed(hold: RelayedHold): void { + this.#entries.set(hold.id, hold); + } + + /** This origin's number on upstream links (one per host, never reused), with `link` as one it relays to. */ + relayTo(link: HostLink): number { + let relay = this.#relay; + if (relay === undefined) { + relay = { origin: this.link.host.relayOrigin(this), links: new Set() }; + this.#relay = relay; + } + relay.links.add(link); + return relay.origin; + } + + /** + * A message from upstream link `from` for this origin's relayed holds, forwarded with this origin: `bound` and + * `state` join the window's senders (their `ack` goes back to `from`), a reply ends its call, `gone` forgets its + * hold. A link answers only for what was relayed over it: an entry, a notice or a reply naming a hold (or a call) + * this origin did not relay over `from` is dropped (Q1); a `bound` or `state` left without entries is acknowledged + * to `from` at once. Parts travel unchecked and unparsed: the end that parses a part checks it against its own + * link's limits. + */ + forward(message: ConsumerMessage, from: HostLink): void { + const relay = this.#relay; + if (relay === undefined || !relay.links.has(from)) return; + const { origin: _, ...forwarded } = message; + switch (forwarded.kind) { + case "bound": + case "state": { + const holds = this.#relayedOver(forwarded.holds, from); + if (holds.length === 0) { + from.send({ kind: "ack", origin: relay.origin }); + return; + } + this.#sent.push(from); + // The entries of the message's own kind. + this.send(holds === forwarded.holds ? forwarded : ({ ...forwarded, holds } as typeof forwarded)); + return; + } + case "result": + case "error": { + const call = this.#calls.get(forwarded.call); + if (call?.kind === "relayed" && call.hold.heldOn(from)) this.reply(forwarded.call, forwarded); + return; + } + case "gone": { + const hold = this.#relayedHold(forwarded.hold, from); + if (hold === undefined) return; + hold.lost(); + this.send(forwarded); + return; + } + default: + if (this.#relayedHold(forwarded.hold, from) !== undefined) this.send(forwarded); + } + } + + /** An upstream link closed: nothing more is relayed to it, nor accepted from it. */ + unlinked(link: HostLink): void { + this.#relay?.links.delete(link); + } + + /** + * The upstream link of relayed hold `hold` dropped with `error`: its calls in flight fail, the hold goes loading + * downstream, and its path's next link holds it again. + */ + upstreamDropped(hold: RelayedHold, error: ChordError): void { + for (const [id, call] of this.#calls) { + if (call.hold === hold) this.reply(id, { kind: "error", call: id, error: wireError(error) }); + } + this.send({ kind: "loading", hold: hold.id }); + } + + /** + * Forgets a hold (released, refused or gone) and every call on it; local calls are aborted with `error` (its + * object's end for a gone hold, `released` otherwise). + */ + forget(hold: Hold | RelayedHold, error?: ChordError): void { + if (this.#entries.get(hold.id) === hold) this.#entries.delete(hold.id); + const reason = error ?? new ChordError({ code: "released", message: `hold ${hold.id}: released` }); + for (const [id, call] of this.#calls) { + if (call.hold !== hold) continue; + this.#calls.delete(id); + if (call.kind === "served") call.abort.abort(reason); + } + if (hold.kind === "served") this.unbind(hold); + } + + /** A call ended with `reply` (a result or an error): its slot is free; nothing is sent for one already gone. */ + reply(id: number, reply: Extract): void { + if (!this.#calls.delete(id)) return; + this.send(reply); + } + + /** A hold's object is live: its `bound` goes now when a slot is free, else with the next free slot's bindings. */ + bind(hold: Hold, context: Context): void { + if (this.#free) { + this.#sendOwn({ kind: "bound", holds: [hold.boundEntry(context)] }); + } else { + this.#binding.add(hold); + } + } + + /** A hold stopped waiting for a slot: it was forgotten. */ + unbind(hold: Hold): void { + this.#waiting.delete(hold); + this.#binding.delete(hold); + } + + /** A hold's members changed in the current batch; its flush sends or queues it. */ + touched(hold: Hold): void { + if (this.#touched.length === 0) this.link.host.batch.flushing((context) => this.#flush(context)); + this.#touched.push(hold); + } + + /** + * `entries` whose holds this origin relayed over `from`, in one pass: the same array when all are, else a copy made + * at the first one that is not. + */ + #relayedOver(entries: readonly E[], from: HostLink): readonly E[] { + let kept: E[] | undefined; + for (let i = 0; i < entries.length; i++) { + const entry = entries[i]!; + const relayed = this.#relayedHold(entry.hold, from) !== undefined; + if (kept !== undefined) { + if (relayed) kept.push(entry); + } else if (!relayed) { + kept = entries.slice(0, i); + } + } + return kept ?? entries; + } + + /** The hold `id` of this origin when it is relayed over `from`. */ + #relayedHold(id: number, from: HostLink): RelayedHold | undefined { + const hold = this.#entries.get(id); + return hold?.kind === "relayed" && hold.heldOn(from) ? hold : undefined; + } + + /** + * A hold message: a protocol error for a live id or compressed params (params are never compressed), otherwise + * admitted once `load` resolved (`Hold.start`). Its activation chain is kept only from a trusted link (Q5); its + * decoded limit is the link's, or the smaller one a parent stamped (Q2), and travels with it when it is relayed. + */ + #hold(message: Extract): void { + if (this.#entries.has(message.hold)) throw new Error(`hold ${message.hold}: the id is in use`); + if (message.params instanceof DeflatedPart) throw new Error(`hold ${message.hold}: compressed params`); + // Whom it acts for is decided here once (`#caller`), never passed on: a relay names it only on a route link. + const { + kind: _kind, + origin: _origin, + hold: id, + principal: _principal, + incarnation: _incarnation, + chain, + limit: stamped, + ...fields + } = message; + const { link } = this; + const limit = Math.min(link.limits.decoded, (link.role === "parent" ? stamped : undefined) ?? Infinity); + const request = { + ...fields, + ...(chain !== undefined && link.trusted ? { chain } : {}), + ...(limit === Infinity ? {} : { limit }), + }; + const limits = limit === link.limits.decoded ? link.limits : { part: link.limits.part, decoded: limit }; + const hold = new Hold(id, this, this.#caller(message), limits); + this.#entries.set(id, hold); + void hold.start(request); + } + + /** + * Whom a hold acts for: on a parent link the hold's own principal and incarnation when it names them, everywhere + * else the link's principal and the peer's incarnation (a child cannot claim to be someone else). + */ + #caller(message: Extract): Caller { + const { link } = this; + if (link.role !== "parent") return { principal: link.principal, incarnation: link.peer }; + // The message schema checked a principal's shape on an untrusted link; a trusted parent is believed. + const principal = (message.principal as Caller["principal"]) ?? link.principal; + return { principal, incarnation: message.incarnation ?? link.peer }; + } + + #release(id: number): void { + const hold = this.#entries.get(id); + if (hold === undefined) return; + this.forget(hold); + hold.release(); + } + + /** + * A call: one on a hold this side does not know is ignored (Q1), more than this side's N at once and compressed + * arguments from a child (clients never compress) are protocol errors. On a relayed hold it is forwarded, its + * arguments checked against this link's part limit first (the bytes are here). On a served hold that is not bound + * yet it is refused `not_ready`; its method is an index into the bound variant (one outside it is a protocol error) + * and its arguments are opened and parsed here within the hold's limits (past one the call fails `too_large/part`; a + * malformed part is a protocol error), absent ones `null`. Arguments this host must inflate asynchronously are + * inflated first, then the call is handled again. + */ + #call(message: Extract): void | Promise { + if (this.#calls.has(message.call)) throw new Error(`call ${message.call}: the id is in use`); + const hold = this.#entries.get(message.hold); + if (hold === undefined) return; + if (this.#calls.size >= this.#limit) throw new Error(`call ${message.call}: more than ${this.#limit} calls`); + const { args: part, segments } = message; + if (part instanceof DeflatedPart && this.link.role === "child") { + throw new Error(`call ${message.call}: compressed arguments from a child`); + } + if (hold.kind === "relayed") { + const error = part === undefined ? undefined : oversized(part, this.link.limits.part); + if (error !== undefined) { + this.#refuse(message.call, error); + return; + } + this.#calls.set(message.call, { kind: "relayed", hold }); + hold.call(message); + return; + } + if (!hold.bound) { + this.#refuse(message.call, new ChordError({ code: "not_ready", message: `hold ${hold.id}: not bound yet` })); + return; + } + const method = hold.method(message.method); + if (this.link.inflates(part, hold.limits)) { + return this.link.inflate([[part, hold.limits.decoded]], () => this.#call(message)); + } + let json: string | Uint8Array | undefined; + try { + json = part === undefined ? undefined : this.link.openPart(part, hold.limits, this.link.context); + } catch (error) { + if (!isChordError(error, "too_large")) throw error; + this.#refuse(message.call, error); + return; + } + const args = json === undefined ? null : parsePayload(json, segments); + const call = { kind: "served", hold, bytes: method.resultBytes, abort: new AbortController() } as const; + this.#calls.set(message.call, call); + void this.#run(message.call, call, method.name, args); + } + + /** + * Runs a call and sends its result (absent when `null`) or error, unless it was aborted, released or gone + * meanwhile. A result that cannot be encoded is sent as its error. + */ + async #run( + id: number, + call: Extract, + method: string, + args: unknown, + ): Promise { + let reply: Extract; + try { + // A link's context has no abort signal, so this is the call's own signal: no `AbortSignal.any` per call. + const result = await call.hold.call(method, args, withAbortSignal(call.abort.signal, this.link.context)); + // Results are not validated: they are the provider's. + reply = + result === null || result === undefined + ? { kind: "result", call: id } + : this.#result(id, result, call.bytes, this.link.context); + } catch (error) { + reply = { kind: "error", call: id, error: wireError(error) }; + } + this.reply(id, reply); + } + + /** Answers call `id` with `error` before it was admitted: it took no slot. */ + #refuse(id: number, error: ChordError): void { + this.send({ kind: "error", call: id, error: wireError(error) }); + } + + #abort(id: number): void { + const call = this.#calls.get(id); + if (call === undefined) return; + this.#calls.delete(id); + if (call.kind === "relayed") call.hold.abort(id); + else call.abort.abort(new ChordError({ code: "aborted", message: `call ${id}: aborted by its caller` })); + } + + /** + * An `ack` answers the oldest message in flight: this host's own frees a slot, a relayed one is acknowledged to its + * upstream link (closed meanwhile: nothing to do). An ack crossing nothing in flight is ignored. + */ + #ack(): void { + const sender = this.#sent.shift(); + if (sender === undefined) return; + if (sender !== "local") { + // A relayed message implies the relay's number. + sender.send({ kind: "ack", origin: this.#relay!.origin }); + return; + } + this.#inFlight--; + this.#slotFreed(); + } + + /** + * The end of a batch: while a slot is free and nothing waits, one `state` message with an entry per touched hold; + * otherwise each touched hold's ops join its pending queues until a slot frees. + */ + #flush(context: Context): void { + const holds = this.#touched; + // The side closed during the batch: nothing is sent. + if (holds.length === 0) return; + this.#touched = []; + if (this.#free) { + this.#sendOwn({ kind: "state", holds: holds.map((hold) => hold.entry(context)) }); + return; + } + // Only a bound hold is touched, and a bound hold never waits for its binding. + for (const hold of holds) { + hold.queue(); + this.#waiting.add(hold); + } + } + + /** Whether this host's next message goes out now: a slot is free and nothing waits for one. */ + get #free(): boolean { + return this.#binding.size === 0 && this.#waiting.size === 0 && this.#inFlight < WINDOW; + } + + /** + * A slot may be free: while there is room, the waiting state goes out as one message and the waiting bindings as + * one `bound`, taking turns when both wait (state first), so neither a hold that keeps changing nor a stream of new + * holds starves the other. + */ + #slotFreed(): void { + const context = this.link.context; + while (this.#inFlight < WINDOW) { + if (this.#binding.size > 0 && (this.#waiting.size === 0 || this.#bindingTurn)) { + const holds = [...this.#binding]; + this.#binding.clear(); + this.#bindingTurn = false; + // Every snapshot in one synchronous step: the binding shows all of them at one moment. + this.#sendOwn({ kind: "bound", holds: holds.map((hold) => hold.boundEntry(context)) }); + } else if (this.#waiting.size > 0) { + const holds = [...this.#waiting]; + this.#waiting.clear(); + this.#bindingTurn = this.#binding.size > 0; + this.#sendOwn({ kind: "state", holds: holds.map((hold) => hold.pendingEntry(context)) }); + } else { + return; + } + } + } + + /** + * A call's result message, its part deflated above the threshold when this host compresses; with byte segments when + * its method's result schema has a `bytes()` leaf. + */ + #result( + id: number, + result: unknown, + bytes: boolean, + context: Context, + ): Extract { + const { json, segments } = payloadPart(result, bytes); + const value = this.link.part(json, context); + return { kind: "result", call: id, value, ...(segments === undefined ? {} : { segments }) }; + } + + /** Sends one of this host's own `bound` or `state` messages, taking a slot. */ + #sendOwn(message: Extract): void { + this.#inFlight++; + this.#sent.push("local"); + this.send(message); + } +} diff --git a/packages/chord/src/next/remote/relay.ts b/packages/chord/src/next/remote/relay.ts new file mode 100644 index 000000000..b8857992c --- /dev/null +++ b/packages/chord/src/next/remote/relay.ts @@ -0,0 +1,130 @@ +// Relayed holds (§4, §22, R1; decision "Relay flow control per origin"): a hold that arrived on one link (downstream) +// and that this host neither serves nor parses, held again on its path's link (upstream) with the same id under its +// downstream origin's number there; messages are forwarded between the two with the origin rewritten and parts +// untouched. + +import { ChordError } from "../data/index.ts"; +import type { Caller } from "../host/internal.ts"; +import { type Message, wireError } from "../link/internal.ts"; +import { actingFor, type HoldRequest } from "./hold.ts"; +import type { HostLink } from "./host-link.ts"; +import type { Path } from "./path.ts"; +import type { ProviderSide } from "./provider.ts"; + +/** + * One relayed hold: `id` on its downstream origin, and the same id upstream on its path's current link (none while + * the path has no link). It keeps its `hold` message to hold again on the next link. Calls, `abort`, `retry` and + * `release` go upstream at once; what comes back is forwarded by its downstream origin (`ProviderSide.forward`). Two + * downstream holds are two upstream holds, each with its own encoders (R1). + */ +export class RelayedHold { + readonly kind = "relayed"; + readonly id: number; + readonly #downstream: ProviderSide; + /** + * What the upstream hold asks for, with the chain and decoded limit this host honoured; its params owned, not a view + * of the message they came in. + */ + readonly #request: HoldRequest; + /** Whom the hold acts for, sent on route links. */ + readonly caller: Caller; + readonly #path: Path; + /** The upstream link while the path has one, and the downstream origin's number on it. */ + #upstream: { readonly link: HostLink; readonly origin: number } | undefined; + + constructor(id: number, downstream: ProviderSide, request: HoldRequest, caller: Caller, path: Path) { + this.id = id; + this.#downstream = downstream; + const { params } = request; + // Params are never deflated: the provider side refused deflated ones. + const owned = typeof params === "string" ? params : (params as Uint8Array).slice(); + this.#request = { ...request, params: owned }; + this.caller = caller; + this.#path = path; + } + + /** Takes the pending hold's place downstream and holds upstream (now, or when the path has a link). */ + start(): void { + this.#downstream.relayed(this); + this.#path.attach(this); + } + + /** Whether it is held on `link` now: only that link answers for it. */ + heldOn(link: HostLink): boolean { + return this.#upstream?.link === link; + } + + /** Holds on `link`, the path's link now: its kept `hold` message, with whom it acts for on a route link. */ + attach(link: HostLink): void { + const origin = this.#downstream.relayTo(link); + this.#upstream = { link, origin }; + link.send({ kind: "hold", origin, hold: this.id, ...this.#request, ...actingFor(link, this.caller) }); + } + + /** The upstream link dropped: downstream its calls fail and the hold goes loading until the path's next link. */ + dropped(error: ChordError): void { + const upstream = this.#upstream; + this.#upstream = undefined; + if (upstream !== undefined) this.#downstream.upstreamDropped(this, error); + } + + /** + * Whether it waits for its path's link (a route still resolving, or a dropped link's next): a revocation ends it then, + * since the link it is held on next gets no past `revoke` (§4). + */ + get waiting(): boolean { + return this.#upstream === undefined; + } + + /** Revoked while it waited for its path's link: forgotten, and `gone` with `error` downstream. */ + revoked(error: ChordError): void { + this.drop(); + this.#downstream.forget(this); + this.#downstream.send({ kind: "gone", hold: this.id, error: wireError(error) }); + } + + /** `gone` from upstream: the hold is forgotten (the downstream origin forwards the message). */ + lost(): void { + this.drop(); + this.#downstream.forget(this); + } + + /** + * A call from downstream (registered there under its id), forwarded upstream at once; its reply comes back by the + * origin. Without an upstream link it is answered `unavailable` at once (it crossed the `loading` notice). + */ + call(message: Extract): void { + const upstream = this.#upstream; + if (upstream !== undefined) { + upstream.link.send({ ...message, origin: upstream.origin }); + return; + } + const error = new ChordError({ code: "unavailable", message: `hold ${this.id}: its upstream link is down` }); + this.#downstream.reply(message.call, { kind: "error", call: message.call, error: wireError(error) }); + } + + /** A downstream `abort` of call `id`, forwarded. */ + abort(id: number): void { + const upstream = this.#upstream; + upstream?.link.send({ kind: "abort", origin: upstream.origin, call: id }); + } + + /** A downstream `retry`, forwarded. */ + retry(): void { + const upstream = this.#upstream; + upstream?.link.send({ kind: "retry", origin: upstream.origin, hold: this.id }); + } + + /** Released downstream: released upstream (its calls with it) and no later link holds it. */ + release(): void { + const upstream = this.#upstream; + this.drop(); + upstream?.link.send({ kind: "release", origin: upstream.origin, hold: this.id }); + } + + /** No later link holds it (gone, or its downstream origin closed, which `closed` tells the upstream links). */ + drop(): void { + this.#upstream = undefined; + this.#path.detach(this); + } +} diff --git a/packages/chord/src/next/remote/remote-object.ts b/packages/chord/src/next/remote/remote-object.ts new file mode 100644 index 000000000..450c051e1 --- /dev/null +++ b/packages/chord/src/next/remote/remote-object.ts @@ -0,0 +1,428 @@ +// An object held over a link (§7): what one remote reference is bound to on the consumer, across the links of its path +// (the parent, or a route address). Its state, the replicas of its last bound variant (kept while its link is down), +// its hold on the path's current link and calls through it. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Contract, ParamsOf } from "../contracts/index.ts"; +import { + currentVariants, + type MethodData, + type VariantNames, + variantNames, + variantOrder, +} from "../contracts/internal.ts"; +import { ChordError } from "../data/index.ts"; +import { asChordError, invalid } from "../data/internal.ts"; +import { + type AnyHost, + type Binding, + type Caller, + type CheckedParams, + type ChordObject, + LIVE, + LOADING, + type PinnedKey, + type RefState, + settlement, + type Target, +} from "../host/internal.ts"; +import { + type BoundEntry, + errorFromWire, + jsonPart, + type Message, + parsePart, + type StateEntry, + type WireAddress, +} from "../link/internal.ts"; +import type { DeriveSource } from "../state/internal.ts"; +import type { ConsumerSide } from "./consumer.ts"; +import type { HoldRequest } from "./hold.ts"; +import type { HostLink } from "./host-link.ts"; +import type { Path } from "./path.ts"; +import { type MemberSnapshot, Replica } from "./replica.ts"; + +/** The last binding: its variant, methods, the bound version's names (the wire's indexes) and a replica per member. */ +type Bound = { + readonly variant: string; + readonly methods: ReadonlyMap; + readonly names: VariantNames; + readonly replicas: ReadonlyMap; +}; + +/** + * One remote reference's object, owned by its host across the links of its path. It is `loading` until a link's hold + * of it is bound, and again whenever that link drops or its relay's upstream starts over; each link holds it with its + * own id. Two references are two holds, each member of each with its own Delta decoder. + */ +export class RemoteObject implements Target { + readonly kind = "remote"; + /** The object in errors: its type, and its params key when it has params. */ + readonly label: string; + readonly contract: Contract; + readonly #host: AnyHost; + /** What its holds ask for: the params as a part (checked by L1 when the reference was made), the address. */ + readonly request: HoldRequest; + /** Whom its reference acts for: named on route links (`actingFor`). */ + readonly caller: Caller; + /** Its params, for placing it anew (`retry` after a part past the limits). */ + readonly #params: ParamsOf; + /** Its path: the parent's or a route address's; the host's `unplaced` one after a part past the limits. */ + #path: Path; + /** The object whose activation declared the reference (`self.get`): its holds carry its chain while it activates. */ + #dependent: ChordObject | undefined; + #state: RefState = LOADING; + /** The last binding, kept while loading or failed: members keep their last values and watches. */ + #bound: Bound | undefined; + /** The hold on the path's link while one is open: that link's consumer side and this hold's id on it. */ + #hold: { readonly side: ConsumerSide; readonly id: number } | undefined; + #binding: Binding | undefined; + /** Pending while loading, resolved while live, rejected while failed: a dependent's activation waits for it (R18). */ + #settled = settlement(); + #lastFailure: ChordError | undefined; + + constructor( + host: AnyHost, + contract: Contract, + { params, key }: CheckedParams, + address: WireAddress | undefined, + caller: Caller, + path: Path, + dependent: ChordObject | undefined, + ) { + this.#host = host; + this.contract = contract; + this.label = key === "" ? contract.name : `${contract.name}(${key})`; + const { name: type, version, fingerprint } = contract; + this.request = { + type, + version, + fingerprint, + params: jsonPart(params), + ...(address === undefined ? {} : { address }), + }; + this.caller = caller; + this.#params = params; + this.#path = path; + this.#dependent = dependent; + } + + get state(): RefState { + return this.#state; + } + + /** Pending while loading; resolves when bound, rejects with a failure or the reference's end. */ + get settled(): Promise { + return this.#settled.promise; + } + + /** The error of the last `failed`, until `bound`. */ + get lastFailure(): ChordError | undefined { + return this.#lastFailure; + } + + /** The variant last bound; `undefined` before the first binding. */ + get variant(): string | undefined { + return this.#bound?.variant; + } + + /** Whether its state members have values: bound once. */ + get hydrated(): boolean { + return this.#bound !== undefined; + } + + /** Its one reference bound to it. */ + hold(binding: Binding): void { + this.#binding = binding; + } + + /** The reference was released: the current link's provider is told and the reference's sent calls rejected. */ + unhold(_binding: Binding): void { + const error = new ChordError({ code: "released", message: `${this.label}: released` }); + this.#hold?.side.release(this.#hold.id, error); + this.#end(error); + } + + /** + * Holds on `link`, the path's link now (a first hold, or a re-hold on a new link), with the activation chain of its + * dependent while that activates (Q5). + */ + attach(link: HostLink): void { + const chain = this.#dependent?.activationChain(); + this.#hold = { side: link.consumer, id: link.consumer.attach(this, chain) }; + } + + /** The link dropped with `error`: loading until a new link's hold is bound (`#startOver`). */ + dropped(error: ChordError): void { + this.#hold = undefined; + this.#startOver(error); + } + + /** + * Calls a method of the bound variant over the link; one outside it fails `invalid/member`. Only a live object is + * called: bound on an open link. + */ + async call(method: string, args: unknown, context: Context): Promise { + const bound = this.#bound!; + const hold = this.#hold!; + const data = bound.methods.get(method); + if (data === undefined) { + throw invalid("member", `${this.label} (variant ${bound.variant}) has no method "${method}"`); + } + // Arguments are checked by the provider. + return hold.side.call(hold.id, bound.names.methods.indexOf(method), data, args, context); + } + + /** + * The replica of `member`; `not_ready` before the first binding, `invalid/member` for a member no binding had (one + * a later variant lacks reads its `invalid/member` failure). + */ + member(member: string): DeriveSource { + const bound = this.#bound; + if (bound === undefined) throw new ChordError({ code: "not_ready", message: `${this.label}: not bound yet` }); + const replica = bound.replicas.get(member); + if (replica === undefined) { + throw invalid("member", `${this.label} (variant ${bound.variant}) has no state member "${member}"`); + } + return replica; + } + + /** A failed remote object's members are not read: its failure is the reads' error (see `Target.lastValues`). */ + get lastValues(): false { + return false; + } + + /** Keyed members are only in local contracts (`keyedMember`). */ + async pinKey(member: string): Promise { + return this.keyedMember(member); + } + + /** Keyed members are only in local contracts, whose objects are never held over a link (slice 6a §12). */ + keyedMember(member: string): never { + throw invalid("member", `${this.label} has no keyed member "${member}": keyed members are local`); + } + + /** An object over a link activates on another host: it is in no chain here (Q5). */ + get activationId(): undefined { + return undefined; + } + + /** An object over a link waits for nothing on this host: no chain of waits (its holds carry the chain, Q5). */ + pathTo(): undefined { + return undefined; + } + + /** + * `ref.retry()`: a failed object's provider is asked for an attempt; one that failed for a part past the link's + * limits is placed anew and held with a new id, now or on its path's next link (a placement that throws fails it + * broken again); ignored otherwise and without a link. + */ + retry(): void { + if (this.#state.kind !== "failed") return; + if (this.#path !== this.#host.unplaced) { + this.#hold?.side.retry(this.#hold.id); + return; + } + let path: Path; + try { + path = this.#host.pathOf(this.contract, this.#params, this.caller.principal); + } catch (error) { + this.#fail(asChordError(error, this.contract.name)); + return; + } + this.#path.detach(this); + this.#path = path; + this.#move(LOADING); + path.attach(this); + } + + /** + * A part of its snapshot or of a state change was past the link's limits (`too_large/part`): the hold is released + * at the provider, and the reference fails broken with `error`, final: no link holds it again (a reconnect + * included) until `retry`. + */ + tooLarge(error: ChordError): void { + const hold = this.#hold; + if (hold === undefined) return; + this.#hold = undefined; + hold.side.release(hold.id, error); + this.#path.detach(this); + this.#path = this.#host.unplaced; + this.#path.attach(this); + this.#fail(error); + } + + /** Fails broken with `error`: no automatic retry. */ + #fail(error: ChordError): void { + this.#lastFailure = error; + this.#move({ kind: "failed", cause: { kind: "broken", error }, retryAt: undefined }); + } + + /** + * Prepares one entry of a `bound` message (a first binding, a re-hold; a live hold is bound again): the + * variant and every state member once, as a value or as failed. Returns what applies it inside the batch of the + * whole message: a member it had gets the snapshot as one frame `[["r", value]]` and a fresh decoder, a member + * outside the new variant fails (its replica stays, for a later binding that has it again), a new one gets its first + * value. Throws on a snapshot that + * does not fit (a protocol error), and `too_large/part` for a value past the link's limits, before anything changed. + * `wentLive` follows the batch. + */ + prepareBound(entry: BoundEntry, link: HostLink): () => void { + const variant = variantOrder(this.contract)[entry.variant]; + if (variant === undefined) throw new Error(`hold ${entry.hold}: no variant ${entry.variant}`); + const data = currentVariants(this.contract).get(variant)!; + const names = variantNames(data); + const members = new Map(); + const add = (index: number, state: MemberSnapshot): void => { + const member = names.state[index]; + if (member === undefined || members.has(member)) + throw new Error(`hold ${entry.hold}: member ${index} in bound`); + members.set(member, state); + }; + // Fresh JSON from the link: state is JSON and never validated (§30). + for (const { member, value } of entry.members) { + add(member, { kind: "value", value: parsePart(link.openPart(value, link.limits, link.context)) as JsonValue }); + } + for (const { member, error } of entry.failed) add(member, { kind: "failed", error: errorFromWire(error) }); + if (members.size !== names.state.length) throw new Error(`hold ${entry.hold}: the snapshot misses a member`); + return () => { + const previous = this.#bound?.replicas; + const replicas = new Map(); + // First: a derive recomputing in this batch reads the new variant through its reference. + this.#bound = { variant, methods: data.methods, names, replicas }; + for (const [member, state] of members) { + const replica = previous?.get(member); + if (replica === undefined) { + replicas.set(member, new Replica(this.#host.batch, state)); + } else { + replica.reset(state); + replicas.set(member, replica); + } + } + for (const [member, replica] of previous ?? []) { + if (members.has(member)) continue; + replica.vanish(invalid("member", `${this.label} (variant ${variant}) has no state member "${member}"`)); + replicas.set(member, replica); + } + }; + } + + /** After a `bound` message's batch: the reference goes live (its parked calls are sent) and R18 retries follow. */ + wentLive(): void { + this.#lastFailure = undefined; + this.#move(LIVE); + // R18: a dependent that failed while waiting for this object is retried now. + this.#binding?.dependent?.dependencyLive(); + } + + /** + * `failed`: the provider's object failed and no attempt runs; `retryIn` is the rest of its backoff window. Ignored + * once live (only a pending hold is told). + */ + failed(message: Extract): void { + if (this.#state.kind === "live") return; + // A plugin's error a provider failed with is this object's failure, as a ChordError. + const error = asChordError(errorFromWire(message.cause.error), this.label); + this.#lastFailure = error; + const retryAt = message.retryIn === undefined ? undefined : Date.now() + message.retryIn; + this.#move({ kind: "failed", cause: { kind: message.cause.kind, error }, retryAt }); + } + + /** + * `loading`: the provider's object started a new attempt after a failure, or its relay's upstream link dropped + * (`#startOver`). + */ + loading(): void { + this.#startOver( + new ChordError({ code: "unavailable", message: `${this.label}: its relay's upstream link dropped` }), + ); + } + + /** + * Decodes and applies one entry of a state message (its members by index, each with its ops), without changing + * anything yet: returns what adopts it into the replicas, run inside the batch that applies the whole message. + * Throws on an entry that does not fit (a protocol error), and `too_large/part` for ops past the link's limits. + */ + prepare({ hold: id, members, failed, retired }: StateEntry, link: HostLink): () => void { + const bound = this.#bound; + if (this.#state.kind !== "live" || bound === undefined) throw new Error(`hold ${id}: state before bound`); + const replica = (index: number): Replica => { + const name = bound.names.state[index]; + const found = name === undefined ? undefined : bound.replicas.get(name); + if (found === undefined) throw new Error(`hold ${id}: no state member ${index}`); + return found; + }; + // Made by the first retirement (rare): no allocation per message otherwise. + let retirements: Set | undefined; + for (const member of retired) { + const target = replica(member); + retirements ??= new Set(); + if (retirements.has(target)) throw new Error(`hold ${id}: member ${member} retired twice in one state`); + retirements.add(target); + } + const changed = new Set(); + const updates = members.map(({ member, ops }) => { + const target = replica(member); + if (changed.has(target)) throw new Error(`hold ${id}: member ${member} twice in one state`); + changed.add(target); + return { target, ...target.decode(parsePart(link.openPart(ops, link.limits, link.context))) }; + }); + const failures = failed.map(({ member, error }) => ({ target: replica(member), error: errorFromWire(error) })); + return () => { + for (const { target, value, ops } of updates) target.adopt(value, ops, retirements?.has(target) === true); + for (const { target, error } of failures) target.fail(error); + // A retirement without ops (a derive whose value stayed, rule 21): the flag alone, after the failures (a + // failed member keeps it for its next value). + if (retirements !== undefined) { + for (const target of retirements) if (!changed.has(target)) target.retireAlone(); + } + }; + } + + /** `gone`, or the host closes: the reference ends, its calls reject with `error`. */ + lost(error: ChordError): void { + const binding = this.#binding; + this.#hold?.side.forget(this.#hold.id, error); + this.#end(error); + binding?.end(error); + } + + /** Ends the object: detached from its path, waits rejected, replicas ended; keeps nothing of its reference. */ + #end(error: ChordError): void { + const bound = this.#bound; + this.#state = { kind: "ended", error }; + this.#binding = undefined; + this.#dependent = undefined; + this.#hold = undefined; + this.#bound = undefined; + this.#settled.reject(error); + this.#path.detach(this); + for (const replica of bound?.replicas.values() ?? []) replica.end(error); + } + + /** + * Loading again until the next binding; the members keep their last values. A dependent's activation waiting for + * this hold while it was loading already fails at once with `error` (transient) and is retried when a new hold is + * bound (R18). + */ + #startOver(error: ChordError): void { + if (this.#state.kind === "ended") return; + if (this.#state === LOADING) { + this.#settled.reject(error); + this.#settled = settlement(); + } else { + this.#move(LOADING); + } + } + + /** Moves to `state` and tells the reference; `settled` follows (a new wait each time it is loading again). */ + #move(state: RefState): void { + const previous = this.#state; + this.#state = state; + if (state.kind === "loading" && previous.kind !== "loading") this.#settled = settlement(); + if (state.kind === "live") this.#settled.resolve(); + if (state.kind === "failed") this.#settled.reject(state.cause.error); + this.#binding?.moved(); + } +} diff --git a/packages/chord/src/next/remote/replica.ts b/packages/chord/src/next/remote/replica.ts new file mode 100644 index 000000000..e1c8ce7a3 --- /dev/null +++ b/packages/chord/src/next/remote/replica.ts @@ -0,0 +1,141 @@ +// Replicas (§7, §22): a state member of a remote reference on the consumer, built from the snapshot and then changed +// by the provider's ops, decoded with the member's own Delta decoder and applied without validation or copy +// (`applyImmutable` shares structure); replaced by the snapshot of each re-hold. + +import type { Context } from "../../context/index.ts"; +import { applyImmutable, decoder, type Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { ChordError } from "../data/index.ts"; +import { type Batch, handedOut, Source, type WatchStart } from "../state/internal.ts"; + +/** + * Where a replica stands: a value, a failed member (reads throw; the provider's member failed, or a newly bound variant + * lacks it, and either may come back; `last` is its last good value, `undefined` when it had none), or ended with its + * reference. + */ +type ReplicaState = + | { readonly kind: "value"; readonly value: JsonValue } + | { readonly kind: "failed"; readonly error: unknown; readonly last: JsonValue | undefined } + | { readonly kind: "ended"; readonly error: ChordError }; + +/** A member as a snapshot has it: a value, or failed. */ +export type MemberSnapshot = + | { readonly kind: "value"; readonly value: JsonValue } + | { readonly kind: "failed"; readonly error: unknown }; + +const NO_OPS: readonly Op[] = []; + +/** One state member of a hold, as the consumer reads and watches it; changed only inside the consumer's batch. */ +export class Replica extends Source { + #state: ReplicaState; + /** The member's decoder, started with the snapshot as the provider's encoder was; a new one per snapshot. */ + #decoder = decoder(); + /** The ops of the current batch's frame, and whether the provider's document was retired in it. */ + #ops = NO_OPS; + #retired = false; + /** A retirement that reached it while failed: its next value carries the flag. */ + #retiredWhileFailed = false; + + constructor(batch: Batch, state: MemberSnapshot) { + super(batch); + this.#state = state.kind === "value" ? state : { kind: "failed", error: handedOut(state.error), last: undefined }; + } + + get value(): JsonValue { + const state = this.#state; + if (state.kind === "value") return state.value; + throw state.error; + } + + /** Its value, or while failed its failure and last good value. */ + start(): WatchStart { + const state = this.#state; + if (state.kind === "failed") return { kind: "failed", error: state.error, last: state.last }; + return { kind: "value", value: this.value }; + } + + /** + * The value `wire` (this member's wire ops from one state part) makes, and its ops; changes nothing yet. Throws on + * ops that do not fit (a protocol error): an ended member's, and a failed member's other than its recovery, which + * the provider sends as `[["r", value]]`. + */ + decode(wire: unknown): { readonly value: JsonValue; readonly ops: readonly Op[] } { + // Trust boundary: fresh JSON from the link; Delta's decoder checks each op. + if (!Array.isArray(wire)) throw new Error("part: a member's changes are a list of ops"); + const ops = this.#decoder.decode(wire); + const state = this.#state; + if (state.kind === "ended") throw state.error; + if (state.kind === "value") return { value: applyImmutable(state.value, ops), ops }; + if (ops[0]?.[0] !== "r") throw new Error("part: a failed member recovers with a root replacement"); + return { value: applyImmutable(null, ops), ops }; + } + + /** + * Adopts the member's new value, the ops that made it and whether the provider's document was retired, in the + * current batch (one adoption per member and batch). + */ + adopt(value: JsonValue, ops: readonly Op[], retired: boolean): void { + const flagged = retired || this.#retiredWhileFailed; + this.#state = { kind: "value", value }; + this.#ops = ops; + this.#retired = flagged; + this.#retiredWhileFailed = false; + this.changed(); + if (flagged) this.retiredInBatch(); + } + + /** + * The provider's document was retired with no ops for this member (a derive whose value stayed, rule 21), in the + * current batch: the flag alone; while failed it waits for the next value (a failed member's value is never read). + */ + retireAlone(): void { + const state = this.#state; + if (state.kind === "value") this.adopt(state.value, NO_OPS, true); + else if (state.kind === "failed") this.#retiredWhileFailed = true; + } + + /** + * A re-hold's snapshot of the member, in the batch that applies it, with a fresh decoder (the provider's encoder + * starts anew with it): a value is one frame `[["r", value]]`, a failed member fails. + */ + reset(state: MemberSnapshot): void { + this.#decoder = decoder(); + if (state.kind === "failed") this.fail(state.error); + else this.adopt(state.value, [["r", state.value]], false); + } + + /** + * The provider's member failed, in the batch that applies it: reads throw `error`, watches get one failure call and + * stay open, derives over it fail; its recovery is `[["r", value]]`. + */ + fail(error: unknown): void { + const state = this.#state; + const last = state.kind === "value" ? state.value : state.kind === "failed" ? state.last : undefined; + this.#state = { kind: "failed", error: handedOut(error), last }; + // Failed already: its watches and derives have the failure (one call per failure episode). + if (state.kind !== "failed") this.changed(); + } + + deliver(context: Context): void { + const state = this.#state; + if (state.kind === "failed") this.failWatches(state.error, context); + else if (state.kind === "value") this.publish(state.value, this.#ops, this.#retired, context); + else this.endWatches(); + this.#ops = NO_OPS; + this.#retired = false; + } + + /** + * The member is not in a newly bound variant, in the batch that binds it: as a failure (`fail`) with `error`; a later + * binding of a variant that has it brings it back (`reset`), as on the provider's host. + */ + vanish(error: ChordError): void { + this.fail(error); + } + + /** Its reference ended (outside a batch): reads throw `error`, watches end `ended`. */ + end(error: ChordError): void { + this.#state = { kind: "ended", error }; + this.endWatches(); + } +} diff --git a/packages/chord/src/next/state/README.md b/packages/chord/src/next/state/README.md new file mode 100644 index 000000000..b386d167b --- /dev/null +++ b/packages/chord/src/next/state/README.md @@ -0,0 +1,70 @@ +# state + +Watchables, derives and the batch: how values change, are computed from each other and reach their watches. + +## For apps (`index.ts`) + +```ts +import { constant, mapShared } from "./index.ts"; + +const title = mapShared((note: Note) => note.text.slice(0, 20)); // called once per new note object +const titles = self.derive([notes], ([n]) => title(n.list)); +const info = constant({ dataDir: "/data" }); // a watchable that never changes +const watch = await titles.watch(ctx); +watch.start(async (call, ctx) => { + if (call.kind === "failed") return showError(call.error); // the watch stays open + render(call.value); +}); +``` + +- `Watchable` (from `src/watch.ts`): `value` and `watch(ctx)`, which returns Chord's `WatchHandle`: the value at + acquisition, then one call `(call, ctx)` per batch that changed it, in order, while the listener keeps up; batches + that arrive while it is busy fold into its next call (the newest value, the ops since its previous call, folded by + Delta's `PendingOps`). A call is `{ kind: "value", value, ops, retired }` or `{ kind: "failed", error }` (the source + cannot compute its value now: the watch stays open, `watch.value` keeps the last value, the next value arrives as + `[["r", value]]`). Values are JSON. A watch ends only for good (its owner ended, `stop()`, an aborted context, a + throwing listener). +- `constant(value)`: a watchable that never changes; a state member or a derive source. +- `mapShared(f)`: maps an array, calling `f` once per object element (by identity) and reusing its result while the + element stays the same object, so a derive over a list keeps unchanged parts identical and its diff cheap. + +Rules: + +- `self.derive(sources, compute)`: a derive has no name; errors name it by the member that exposes it, or by its + position (`derive #2`). Its `compute` is synchronous and pure; it reads only its declared sources. A derive that + throws fails only itself and the derives over it: their reads throw, their watches get one `failed` call per failure + episode (a recompute that throws again is quiet) and stay open, and its owner reports `derive_failed` once per + episode (not while its attempt activates: `activation_failed` covers that). It recomputes on every later change of + its sources (and on a re-bind of a member it reads) and heals as soon as it computes a value: its watches get it as + `[["r", value]]`, with `retired` when a source's document was retired during the episode. +- A watch opened on a failed source starts at its last good value and gets the failure as its first call, then the + heal; a source that never had a value (a remote member failed since its binding) rejects `watch` with its error. An + `on` routine over a failed source waits for its next good value; a value queued behind its busy call is dropped + when a failure follows it. +- An unwatched derive computes only when read. Reading `.value` once does not make it watched. +- Inside a commit's adoptions and in any derive compute, `self.tx`, `watch`, `release`, `get`, `load` and `close` + throw `invalid/reentrant`. A raw `constant(...)` belongs to no host, so its own `watch` is not checked; through a + reference it is. +- A frame delivery that throws (a derive whose value is no JSON) is reported to `onError` as `delivery_failed`; the + commit's other frames are still delivered. A throwing watch listener is reported as `listener_error`. + +## For Chord's layers (`internal.ts`) + +- `Batch`, one per host, reporting `BatchError`s (`listener_error`, `delivery_failed`; watches get `reportListener`): + `commit(adopt, ctx)` runs a commit's adoptions, then delivers one frame per watched source they changed, then runs + the flushes listed with `flushing(flush)` during it (a link sends one state message per batch, W1); + `within(adopt, ctx)` adopts a change inside the commit running now, or as a commit of its own, or, while a commit's + flushes send, as the next commit (presence and keep-alive counts: a departure found while sending lands there); + `run(work)` sets the reentrancy flag (nested); `assertOutside(operation)`; `count(counter)` counts derive work on + the host's metrics (`chord.derive.computes`, `chord.derive.diffs`). +- `Source`: a changing value on a host (a singleton document, a derive, a replica) with its watches, subscribers and + dependents (`Dependent`: derives, and member sources passing changes on to theirs). Subclasses call `changed()` when + a commit changes them and implement `deliver(ctx)` (`publish` a value, `failWatches` a recoverable failure, + `endWatches` for good). `openWatch(ctx)` is `watch` without the Promise. `subscribe(subscriber)`: a `Subscriber` (a + hold over a link, a member source) takes the watch's own sink shape, `advance(call, ctx)` synchronously inside each + batch that changes the source (a value or a failure call) and `end(end)` when the source ends; a subscriber counts + as watching, so a derive it follows recomputes in the batch. +- `Derive`: declared with its raw sources; `attach(resolve, report)` resolves them at activation (and drops the raw + list; `report` gets a failure of its compute, once per episode), `detach(error)` when its object ends (later reads + throw `error` for good), `invalidate()` from its sources (a failed derive too: it recomputes on its next read). +- `Constant`: the class behind `constant()`. `DeriveSource` is what a derive reads: a `Source` or a `Constant`. diff --git a/packages/chord/src/next/state/batch.ts b/packages/chord/src/next/state/batch.ts new file mode 100644 index 000000000..d58c5c347 --- /dev/null +++ b/packages/chord/src/next/state/batch.ts @@ -0,0 +1,118 @@ +// The batch (W1), one per host: a commit adopts inside it, then every watched value that changed gets one frame, then +// the flushes it collected run (a link sends one message); Chord calls that would reenter it are refused while it runs. + +import type { TelemetryMetrics } from "@earendil-works/pi-telemetry"; +import type { Context } from "../../context/index.ts"; +import { invalid } from "../data/internal.ts"; +import type { Source } from "./watchable.ts"; + +/** What a batch reports: a watch listener that threw, or a frame delivery that threw (a derive's value is no JSON). */ +export type BatchError = + | { readonly kind: "listener_error"; readonly error: unknown } + | { readonly kind: "delivery_failed"; readonly error: unknown }; + +/** A host's batch: the reentrancy flag, the watched sources a commit changed, and where background errors go. */ +export class Batch { + /** Reports a throwing watch listener's error (to the host's `onError`); what watches get as their `report`. */ + readonly reportListener: (error: unknown) => void; + readonly #report: (error: BatchError) => void; + readonly #metrics: TelemetryMetrics | undefined; + /** Set while adopting, delivering or computing a derive; one flag per host. */ + #running = false; + /** The watched sources the current commit changed, each listed once by its `changed`. */ + #changed: Source[] = []; + /** What runs once the current commit's frames were delivered, each listed once by its `flushing`. */ + #flushes: ((context: Context) => void)[] = []; + /** Set while the current commit's flushes run (its links send). */ + #sending = false; + /** Changes `within` deferred while the flushes sent, for the next commit. */ + #later: (() => void)[] = []; + + constructor(report: (error: BatchError) => void, metrics: TelemetryMetrics | undefined) { + this.#report = report; + this.reportListener = (error) => report({ kind: "listener_error", error }); + this.#metrics = metrics; + } + + /** Throws `invalid/reentrant` inside the batch: `operation` may not run during adoptions or derive computes. */ + assertOutside(operation: string): void { + if (this.#running) throw invalid("reentrant", `${operation} inside a commit or a derive compute`); + } + + /** Runs `work` with reentrant calls refused; nests (a derive compute inside a commit). */ + run(work: () => T): T { + const outer = this.#running; + this.#running = true; + try { + return work(); + } finally { + this.#running = outer; + } + } + + /** + * Runs one commit's adoptions, then, still synchronously, delivers a frame for every watched value they changed + * (watched derives recompute through their reads), then runs the flushes the deliveries listed. A delivery or flush + * that throws is reported; the others still run. What `within` deferred while the flushes sent runs as the next + * commit, right after. + */ + commit(adopt: () => void, context: Context): void { + this.run(() => { + try { + adopt(); + for (const source of this.#changed) { + try { + source.deliver(context); + } catch (error) { + this.#report({ kind: "delivery_failed", error }); + } + } + this.#sending = true; + for (const flush of this.#flushes) { + try { + flush(context); + } catch (error) { + this.#report({ kind: "delivery_failed", error }); + } + } + } finally { + this.#sending = false; + this.#changed = []; + this.#flushes = []; + } + }); + const later = this.#later; + if (later.length === 0) return; + this.#later = []; + this.commit(() => { + for (const each of later) each(); + }, context); + } + + /** + * Runs `adopt` inside the commit running now, which delivers what it changed; outside one, as a commit of its own; + * while a commit's flushes send, as the next commit (a link closing while it sends ends its holds: their departure + * lands in the next batch, so one commit stays one message per link and origin). For changes that happen wherever a + * reference binds or ends (`host.references`, `host.activity`). + */ + within(adopt: () => void, context: Context): void { + if (this.#sending) this.#later.push(adopt); + else if (this.#running) adopt(); + else this.commit(adopt, context); + } + + /** Lists a watched source the current commit changed. */ + changed(source: Source): void { + this.#changed.push(source); + } + + /** Lists `flush` to run at the end of the current commit, after its deliveries (a link sending what they fed it). */ + flushing(flush: (context: Context) => void): void { + this.#flushes.push(flush); + } + + /** Counts one unit of derive work on the host's metrics (a lazy read has no context to count through). */ + count(counter: "chord.derive.computes" | "chord.derive.diffs"): void { + this.#metrics?.counter(counter).add(1); + } +} diff --git a/packages/chord/src/next/state/derive.ts b/packages/chord/src/next/state/derive.ts new file mode 100644 index 000000000..6bb86f617 --- /dev/null +++ b/packages/chord/src/next/state/derive.ts @@ -0,0 +1,191 @@ +// Derives (§12): values computed from watchables, pulled when read and marked dirty by commits; and `mapShared`, which +// keeps unchanged elements identical so a derive's diff stays cheap. + +import type { Context } from "../../context/index.ts"; +import { diffRevisions, type Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Batch } from "./batch.ts"; +import { type Constant, type Dependent, handedOut, Source, type WatchStart } from "./watchable.ts"; + +/** What a derive reads: a source it depends on, or a constant. */ +export type DeriveSource = Source | Constant; + +/** + * Where a derive stands. `dirty`: a source changed since the last compute (or there was none). A failed derive (its + * compute threw) throws its error until a source it reads changes or is re-bound, then recomputes (self-healing, design + * §11); a detached one (its object ended) throws its object's error for good. + */ +type DeriveState = + | { readonly kind: "dirty" } + | { readonly kind: "clean"; readonly value: JsonValue } + | { readonly kind: "failed"; readonly error: unknown } + | { readonly kind: "detached"; readonly error: unknown }; + +const DIRTY: DeriveState = { kind: "dirty" }; + +const NO_OPS: readonly Op[] = []; + +/** What a failed derive's watches had: no value, so its next value frame replaces the value whole. */ +const NO_VALUE: unique symbol = Symbol("no value"); + +/** + * A derive: `compute` over its sources' values, recomputed on read when dirty, so each compute runs at most once per + * batch and after its sources. A commit only marks it; the batch recomputes it only when something watched depends on + * it. + */ +export class Derive extends Source implements Dependent { + readonly #compute: (values: readonly JsonValue[]) => JsonValue; + /** The sources as declared, until `attach` resolves them. */ + #declared: readonly unknown[] | undefined; + #sources: readonly DeriveSource[] = []; + #state: DeriveState = { kind: "dirty" }; + /** The value this derive's watches have, kept from its marking to its frame in the batch. */ + #framed: JsonValue | typeof NO_VALUE | undefined; + /** Whether a source's document was retired in the current batch while this derive is watched (rule 21). */ + #retired = false; + /** Reports a failure of its compute to its owner (`attach`); once per failure episode, which a value ends. */ + #report: ((error: unknown) => void) | undefined; + #failing = false; + /** Its last good value, which a watch opened while it is failed starts at (`NO_VALUE` before the first). */ + #good: JsonValue | typeof NO_VALUE = NO_VALUE; + + constructor(sources: readonly unknown[], compute: (values: readonly JsonValue[]) => JsonValue, batch: Batch) { + super(batch); + this.#declared = sources; + this.#compute = compute; + } + + get value(): JsonValue { + const state = this.#state; + if (state.kind === "clean") return state.value; + if (state.kind === "dirty") return this.#recompute(); + throw state.error; + } + + /** Its value, or while failed its failure and last good value (a detached derive throws). */ + start(): WatchStart { + try { + return { kind: "value", value: this.value }; + } catch (error) { + if (this.#state.kind !== "failed") throw error; + return { kind: "failed", error, last: this.#good === NO_VALUE ? undefined : this.#good }; + } + } + + /** + * Starts reading its sources, as `resolve` maps the declared ones (at activation); the first read computes. + * `report` gets a throw of its compute, once per failure episode (a value ends it); a failing source's error is its + * source's to report. + */ + attach(resolve: (source: unknown, index: number) => DeriveSource, report: (error: unknown) => void): void { + this.#report = report; + this.#sources = this.#declared!.map(resolve); + this.#declared = undefined; + for (const source of this.#sources) if (source instanceof Source) source.dependents.add(this); + } + + /** Its object ended: no source keeps it, its watches end `ended`, reads throw `error` (the object's). */ + detach(error: Error): void { + for (const source of this.#sources) if (source instanceof Source) source.dependents.delete(this); + this.#sources = []; + this.#declared = undefined; + this.#state = { kind: "detached", error }; + this.#framed = undefined; + this.endWatches(); + } + + /** + * A source changed: mark this derive and its dependents dirty, flags only; a failed one too, which recomputes on its + * next read and heals when its sources give a value it can compute. An already dirty derive has dirty dependents + * (every read recomputes its sources first), so marking stops there. + */ + invalidate(): void { + const state = this.#state; + if (state.kind === "dirty" || state.kind === "detached") return; + if (this.watched) this.#framed = state.kind === "clean" ? state.value : NO_VALUE; + this.#state = DIRTY; + this.changed(); + } + + /** + * A source's document was retired in this batch (`Source.retiredInBatch`): a watched derive's frame carries the + * flag, and the derives reading this one are told too. + */ + sourceRetired(visited: Set): void { + // Once per derive per retirement: a diamond of derives is walked in its size, not once per path. + if (visited.has(this)) return; + visited.add(this); + if (this.watched) this.#retired = true; + for (const derive of this.dependents) derive.sourceRetired(visited); + } + + /** + * Delivers the frame: the diff from the watches' value, a retirement even without ops (rule 21). A compute that + * throws gives one failure call when the watches had a value (one per failure episode), the watches stay open; a + * retirement waits for the next value. + */ + deliver(context: Context): void { + const before = this.#framed!; + const retired = this.#retired; + this.#framed = undefined; + let after: JsonValue; + try { + after = this.value; + } catch (error) { + // Still failed since the last call: nothing new for the watches. + if (before !== NO_VALUE) this.failWatches(error, context); + return; + } + this.#retired = false; + let ops: readonly Op[] = NO_OPS; + if (after !== before) { + this.batch.count("chord.derive.diffs"); + ops = before === NO_VALUE ? [["r", after]] : diffRevisions(before, after); + } + if (ops.length > 0 || retired) this.publish(after, ops, retired, context); + } + + #recompute(): JsonValue { + return this.batch.run(() => { + try { + this.batch.count("chord.derive.computes"); + // A failing source throws its own error here, so a derive over a failed derive fails with the same error. + const values = this.#sources.map((source) => source.value); + let value: JsonValue; + try { + value = this.#compute(values); + } catch (error) { + if (!this.#failing) this.#report?.(handedOut(error)); + this.#failing = true; + throw error; + } + this.#failing = false; + this.#state = { kind: "clean", value }; + this.#good = value; + return value; + } catch (error) { + this.#state = { kind: "failed", error }; + throw error; + } + }); + } +} + +/** + * Maps an array calling `f` only for object elements not seen before (by identity), returning the previous result for + * the others; primitive elements are mapped every time. The cache is weak: dropped elements take their results along. + */ +export function mapShared(f: (element: E) => R): (elements: readonly E[]) => R[] { + const results = new WeakMap(); + return (elements) => + elements.map((element) => { + // The decided rule: identity caching for objects only. + if (typeof element !== "object" || element === null) return f(element); + let result = results.get(element); + if (result === undefined) { + result = f(element); + results.set(element, result); + } + return result; + }); +} diff --git a/packages/chord/src/next/state/index.ts b/packages/chord/src/next/state/index.ts new file mode 100644 index 000000000..8fbd9ad1e --- /dev/null +++ b/packages/chord/src/next/state/index.ts @@ -0,0 +1,5 @@ +// State for apps: watchables, constants, and the derive helper `mapShared`. + +export type { Watchable } from "../../watch.ts"; +export { mapShared } from "./derive.ts"; +export { constant } from "./watchable.ts"; diff --git a/packages/chord/src/next/state/internal.ts b/packages/chord/src/next/state/internal.ts new file mode 100644 index 000000000..5ecd3641f --- /dev/null +++ b/packages/chord/src/next/state/internal.ts @@ -0,0 +1,5 @@ +// State for Chord's layers: the batch, sources, derives and constants behind the app's watchables. + +export { Batch, type BatchError } from "./batch.ts"; +export { Derive, type DeriveSource } from "./derive.ts"; +export { Constant, type Dependent, handedOut, Source, type Subscriber, type WatchStart } from "./watchable.ts"; diff --git a/packages/chord/src/next/state/watchable.ts b/packages/chord/src/next/state/watchable.ts new file mode 100644 index 000000000..8bdbac622 --- /dev/null +++ b/packages/chord/src/next/state/watchable.ts @@ -0,0 +1,190 @@ +// Chord's JSON watchables: sources (documents and derives, with their watches and dependent derives) and constants. + +import type { Context } from "../../context/index.ts"; +import type { Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import { Watch, type Watchable, type WatchCall, type WatchEnd } from "../../watch.ts"; +import type { Batch } from "./batch.ts"; + +/** + * A synchronous subscriber of a source (a hold over a link, a member source): fed the source's call inside the batch + * that made it, before the batch ends, so everything one commit changed can leave in one message. The watch's own sink + * shape (`Watch.advance`, `Watch.end`). + */ +export interface Subscriber { + /** + * The source's call of this batch, in its batch's `context`: a value (the ops from the previous one, and whether its + * document was retired in it, or for a derive one of its sources' documents), or a failure it may recover from; the + * next value after a failure is `[["r", value]]`. + */ + advance(call: WatchCall, context: Context): void; + /** The source ended for good with `end`; nothing follows, and the subscriber is dropped. */ + end(end: WatchEnd): void; +} + +/** + * What a new watch of a source starts at (design §11): its value, or, while it is failed, the failure and its last good + * value (`undefined` when it never had one: a watch cannot start then, there is nothing to start at). + */ +export type WatchStart = + | { readonly kind: "value"; readonly value: JsonValue } + | { readonly kind: "failed"; readonly error: unknown; readonly last: JsonValue | undefined }; + +/** + * An error Chord hands to apps (a failure call, a report), its stack made text first: until then V8 keeps the frames' + * receivers, which would keep the host behind the throw alive for as long as an app keeps the error. + */ +export function handedOut(error: E): E { + if (error instanceof Error) void error.stack; + return error; +} + +/** + * What reads a source and is marked by its changes: a derive, or a member source forwarding to derives (`Source. + * dependents`). + */ +export interface Dependent { + /** The source changed in the current batch: flags only, no compute. */ + invalidate(): void; + /** A source's document was retired in the current batch (rule 21); `visited` keeps a diamond walked once. */ + sourceRetired(visited: Set): void; +} + +const ENDED: WatchEnd = { kind: "ended" }; + +/** + * A changing value on a host (a singleton document, a derive, a replica): its watches and subscribers get one frame + * per batch that changed it, and the derives reading it are marked dirty when it changes. + */ +export abstract class Source implements Watchable { + /** Derives reading this source (and member sources forwarding to some); each removes itself when it stops. */ + readonly dependents = new Set(); + protected readonly batch: Batch; + readonly #watches = new Set>(); + readonly #subscribers = new Set(); + + constructor(batch: Batch) { + this.batch = batch; + } + + abstract get value(): JsonValue; + + /** Whether a watch or a subscriber follows this source. */ + get watched(): boolean { + return this.#watches.size > 0 || this.#subscribers.size > 0; + } + + async watch(context: Context): Promise> { + return this.openWatch(context); + } + + /** + * `watch` without the Promise, for references, which must register the watch before anything else runs. On a failed + * source it opens at the last good value and its first call is the failure; with no value to start at it throws the + * failure. + */ + openWatch(context: Context): Watch { + this.batch.assertOutside("watch"); + const start = this.start(); + if (start.kind === "failed" && start.last === undefined) throw start.error; + const value = start.kind === "value" ? start.value : start.last!; + const watch: Watch = new Watch(value, () => this.#watches.delete(watch), this.batch.reportListener); + this.#watches.add(watch); + if (context.abortSignal !== undefined) watch.observeAbort(context.abortSignal); + if (start.kind === "failed") watch.fail(start.error, context); + return watch; + } + + /** What a new watch starts at; a source that can fail and recover tells its failure (`WatchStart`). */ + start(): WatchStart { + return { kind: "value", value: this.value }; + } + + /** Feeds `subscriber` every frame from now on, inside each batch; the caller took the current value first. */ + subscribe(subscriber: Subscriber): void { + this.#subscribers.add(subscriber); + } + + unsubscribe(subscriber: Subscriber): void { + this.#subscribers.delete(subscriber); + } + + /** Delivers this batch's frame to its watches; called once per batch that listed it with `changed`. */ + abstract deliver(context: Context): void; + + /** Records a change in the current batch: listed for a frame when watched, and every dependent marked dirty. */ + protected changed(): void { + if (this.watched) this.batch.changed(this); + for (const derive of this.dependents) derive.invalidate(); + } + + /** + * Records a retirement of this source's document in the current batch: every derive reading it, directly or + * through other derives, carries the flag on in its frame (rule 21). Called after `changed`. + */ + protected retiredInBatch(): void { + // Allocated only for a retirement, which is rare. + const visited = new Set(); + for (const derive of this.dependents) derive.sourceRetired(visited); + } + + /** Queues one frame on every watch and feeds it to every subscriber. */ + protected publish(value: JsonValue, ops: readonly Op[], retired: boolean, context: Context): void { + this.forward({ kind: "value", value, ops, retired }, context); + } + + /** Queues `call` on every watch and feeds it to every subscriber, as it is (a member source passes one on). */ + protected forward(call: WatchCall, context: Context): void { + for (const watch of this.#watches) watch.advance(call, context); + for (const subscriber of this.#subscribers) subscriber.advance(call, context); + } + + /** + * Gives every watch and subscriber one failure call (design §11): the source cannot compute its value now and may + * recover; they stay open, and its next value is `[["r", value]]`. + */ + protected failWatches(error: unknown, context: Context): void { + this.forward({ kind: "failed", error: handedOut(error) }, context); + } + + /** Ends every watch and subscriber for good (its owner ended): `ended`; nothing follows. */ + protected endWatches(): void { + // Ending detaches, which deletes the current entry (safe while iterating) and runs no listener. + for (const watch of this.#watches) watch.end(ENDED); + if (this.#subscribers.size === 0) return; + // A copy: a subscriber may unsubscribe as it ends. + const subscribers = [...this.#subscribers]; + this.#subscribers.clear(); + for (const subscriber of subscribers) subscriber.end(ENDED); + } +} + +/** A watchable that never changes (start data, §18); usable as a state member and as a derive source. */ +export function constant(value: T): Watchable { + // The constant holds `value` as it was given. + return new Constant(value) as unknown as Watchable; +} + +/** The watchable behind `constant(value)`: its watches never get a frame and end only by stop or cancellation. */ +export class Constant implements Watchable { + readonly value: JsonValue; + + constructor(value: JsonValue) { + this.value = value; + } + + async watch(context: Context): Promise> { + return this.openWatch(context); + } + + /** As `Source.openWatch`; nothing is delivered, so nothing is reported. */ + openWatch(context: Context): Watch { + const watch = new Watch( + this.value, + () => {}, + () => {}, + ); + if (context.abortSignal !== undefined) watch.observeAbort(context.abortSignal); + return watch; + } +} diff --git a/packages/chord/src/next/store/README.md b/packages/chord/src/next/store/README.md new file mode 100644 index 000000000..933fb6df6 --- /dev/null +++ b/packages/chord/src/next/store/README.md @@ -0,0 +1,67 @@ +# store + +The store (slice 6a): one per host, opened by the app over a backend (SQLite: `chord/sqlite/node`, Durable Objects' +storage), holding the host's tables, persisted documents and outbox; one commit queue serializing only the commit +step; eager table migrations; export and import. + +## For apps (`index.ts`) + +```ts +import { defineTable, evolveTable, importStore, json, openStore, query, sqliteBackend } from "./index.ts"; + +const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), ctx, { outbox: true }); +const host = createHost({ store, onError }); // one open host per store; the app closes the store after it + +const EditsV1 = defineTable("edits", { + version: 1, + columns: { id: str(), path: str(), at: int(), done: bool({ default: false }), meta: opt(json(Meta, { version: 1 })) }, + key: ["id"], + indexes: { by_path: ["path", "at"] }, +}); +const EditsV2 = evolveTable(EditsV1, (t) => t.add("dir", str({ default: "" })).rows((row) => ({ dir: dirname(row.path) }))); + +host.serve(History, (_, self) => { // a local type with empty params (P3) + const edits = self.table(EditsV2); // created or migrated at activation + const q = query(EditsV2).where((e) => prefix(e.path, "src/")).orderBy((e) => e.at, "desc"); + return { + add: async (row, ctx) => self.tx(async (tx) => tx.put(edits, row), ctx), + list: async ({ after }, ctx) => edits.page(q, 50, ctx, { after }), + }; +}); + +for await (const part of store.export(ctx)) write(part); // or encodeJsonl(store.export(ctx)) +const copy = await importStore(sqliteBackend(db), decodeJsonl(bytes), ctx); +``` + +- `openStore(backend, ctx, { outbox? })`: the store's own schema migrations, a UTF-8 check, the metadata read once; + `outbox` turns the outbox on (from the current `lastSeq`) or off; omitted, the stored setting stays. A failure + closes the backend. `store.lastSeq`, `store.outbox` (`read(afterSeq, limit, ctx)` from its floor, `trim(throughSeq, ctx)`), + `store.close(ctx)` (idempotent; waits for a running step; `ctx` ends only the caller's wait). +- `defineTable(name, { version, columns, key, indexes? })`: flat columns of `str int num bool`, `opt(...)` of them, + `json(schema, { version, migrate? })` (one JSON cell, migrated on access); defaults; a key and index columns of + non-`opt` `str`/`int`. `evolveTable(prev, (t) => t…)` appends the next version: `rename`, `add`, `json` before an + optional data step (`rows(fn)` or `update({ where?, set })`), `drop`, `index`, `dropIndex` after it. +- Queries: `query(Table).where((t) => cond).orderBy((t) => col, dir)` with `eq ne lt lte gt gte inArray isNull prefix + and or not` (SQL's `NULL` rules); pages `{ items, next? }` with opaque cursors, `rows()` paging by 100, `count`. +- Writes in `self.tx`: `put` (an upsert), `update(key, set)`, `delete(key)`, `updateWhere(where, set)`, + `deleteWhere(where)`; a `set` is values or expressions (`add sub max min concat replacePrefix coalesce when`); key + columns never change. `tx.get` sees the transaction's own single-key writes, not after a bulk one (W10). +- Text is UTF-8 compared by code points (`compareText`); NUL is refused, lone surrogates replaced. +- `store.export(ctx)` holds the commit queue from its first `next()` until it ends, is closed or `ctx` aborts; parts: + the manifest, per table its `table` write, `put`s and `index` writes, the live documents, `end`. `importStore` + checks every part (a trust boundary) and applies the archive in one backend transaction; JSONL via `encodeJsonl` and + `decodeJsonl`. + +## For Chord's layers + +`ChordStore` (`store.ts`: `commit(staged, fence, ctx)`, `read(fn, ctx)` guarded as Durable's `GuardedStorage`, +`take()` for an export, the catalog), the commit `Step` and `StagedWrite`s in their `WriteOrder` (`step.ts`), the documents engine +(`engine.ts`), the catalog (`catalog.ts`), the write vocabulary (`writes.ts`, `table-writes.ts`), rows and columns +(`rows.ts`, `columns.ts`, `json-column.ts`), the compiled where/set vocabulary (`expression.ts`) and its JS evaluator (`evaluate.ts`), +SQL (`sql.ts`), the table handle and a transaction's table changes (`table-handle.ts`, `table-changes.ts`), table +histories and migrations (`table.ts`, `table-evolve.ts`, `table-history.ts`, `table-migration.ts`), the outbox +(`outbox.ts`), archives (`archive.ts`, `export.ts`, `import.ts`, `applier.ts`, `jsonl.ts`), the store's SQL schema +(`schema.ts`) and backends (`backend.ts`). + +Tests: `test/next/store/` (one file per source); the benchmarks in `test/next/bench/store.ts` (results in +`docs/next/benchmarks.md`). Examples: `20-tables.ts`, `22-migrations.ts`, `25-export-import.ts`. diff --git a/packages/chord/src/next/store/applier.ts b/packages/chord/src/next/store/applier.ts new file mode 100644 index 000000000..f6e55e214 --- /dev/null +++ b/packages/chord/src/next/store/applier.ts @@ -0,0 +1,163 @@ +// The applier (slice 6a §13): recorded writes applied as they were recorded, inside a backend transaction, checked +// against the store's own records (a trust boundary: archives, later a replica's batches). One code path for import, +// bootstrap and, from slice 6b, batch apply; 6a applies the kinds an archive holds. + +import type { SqliteValue } from "../../sqlite/index.ts"; +import { checkName } from "../contracts/internal.ts"; +import { invalid } from "../data/internal.ts"; +import type { ArchivePart } from "./archive.ts"; +import type { TableRecord } from "./catalog.ts"; +import { COLUMN_NAME, type ColumnRecord } from "./columns.ts"; +import { insertDocument } from "./engine.ts"; +import { checkScalar, ROW_VERSION, toSqlite, versionColumn } from "./rows.ts"; +import { recordedPut } from "./sql.ts"; +import type { Step } from "./step.ts"; +import { INDEX_NAME } from "./table.ts"; +import { applyIndex, applyTable, type TableWrite } from "./table-writes.ts"; + +/** A recorded write the applier takes: the writes of an archive. */ +export type AppliedWrite = Exclude; + +/** Applies `write` in `step`; a write the records refuse throws `invalid/args` (`what` names its place). */ +export async function applyWrite(step: Step, write: AppliedWrite, what: string): Promise { + switch (write.kind) { + case "table": + checkTable(write, what); + await applyTable(step, write); + return; + case "index": + checkIndex(recordOf(step, write.table, what), write, what); + await applyIndex(step, write); + return; + case "put": { + const record = recordOf(step, write.table, what); + const { sql, physical } = recordedPut(write.table, record.columns); + await step.run(sql, ...putParams(record, write, physical, what)); + return; + } + case "document.create": { + let text: string; + try { + text = JSON.stringify(write.base.value); + } catch (error) { + throw invalid("args", `${what}: the base is not JSON (${String(error)})`); + } + await insertDocument(step, write.record, text); + return; + } + } +} + +/** A recorded table's SQL name (three names joined by `:`), columns, key and json versions, by the name rules. */ +function checkTable(write: Extract, what: string): void { + const parts = write.table.split(":"); + if (parts.length !== 3) throw invalid("args", `${what}: table ${JSON.stringify(write.table)}: not kind:type:table`); + try { + for (const part of parts) checkName(part, `${what}: table name part`); + } catch (error) { + // `checkName` throws plain Errors. + throw invalid("args", (error as Error).message); + } + const folded = new Set(); + for (const [name, column] of Object.entries(write.columns)) { + const where = `${what}: column ${JSON.stringify(name)}`; + if (!COLUMN_NAME.test(name) || folded.has(name.toLowerCase())) + throw invalid("args", `${where}: not a column name`); + folded.add(name.toLowerCase()); + checkColumn(column, where); + } + if (write.key.length === 0) throw invalid("args", `${what}: the key has no column`); + checkOrderColumns(write.columns, write.key, `${what}: the key`); +} + +function checkColumn(column: ColumnRecord, what: string): void { + const json = column.type === "json"; + if (json !== (column.version !== undefined) || json !== (column.fingerprint !== undefined)) { + throw invalid("args", `${what}: a json(...) column has a version and a fingerprint, no other column has`); + } + if (column.default === undefined) return; + if (json || column.optional) throw invalid("args", `${what}: only a non-opt scalar column has a default`); + checkScalar(column.type, column.default, `${what}: the default`); +} + +/** `names` are distinct, recorded, non-`opt` `str` or `int` columns. */ +function checkOrderColumns( + columns: { readonly [name: string]: ColumnRecord }, + names: readonly string[], + what: string, +): void { + if (new Set(names).size !== names.length) throw invalid("args", `${what}: a column occurs twice`); + for (const name of names) { + const column = Object.hasOwn(columns, name) ? columns[name] : undefined; + if (column === undefined || column.optional || (column.type !== "str" && column.type !== "int")) { + throw invalid("args", `${what}: ${JSON.stringify(name)} is not a non-opt str or int column`); + } + } +} + +function checkIndex(record: TableRecord, write: Extract, what: string): void { + if (!INDEX_NAME.test(write.name) || Object.hasOwn(record.indexes, write.name)) { + throw invalid("args", `${what}: index ${JSON.stringify(write.name)}: not a new index name`); + } + checkOrderColumns(record.columns, write.columns, `${what}: index ${write.name}`); +} + +/** The record of `table` in `step`; a write for a table not recorded is refused. */ +function recordOf(step: Step, table: string, what: string): TableRecord { + const record = step.table(table); + if (record === undefined) throw invalid("args", `${what}: table ${JSON.stringify(table)} is not recorded`); + return record; +} + +/** + * The parameters of a `put` in `physical` order, checked against the recorded columns: every non-`opt` column, no + * other field, values of their types, `json(...)` cells as JSON (no plugin code runs) with a version up to the + * column's, and `version` up to the table's. + */ +function putParams( + record: TableRecord, + write: Extract, + physical: readonly string[], + what: string, +): SqliteValue[] { + const { row, columns: versions, version } = write; + for (const field of Object.keys(row)) { + if (!Object.hasOwn(record.columns, field)) + throw invalid("args", `${what}: unknown column ${JSON.stringify(field)}`); + } + for (const field of Object.keys(versions)) { + if (record.columns[field]?.type !== "json" || !Object.hasOwn(row, field)) { + throw invalid( + "args", + `${what}: a version for ${JSON.stringify(field)}, which is no json(...) cell of the row`, + ); + } + } + if (version > record.version) { + throw invalid("args", `${what}: version ${version} is newer than the table's version ${record.version}`); + } + const cells = new Map(); + for (const [name, column] of Object.entries(record.columns)) { + const where = `${what}: column ${JSON.stringify(name)}`; + const value = row[name]; + // A row has no explicit `null`: an absent field is `NULL`; a `json(...)` cell's `null` is the JSON value. + if (column.type === "json" ? !Object.hasOwn(row, name) : value === undefined) { + if (!column.optional) throw invalid("args", `${where} is missing`); + cells.set(name, null); + continue; + } + if (column.type !== "json") { + cells.set(name, toSqlite(checkScalar(column.type, value, where))); + continue; + } + const version = versions[name]; + if (version === undefined || version > column.version!) throw invalid("args", `${where}: no readable version`); + try { + cells.set(name, JSON.stringify(value)); + } catch (error) { + throw invalid("args", `${where}: not JSON (${String(error)})`); + } + cells.set(versionColumn(name), version); + } + return physical.map((name) => (name === ROW_VERSION ? version : (cells.get(name) ?? null))); +} diff --git a/packages/chord/src/next/store/archive.ts b/packages/chord/src/next/store/archive.ts new file mode 100644 index 000000000..963cfae48 --- /dev/null +++ b/packages/chord/src/next/store/archive.ts @@ -0,0 +1,106 @@ +// Archives (slice 6a §13; decisions "Export archives are operations", "Writes carry their versions; archives are +// lossless"): the parts an export yields and an import takes, their order, and the one compiled schema every +// imported part is checked against first. + +import Type, { type TInteger } from "typebox"; +import type { Context } from "../../context/index.ts"; +import { arr, bool, int, lit, num, obj, opt, str, union, unknown } from "../contracts/index.ts"; +import { invalid, validatorFor } from "../data/internal.ts"; +import type { DocumentWrite } from "./engine.ts"; +import type { TableWrite } from "./table-writes.ts"; + +/** The archive format `export` writes; `import` takes this one only. */ +export const ARCHIVE_FORMAT = 1; + +/** The first part: the format, the store's schema version, its `lastSeq` and the next document id (rule 24). */ +export type ArchiveManifest = { + readonly kind: "manifest"; + readonly format: number; + readonly schema: number; + readonly lastSeq: number; + readonly nextDocumentId: number; +}; + +/** + * One part of an archive, in this order: the manifest; per table (by SQL name) its `table` write, its `put`s and its + * `index` writes; the live documents' `document.create`s (by id, the current value as the base); `end` with the + * count of the parts between. + */ +export type ArchivePart = + | ArchiveManifest + | Extract + | Extract + | { readonly kind: "end"; readonly writes: number }; + +/** An integer from `min` up to 2^53 - 1: seqs, ids, versions and counts stay exact (a seq after 2^53 would repeat). */ +function safe(min: number): TInteger { + return int({ min, max: Number.MAX_SAFE_INTEGER }); +} + +const version = safe(1); + +const column = obj({ + type: union([lit("str"), lit("int"), lit("num"), lit("bool"), lit("json")]), + optional: opt(lit(true)), + default: opt(union([str(), num(), bool()])), + version: opt(version), + fingerprint: opt(str()), +}); + +const PART = union([ + obj({ + kind: lit("manifest"), + format: safe(1), + schema: safe(1), + lastSeq: safe(0), + nextDocumentId: safe(1), + }), + obj({ + kind: lit("table"), + table: str(), + version, + state: union([lit("ready"), lit("migrating")]), + columns: Type.Record(Type.String(), column), + key: arr(str()), + fingerprint: str(), + }), + obj({ + kind: lit("put"), + table: str(), + row: Type.Record(Type.String(), unknown()), + version, + columns: Type.Record(Type.String(), version), + }), + obj({ kind: lit("index"), table: str(), name: str(), columns: arr(str()) }), + obj({ + kind: lit("document.create"), + record: obj({ + id: safe(1), + namespace: str(), + owner: str(), + kind: str(), + // Absent for a singleton. + key: opt(str()), + history: union([lit("latest"), lit("rewindable")]), + createdAt: safe(1), + meta: opt(str()), + revision: safe(1), + version, + fingerprint: str(), + }), + base: obj({ version, fingerprint: str(), value: unknown() }), + }), + obj({ kind: lit("end"), writes: safe(0) }), +]); + +const PART_SCHEMA = { schema: PART }; + +/** Checks one imported part against the archive schema (a trust boundary); `invalid/args` naming its position. */ +export function checkPart(part: unknown, position: number, context: Context): ArchivePart { + context.telemetry.metrics?.counter("chord.data.validations").add(1); + if (!validatorFor(PART_SCHEMA, context).Check(part)) { + throw invalid("args", `import: part ${position}: not an archive part`); + } + // The check above establishes the type. + return part as ArchivePart; +} diff --git a/packages/chord/src/next/store/backend.ts b/packages/chord/src/next/store/backend.ts new file mode 100644 index 000000000..3d1a52a84 --- /dev/null +++ b/packages/chord/src/next/store/backend.ts @@ -0,0 +1,19 @@ +// Store backends (slice 6a §2): what a store runs on. One kind today, any `SqliteDatabase` (Node through +// `chord/sqlite/node`, Durable Objects through `chord/sqlite/cloudflare`); its interface is internal to the store. + +import type { SqliteDatabase } from "../../sqlite/index.ts"; + +/** A backend for `openStore` and `importStore`: opaque to apps, owned by the store once given to it. */ +export class Backend { + /** The database the store runs its SQL on. */ + readonly db: SqliteDatabase; + + constructor(db: SqliteDatabase) { + this.db = db; + } +} + +/** A backend over a SQLite database (`openNodeSqliteDatabase(path)`, a Durable Object's storage, `:memory:` in tests). */ +export function sqliteBackend(db: SqliteDatabase): Backend { + return new Backend(db); +} diff --git a/packages/chord/src/next/store/catalog.ts b/packages/chord/src/next/store/catalog.ts new file mode 100644 index 000000000..ae2fc9f6c --- /dev/null +++ b/packages/chord/src/next/store/catalog.ts @@ -0,0 +1,37 @@ +// The catalog (slice 6a §2, §5): the table definitions the store records (version, state, columns, key, fingerprint, +// indexes), read once at open, changed only by `table`, `index`, `dropIndex` and `alter` writes inside a step, and in +// memory again only after `COMMIT` returned. + +import type { SqliteExecutor } from "../../sqlite/index.ts"; +import type { ColumnRecord } from "./columns.ts"; +import { TABLES } from "./schema.ts"; +import type { Step } from "./step.ts"; + +/** A table as the store records it, by its SQL name. */ +export type TableRecord = { + readonly table: string; + readonly version: number; + /** `migrating` while an eager migration of `version` runs (slice 6a §9). */ + readonly state: "ready" | "migrating"; + readonly columns: { readonly [name: string]: ColumnRecord }; + readonly key: readonly string[]; + readonly fingerprint: string; + /** Index definitions by name: their declared columns (the key is appended when rendered). */ + readonly indexes: { readonly [name: string]: readonly string[] }; +}; + +/** Every recorded table, by SQL name. */ +export async function readCatalog(executor: SqliteExecutor): Promise> { + const rows = await executor.all<{ readonly name: string; readonly definition: string }>( + `SELECT name, definition FROM ${TABLES.tables}`, + ); + return new Map(rows.map(({ name, definition }) => [name, JSON.parse(definition) as TableRecord])); +} + +/** Writes the table records `step` changed, once each, inside its backend transaction. */ +export async function writeCatalog(step: Step): Promise { + for (const [name, record] of step.tables) { + const definition = JSON.stringify(record); + await step.run(`INSERT OR REPLACE INTO ${TABLES.tables} (name, definition) VALUES (?, ?)`, name, definition); + } +} diff --git a/packages/chord/src/next/store/columns.ts b/packages/chord/src/next/store/columns.ts new file mode 100644 index 000000000..8d4118262 --- /dev/null +++ b/packages/chord/src/next/store/columns.ts @@ -0,0 +1,153 @@ +// Table columns (decisions "Flat columns", "Nullable columns and the `set` form", "Restricted names"): the restricted +// subset of L1's helpers a table's `columns` take (`str int num bool`, `opt(...)` of them, `json(...)`, defaults), +// their types for rows, keys and column references, and the column descriptors and records the store keeps. + +import type { TBoolean, TInteger, TNumber, TOptional, TString } from "typebox"; +import type { Static, TSchema } from "../contracts/index.ts"; +import { JsonColumn, type JsonColumnSchema } from "./json-column.ts"; +import { storedText } from "./text.ts"; + +/** A schema a table column takes. */ +export type ColumnSchema = + | ScalarSchema + | JsonColumnSchema + | TOptional>; + +type ScalarSchema = TString | TInteger | TNumber | TBoolean; + +/** A table's `columns`: a flat record of column schemas. */ +export type Columns = { readonly [name: string]: ColumnSchema }; + +/** A column's type in the store. */ +export type ColumnType = "str" | "int" | "num" | "bool" | "json"; + +/** The store type of column schema `S`. */ +export type ColumnTypeOf = S extends TOptional + ? ColumnTypeOf + : S extends TString + ? "str" + : S extends TInteger + ? "int" + : S extends TNumber + ? "num" + : S extends TBoolean + ? "bool" + : "json"; + +/** The value type of column schema `S` (without `undefined` for an `opt(...)` one). */ +export type ValueOf = S extends TOptional ? ValueOf : S extends TSchema ? Static : never; + +/** The names of the columns that are not `opt(...)`. */ +type RequiredNames = { [K in keyof C]: C[K] extends TOptional ? never : K }[keyof C]; + +/** The names of the columns with a declared default. */ +type DefaultedNames = { + [K in keyof C]: C[K] extends { readonly default: unknown } ? K : never; +}[keyof C]; + +/** A row as reads return it: every non-`opt` column, the `opt(...)` ones when not `NULL`. */ +export type RowOf = { [K in RequiredNames]: ValueOf } & { + [K in Exclude>]?: ValueOf; +}; + +/** A row as `put` takes it: as read, the columns with a default may be left out (rule 15). */ +export type PutRowOf = { [K in Exclude, DefaultedNames>]: ValueOf } & { + [K in Exclude, DefaultedNames>>]?: ValueOf; +}; + +/** The columns a key, an index or an `orderBy` may use: non-`opt` `str` or `int`. */ +export type OrderColumn = { + [K in keyof C]: C[K] extends TString | TInteger ? (C[K] extends TOptional ? never : K) : never; +}[keyof C] & + string; + +/** A column of the store: its name, type, nullability and default; a `json(...)` one with its definition. */ +export type Column = + | { + readonly name: string; + readonly type: "str" | "int" | "num" | "bool"; + readonly optional: boolean; + readonly default: string | number | boolean | undefined; + } + | { readonly name: string; readonly type: "json"; readonly optional: boolean; readonly json: JsonColumn }; + +/** A column as the `table` write records it (slice 6a §5): what a replica needs without the plugin's code. */ +export type ColumnRecord = { + readonly type: ColumnType; + readonly optional?: true; + readonly default?: string | number | boolean; + /** A `json(...)` column's version and schema fingerprint. */ + readonly version?: number; + readonly fingerprint?: string; +}; + +/** The column name rule (decision "Restricted names"). */ +export const COLUMN_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/; + +/** + * The columns of `columns` in declaration order, checked (declaration policy: JavaScript callers reach it): names + * `[A-Za-z_][A-Za-z0-9_]*` without two equal ignoring ASCII case, only the helpers' types, no bounds or patterns (a + * bulk `set` could not keep them), a default of the column's type and not on an `opt(...)` column. + */ +export function describeColumns(columns: Columns, table: string): Column[] { + const seen = new Set(); + return Object.entries(columns).map(([name, schema]) => { + checkColumnName(name, table); + const folded = name.toLowerCase(); + if (seen.has(folded)) throw new Error(`${table}: column "${name}" differs from another only in case`); + seen.add(folded); + return describeColumn(name, schema, `${table}: column "${name}"`); + }); +} + +/** Throws unless `name` follows the column name rule (declaration policy); `what` names the table or the edit. */ +export function checkColumnName(name: string, what: string): void { + if (!COLUMN_NAME.test(name)) throw new Error(`${what}: column "${name}": expected [A-Za-z_][A-Za-z0-9_]*`); +} + +const SCALAR_TYPES: { readonly [type: string]: "str" | "int" | "num" | "bool" } = { + string: "str", + integer: "int", + number: "num", + boolean: "bool", +}; + +function describeColumn(name: string, schema: ColumnSchema, what: string): Column { + const optional = Object.hasOwn(schema, "~optional"); + // Declaration policy: a JavaScript caller may pass any schema, so its fields are read as unknown data. + const fields = schema as unknown as { readonly [key: string]: unknown }; + const json = fields.json; + if (json instanceof JsonColumn) return { name, type: "json", optional, json }; + const type = SCALAR_TYPES[String(fields.type)]; + if (type === undefined) throw new Error(`${what}: expected str(), int(), num(), bool(), opt(...) or json(...)`); + for (const key of Object.keys(schema)) { + if (key !== "type" && key !== "default") throw new Error(`${what}: "${key}" is not supported in a table column`); + } + const value = fields.default; + if (value !== undefined) { + if (optional) throw new Error(`${what}: an opt(...) column has no default`); + checkDefault(type, value, what); + } + return { name, type, optional, default: value as string | number | boolean | undefined }; +} + +function checkDefault(type: "str" | "int" | "num" | "bool", value: unknown, what: string): void { + const valid = + type === "str" + ? typeof value === "string" && storedText(value, what) === value + : type === "int" + ? Number.isSafeInteger(value) + : type === "num" + ? typeof value === "number" && Number.isFinite(value) + : typeof value === "boolean"; + if (!valid) throw new Error(`${what}: the default ${JSON.stringify(value)} is not a ${type} value`); +} + +/** The record of `column` in the `table` write. */ +export function columnRecord(column: Column): ColumnRecord { + const optional = column.optional ? { optional: true as const } : {}; + if (column.type === "json") { + return { type: "json", ...optional, version: column.json.version, fingerprint: column.json.fingerprint }; + } + return { type: column.type, ...optional, ...(column.default === undefined ? {} : { default: column.default }) }; +} diff --git a/packages/chord/src/next/store/engine.ts b/packages/chord/src/next/store/engine.ts new file mode 100644 index 000000000..b5fbc8f80 --- /dev/null +++ b/packages/chord/src/next/store/engine.ts @@ -0,0 +1,295 @@ +// The documents engine (S13, slice 6a §8): one record per incarnation of an address, revisions made of a base and +// later deltas; its writes inside the commit step, checked by the writes themselves (the partial unique index on the +// live address, the counted conditional `UPDATE` on the record's revision), and its plain reads. + +import { apply, type Op } from "../../delta/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { SqliteExecutor } from "../../sqlite/index.ts"; +import { TABLES } from "./schema.ts"; +import type { StagedWrite, Step } from "./step.ts"; + +/** + * A document's address: the adapter's namespace, the owner string, the definition's name and, for a keyed document, its + * key; a singleton has none (as Durable's addresses). The column keeps `""` for it, which a keyed document's empty key + * shares (its definition says which it is): a record read back gives no key for either. + */ +export type DocumentAddress = { + readonly namespace: string; + readonly owner: string; + readonly kind: string; + readonly key?: string; +}; + +/** A stored record: one incarnation of an address. Chord keeps only the latest revision (`history: "latest"`). */ +export type DocumentRecord = DocumentAddress & { + readonly id: number; + readonly history: "latest" | "rewindable"; + /** The seq that created it. */ + readonly createdAt: number; + readonly meta?: string; + /** Counts its changes: a change names the revision it was prepared on. */ + readonly revision: number; + /** The version and schema fingerprint of its current base. */ + readonly version: number; + readonly fingerprint: string; +}; + +/** A base: a whole value at a version. */ +export type DocumentBase = { readonly version: number; readonly fingerprint: string; readonly value: JsonValue }; + +/** A change's content: a new base, or a delta on the current base's version. */ +export type DocumentContent = + | ({ readonly kind: "base" } & DocumentBase) + | { readonly kind: "delta"; readonly version: number; readonly ops: readonly Op[] }; + +/** The engine's writes as the outbox and archives record them. */ +export type DocumentWrite = + | { readonly kind: "document.create"; readonly record: DocumentRecord; readonly base: DocumentBase } + | { + readonly kind: "document.change"; + readonly id: number; + readonly revision: number; + readonly content: DocumentContent; + } + | { readonly kind: "document.retire"; readonly id: number; readonly revision: number }; + +/** A loaded document: its record and its value (base plus deltas applied once). */ +export type StoredDocument = { + readonly record: DocumentRecord; + readonly value: JsonValue; + /** The deltas stored after its base (what `checkpointWhen` sees). */ + readonly deltasSinceBase: number; +}; + +/** + * Stages the creation of a document at `address` with `base`; its id comes from the step. `text` is the base's value + * serialized before the queue (rule 8). + */ +export function createDocument( + address: DocumentAddress, + history: "latest" | "rewindable", + base: DocumentBase, + text: string, +): StagedWrite { + return { + order: "create", + apply: async (step) => { + // The next id is stored too: it stays in the safe range, or ids would stop counting exactly and repeat. + if (step.nextDocumentId >= Number.MAX_SAFE_INTEGER) { + throw new Error("store: no document id past Number.MAX_SAFE_INTEGER"); + } + const id = step.nextDocumentId++; + step.created.push(id); + const { version, fingerprint } = base; + const record: DocumentRecord = { + id, + ...address, + history, + createdAt: step.seq, + revision: 1, + version, + fingerprint, + }; + await insertDocument(step, record, text); + return { kind: "document.create", record, base }; + }, + }; +} + +/** + * Inserts `record` as it is (id, `createdAt`, revision kept: a creation, or an import) with its base, `text` being + * the base's value serialized. + */ +export async function insertDocument(step: Step, record: DocumentRecord, text: string): Promise { + const { id, namespace, owner, kind, key, history, createdAt, meta, revision, version, fingerprint } = record; + await step.run( + `INSERT INTO ${TABLES.documents} + (id, namespace, owner, kind, key, history, created_at, meta, revision, version, fingerprint) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`, + id, + namespace, + owner, + kind, + key ?? "", + history, + createdAt, + meta ?? null, + revision, + version, + fingerprint, + ); + await insertRevision(step, id, "base", version, text); +} + +/** + * Stages a change of document `id` prepared on `revision`: a delta on the current base's version, or a new base + * (which prunes the older revisions of a `latest` record). `text` is the content serialized before the queue. + */ +export function changeDocument(id: number, revision: number, content: DocumentContent, text: string): StagedWrite { + const write: DocumentWrite = { kind: "document.change", id, revision, content }; + return { + order: "write", + apply: async (step) => { + // As ids: a revision past the safe range would stop counting exactly. + if (revision >= Number.MAX_SAFE_INTEGER) throw new Error(`document ${id}: no revision past the safe range`); + await insertRevision(step, id, content.kind, content.version, text); + const changed = + content.kind === "delta" + ? await step.runCounted( + `UPDATE ${TABLES.documents} SET revision = revision + 1 + WHERE id = ? AND revision = ? AND version = ? AND retired_at IS NULL`, + id, + revision, + content.version, + ) + : await step.runCounted( + `UPDATE ${TABLES.documents} SET revision = revision + 1, version = ?, fingerprint = ? + WHERE id = ? AND revision = ? AND retired_at IS NULL`, + content.version, + content.fingerprint, + id, + revision, + ); + // Memory and disk disagree: the step fails, which stops the store (Durable's rule). + if (changed !== 1) throw new Error(`document ${id}: not live at revision ${revision}`); + if (content.kind === "base") { + await step.run(`DELETE FROM ${TABLES.revisions} WHERE document_id = ? AND seq < ?`, id, step.seq); + } + return write; + }, + }; +} + +/** Stages the retirement of `latest` document `id` at `revision`: its record and revisions go (rule 19). */ +export function retireDocument(id: number, revision: number): StagedWrite { + const write: DocumentWrite = { kind: "document.retire", id, revision }; + return { + order: "retire", + apply: async (step) => { + const deleted = await step.runCounted( + `DELETE FROM ${TABLES.documents} WHERE id = ? AND revision = ? AND retired_at IS NULL`, + id, + revision, + ); + if (deleted !== 1) throw new Error(`document ${id}: not live at revision ${revision}`); + await step.run(`DELETE FROM ${TABLES.revisions} WHERE document_id = ?`, id); + return write; + }, + }; +} + +type RecordRow = { + readonly id: number; + readonly namespace: string; + readonly owner: string; + readonly kind: string; + readonly key: string; + readonly history: "latest" | "rewindable"; + readonly created_at: number; + readonly meta: string | null; + readonly revision: number; + readonly version: number; + readonly fingerprint: string; +}; + +const RECORD_COLUMNS = "id, namespace, owner, kind, key, history, created_at, meta, revision, version, fingerprint"; + +/** The live record at `address`, if any. */ +export async function findDocument( + executor: SqliteExecutor, + address: DocumentAddress, +): Promise { + const row = await executor.get( + `SELECT ${RECORD_COLUMNS} FROM ${TABLES.documents} + WHERE namespace = ? AND owner = ? AND kind = ? AND key = ? AND retired_at IS NULL`, + address.namespace, + address.owner, + address.kind, + address.key ?? "", + ); + return row === undefined ? undefined : recordOf(row); +} + +type RevisionRow = { + readonly seq: number; + readonly kind: "base" | "delta"; + readonly version: number; + readonly content: string; +}; + +/** Loads the live document `record` found: its latest base and the deltas after it, applied once. */ +export async function loadDocument(executor: SqliteExecutor, record: DocumentRecord): Promise { + const revisions = await executor.all( + `SELECT seq, kind, version, content FROM ${TABLES.revisions} WHERE document_id = ? AND seq >= ( + SELECT max(seq) FROM ${TABLES.revisions} WHERE document_id = ? AND kind = 'base') ORDER BY seq`, + record.id, + record.id, + ); + const [base, ...deltas] = revisions; + if (base === undefined || base.kind !== "base") throw new Error(`document ${record.id}: no base`); + let value = JSON.parse(base.content) as JsonValue; + for (const delta of deltas) { + if (delta.version !== base.version) throw new Error(`document ${record.id}: a delta crosses a version`); + value = apply(value, JSON.parse(delta.content) as Op[]); + } + return { record, value, deltasSinceBase: deltas.length }; +} + +/** Up to `limit` keys of live documents of `kind` under `owner` after `after` (from the first), in code point order. */ +export async function listDocumentKeys( + executor: SqliteExecutor, + address: Omit, + after: string | undefined, + limit: number, +): Promise { + // The first page has no lower bound: `""` is a key too. + const from = after === undefined ? "" : " AND key > ?"; + const rows = await executor.all<{ readonly key: string }>( + `SELECT key FROM ${TABLES.documents} WHERE namespace = ? AND owner = ? AND kind = ?${from} + AND retired_at IS NULL ORDER BY key LIMIT ?`, + address.namespace, + address.owner, + address.kind, + ...(after === undefined ? [] : [after]), + limit, + ); + return rows.map(({ key }) => key); +} + +/** Up to `limit` live records with ids above `after`, by id (an export's pages). */ +export async function listDocuments(executor: SqliteExecutor, after: number, limit: number): Promise { + const rows = await executor.all( + `SELECT ${RECORD_COLUMNS} FROM ${TABLES.documents} WHERE id > ? AND retired_at IS NULL ORDER BY id LIMIT ?`, + after, + limit, + ); + return rows.map(recordOf); +} + +function recordOf(row: RecordRow): DocumentRecord { + const { id, namespace, owner, kind, key, history, created_at, meta, revision, version, fingerprint } = row; + return { + id, + namespace, + owner, + kind, + ...(key === "" ? {} : { key }), + history, + createdAt: created_at, + ...(meta === null ? {} : { meta }), + revision, + version, + fingerprint, + }; +} + +function insertRevision(step: Step, id: number, kind: "base" | "delta", version: number, text: string): Promise { + return step.run( + `INSERT INTO ${TABLES.revisions} (document_id, seq, kind, version, content) VALUES (?, ?, ?, ?, ?)`, + id, + step.seq, + kind, + version, + text, + ); +} diff --git a/packages/chord/src/next/store/evaluate.ts b/packages/chord/src/next/store/evaluate.ts new file mode 100644 index 000000000..0699c83db --- /dev/null +++ b/packages/chord/src/next/store/evaluate.ts @@ -0,0 +1,131 @@ +// The JavaScript evaluator of the compiled table vocabulary (slice 6a §6; rule 10): the `tx.get` overlay matches +// `where` nodes and applies `set` nodes with SQL's three-valued `NULL` rules and code point text order, as the SQL +// that `sql.ts` renders from the same nodes does in SQLite. + +import type { JsonValue } from "../../json.ts"; +import { invalid } from "../data/internal.ts"; +import type { Comparison, ExprNode, Operation, SetNode, WhereNode } from "./expression.ts"; +import type { Scalar } from "./rows.ts"; +import { compareText } from "./text.ts"; + +/** A row as the evaluator reads it: stored cells, an absent column being `NULL`. */ +export type EvaluatedRow = { readonly [column: string]: Scalar | JsonValue | undefined }; + +/** SQL's three-valued truth: `null` is unknown (a comparison with `NULL`). */ +type Truth = boolean | null; + +/** Whether `row` matches `node` under SQL's rules: unknown (`NULL`) does not match. */ +export function matches(node: WhereNode, row: EvaluatedRow): boolean { + return truth(node, row) === true; +} + +function truth(node: WhereNode, row: EvaluatedRow): Truth { + if ("and" in node) + return node.and.reduce((all, each) => (all === false ? false : both(all, truth(each, row))), true); + if ("or" in node) + return node.or.reduce((any, each) => (any === true ? true : either(any, truth(each, row))), false); + if ("not" in node) { + const inner = truth(node.not, row); + return inner === null ? null : !inner; + } + if ("isNull" in node) return row[node.isNull] === undefined; + // The row is checked against the table the condition was compiled for: its columns hold scalars. + if ("inArray" in node) { + const [column, values] = node.inArray; + // SQL: `x IN ()` is false, for a `NULL` x too. + if (values.length === 0) return false; + const value = row[column] as Scalar | undefined; + return value === undefined ? null : values.some((each) => compareScalar(value, each) === 0); + } + if ("prefix" in node) { + const value = row[node.prefix[0]] as string | undefined; + return value === undefined ? null : value.startsWith(node.prefix[1]); + } + // The remaining nodes are the comparisons, one entry each; `Object.entries` loses that. + const [op, [column, operand]] = Object.entries(node)[0] as [Comparison, readonly [string, Scalar]]; + const value = row[column] as Scalar | undefined; + if (value === undefined) return null; + const order = compareScalar(value, operand); + switch (op) { + case "eq": + return order === 0; + case "ne": + return order !== 0; + case "lt": + return order < 0; + case "lte": + return order <= 0; + case "gt": + return order > 0; + case "gte": + return order >= 0; + } +} + +function both(a: Truth, b: Truth): Truth { + if (a === false || b === false) return false; + return a === null || b === null ? null : true; +} + +function either(a: Truth, b: Truth): Truth { + if (a === true || b === true) return true; + return a === null || b === null ? null : false; +} + +/** Orders two cells of one column as SQLite does: text by code point, booleans as 0 and 1, numbers numerically. */ +export function compareScalar(a: Scalar, b: Scalar): number { + if (typeof a === "string") return compareText(a, b as string); + return Number(a) - Number(b); +} + +const SAFE = Number.MAX_SAFE_INTEGER; + +/** + * `row` after `set`, every right side evaluated against `row` (simultaneous: `{ a: b, b: a }` swaps); an integer + * result outside ±(2^53 - 1) throws `invalid/args` (rule 10), as the column's `CHECK` fails the step. + */ +export function applySet(set: SetNode, row: EvaluatedRow, table: string): EvaluatedRow { + const next: { [column: string]: Scalar | JsonValue | undefined } = { ...row }; + for (const [column, node] of Object.entries(set)) { + const value = evaluate(node, row, table); + if (value === undefined) delete next[column]; + else next[column] = value; + } + return next; +} + +function evaluate(node: ExprNode, row: EvaluatedRow, table: string): Scalar | JsonValue | undefined { + if (node === null) return undefined; + if (typeof node !== "object") return node; + if ("col" in node) return row[node.col]; + if ("json" in node) return node.json; + if ("when" in node) { + const [condition, a, b] = node.when; + return evaluate(matches(condition, row) ? a : b, row, table); + } + if ("coalesce" in node) return evaluate(node.coalesce[0], row, table) ?? evaluate(node.coalesce[1], row, table); + if ("replacePrefix" in node) { + const [subject, from, to] = node.replacePrefix; + const value = evaluate(subject, row, table) as string | undefined; + return value === undefined || !value.startsWith(from) ? value : to + value.slice(from.length); + } + // The remaining nodes are the arithmetic and text operations, one entry each, over scalar operands (compiled so). + const [op, [x, y]] = Object.entries(node)[0] as [Operation, readonly [ExprNode, ExprNode]]; + const a = evaluate(x, row, table) as Scalar | undefined; + const b = evaluate(y, row, table) as Scalar | undefined; + if (a === undefined || b === undefined) return undefined; + switch (op) { + case "concat": + return (a as string) + (b as string); + case "max": + return compareScalar(a, b) >= 0 ? a : b; + case "min": + return compareScalar(a, b) <= 0 ? a : b; + default: { + const result = op === "add" ? (a as number) + (b as number) : (a as number) - (b as number); + if (result > SAFE || result < -SAFE) + throw invalid("args", `${table}: ${op} gives ${result}, outside ±(2^53 - 1)`); + return result; + } + } +} diff --git a/packages/chord/src/next/store/export.ts b/packages/chord/src/next/store/export.ts new file mode 100644 index 000000000..cdadc0424 --- /dev/null +++ b/packages/chord/src/next/store/export.ts @@ -0,0 +1,207 @@ +// `store.export(ctx)` (slice 6a §13, rule 26): the store's state after its `lastSeq` as archive parts, read page by +// page with plain reads while the export holds the commit queue (no commit can interleave, no long read transaction). +// The queue is taken at the first `next()` and given back when the iteration ends, throws, is closed or `ctx` aborts; +// an abort while the consumer is suspended gives it back at once and drops the pages and records the export held, and +// an abort while it waits for the queue ends the wait. Closing the store ends it too (`released`). + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { SqliteValue } from "../../sqlite/index.ts"; +import { ChordError } from "../data/index.ts"; +import { ARCHIVE_FORMAT, type ArchivePart } from "./archive.ts"; +import type { TableRecord } from "./catalog.ts"; +import { listDocuments, loadDocument } from "./engine.ts"; +import { ROW_VERSION, type Scalar, versionColumn } from "./rows.ts"; +import { STORE_SCHEMA } from "./schema.ts"; +import { quote, recordedPut } from "./sql.ts"; +import type { ChordStore } from "./store.ts"; + +/** The rows and documents an export reads per page. */ +export const EXPORT_PAGE = 100; + +/** + * An export of `store` under `context`: an async iterator of its parts. One pass; after an abort `next()` rejects + * `aborted`, after the store closed `released`. + */ +export class StoreExport implements AsyncIterableIterator { + readonly #context: Context; + /** The parts, until the export ends; dropped then, with the pages they hold. */ + #parts: AsyncGenerator | undefined; + #state: "open" | "aborted" | "closed" | "ended" = "open"; + /** Gives the commit queue back while the export holds it. */ + #release: (() => void) | undefined; + readonly #abort = (): void => this.#end("aborted"); + + constructor(store: ChordStore, context: Context) { + this.#context = context; + const hold = (release: () => void): void => this.#hold(release); + this.#parts = parts(store, context, hold, () => this.#end("closed")); + } + + [Symbol.asyncIterator](): this { + return this; + } + + async next(): Promise> { + if (this.#context.abortSignal?.aborted) this.#end("aborted"); + this.#assertOpen(); + if (this.#parts === undefined) return { done: true, value: undefined }; + const pending = this.#parts.next(); + let result: IteratorResult; + try { + // Until the export holds the queue only `context` ends the wait; from then on its abort listener does. + result = await (this.#release === undefined ? awaitWithContext(pending, this.#context) : pending); + } catch (error) { + if (this.#context.abortSignal?.aborted) this.#end("aborted"); + this.#assertOpen(); + this.#end("ended"); + throw error; + } + // Ended meanwhile (an abort, the store's close): no part after that, even one that needed no read. + this.#assertOpen(); + if (result.done) this.#end("ended"); + return result; + } + + async return(): Promise> { + this.#end("ended"); + return { done: true, value: undefined }; + } + + /** The parts took the queue: the export holds it until it ends, and an abort ends it; ended already, it gives it back. */ + #hold(release: () => void): void { + if (this.#state !== "open") { + release(); + return; + } + this.#release = release; + this.#context.abortSignal?.addEventListener("abort", this.#abort, { once: true }); + } + + /** Ends the export once: the queue given back, the parts closed (once they reach a `yield`) and dropped. */ + #end(state: "aborted" | "closed" | "ended"): void { + if (this.#state !== "open") return; + this.#state = state; + this.#release?.(); + this.#release = undefined; + this.#context.abortSignal?.removeEventListener("abort", this.#abort); + void this.#parts?.return(undefined).catch(() => {}); + this.#parts = undefined; + } + + /** Throws `aborted` after an abort, `released` after the store closed. */ + #assertOpen(): void { + if (this.#state === "aborted") throw aborted(this.#context); + if (this.#state === "closed") + throw new ChordError({ code: "released", reason: "closed", message: "store: closed" }); + } +} + +/** The error of a use of an aborted export: `aborted`, the signal's reason as the cause. */ +function aborted(context: Context): ChordError { + return new ChordError({ code: "aborted", message: "store: export aborted" }, { cause: context.abortSignal?.reason }); +} + +/** + * The parts in archive order, after taking the queue (handed to `hold`; the store's close calls `end`); the queue is + * given back when they end. + */ +async function* parts( + store: ChordStore, + context: Context, + hold: (release: () => void) => void, + end: () => void, +): AsyncGenerator { + const release = await store.take(end); + hold(release); + try { + if (context.abortSignal?.aborted) throw aborted(context); + const { lastSeq, nextDocumentId } = store; + yield { kind: "manifest", format: ARCHIVE_FORMAT, schema: STORE_SCHEMA, lastSeq, nextDocumentId }; + let writes = 0; + const tables = [...store.tables.values()].sort((a, b) => (a.table < b.table ? -1 : 1)); + for (const record of tables) { + const { table, version, state, columns, key, fingerprint } = record; + yield { kind: "table", table, version, state, columns, key, fingerprint }; + writes++; + for await (const put of puts(store, record, context)) { + yield put; + writes++; + } + for (const [name, on] of Object.entries(record.indexes)) { + yield { kind: "index", table, name, columns: on }; + writes++; + } + } + for await (const create of documents(store, context)) { + yield create; + writes++; + } + yield { kind: "end", writes }; + } finally { + release(); + } +} + +/** The `put`s of a recorded table in key order, a page per read: the stored values and their versions. */ +async function* puts( + store: ChordStore, + record: TableRecord, + context: Context, +): AsyncGenerator> { + const { table, columns, key } = record; + const select = recordedPut(table, columns) + .physical.map((name) => quote(name)) + .join(", "); + const order = key.map(quote).join(", "); + let after: SqliteValue[] | undefined; + do { + const page = after; + const from = page === undefined ? "" : ` WHERE (${order}) > (${page.map(() => "?").join(", ")})`; + const sql = `SELECT ${select} FROM ${quote(table)}${from} ORDER BY ${order} LIMIT ?`; + const rows = await store.read( + (executor) => executor.all<{ readonly [column: string]: SqliteValue }>(sql, ...(page ?? []), EXPORT_PAGE), + context, + ); + for (const stored of rows) yield putOf(record, stored); + after = rows.length < EXPORT_PAGE ? undefined : key.map((name) => rows.at(-1)![name]!); + } while (after !== undefined); +} + +/** A stored row as its `put`: `NULL` cells absent, booleans from 0 and 1, `json(...)` cells parsed with versions. */ +function putOf( + record: TableRecord, + stored: { readonly [column: string]: SqliteValue }, +): Extract { + const row: { [column: string]: Scalar | JsonValue } = {}; + const versions: { [column: string]: number } = {}; + for (const [name, column] of Object.entries(record.columns)) { + const value = stored[name]; + if (value === null || value === undefined) continue; + // The recorded columns' types say what SQLite stored (STRICT tables): text for `json(...)` with its version. + if (column.type === "bool") row[name] = value === 1; + else if (column.type === "json") { + row[name] = JSON.parse(value as string) as JsonValue; + versions[name] = stored[versionColumn(name)] as number; + } else row[name] = value as Scalar; + } + return { kind: "put", table: record.table, row, version: stored[ROW_VERSION] as number, columns: versions }; +} + +/** The live documents by id as their creations: the record and the current value as the base. */ +async function* documents( + store: ChordStore, + context: Context, +): AsyncGenerator> { + let after = 0; + for (;;) { + const records = await store.read((executor) => listDocuments(executor, after, EXPORT_PAGE), context); + for (const record of records) { + const { value } = await store.read((executor) => loadDocument(executor, record), context); + const { version, fingerprint } = record; + yield { kind: "document.create", record, base: { version, fingerprint, value } }; + } + if (records.length < EXPORT_PAGE) return; + after = records.at(-1)!.id; + } +} diff --git a/packages/chord/src/next/store/expression.ts b/packages/chord/src/next/store/expression.ts new file mode 100644 index 000000000..f62f47cf4 --- /dev/null +++ b/packages/chord/src/next/store/expression.ts @@ -0,0 +1,356 @@ +// The table expression language (slice 6a §6, §7; rules 10, 13, 14, 16; decision "Nullable columns and the `set` +// form"): the builders plugins write `where` and `set` with, and their compilation at staging into one data-only +// vocabulary (what the outbox and archives carry; `evaluate.ts` runs it in JavaScript, `sql.ts` renders it). + +import type { TOptional } from "typebox"; +import type { JsonValue } from "../../json.ts"; +import type { TSchema } from "../contracts/index.ts"; +import { invalid } from "../data/internal.ts"; +import type { Column, Columns, ColumnType, ColumnTypeOf, ValueOf } from "./columns.ts"; +import { checkScalar, encodeCell, type Scalar } from "./rows.ts"; +import type { TableDefinition } from "./table.ts"; + +/** A compiled condition: column comparisons with checked operands, combined by `and`, `or`, `not`. */ +export type WhereNode = + | { readonly eq: readonly [string, Scalar] } + | { readonly ne: readonly [string, Scalar] } + | { readonly lt: readonly [string, Scalar] } + | { readonly lte: readonly [string, Scalar] } + | { readonly gt: readonly [string, Scalar] } + | { readonly gte: readonly [string, Scalar] } + | { readonly inArray: readonly [string, readonly Scalar[]] } + | { readonly isNull: string } + | { readonly prefix: readonly [string, string] } + | { readonly and: readonly WhereNode[] } + | { readonly or: readonly WhereNode[] } + | { readonly not: WhereNode }; + +/** A compiled value expression: a literal (`null` clears), a column, a `json(...)` cell, or an operation. */ +export type ExprNode = + | Scalar + | null + | { readonly col: string } + | { readonly json: JsonValue } + | { readonly add: readonly [ExprNode, ExprNode] } + | { readonly sub: readonly [ExprNode, ExprNode] } + | { readonly max: readonly [ExprNode, ExprNode] } + | { readonly min: readonly [ExprNode, ExprNode] } + | { readonly concat: readonly [ExprNode, ExprNode] } + | { readonly coalesce: readonly [ExprNode, ExprNode] } + | { readonly replacePrefix: readonly [ExprNode, string, string] } + | { readonly when: readonly [WhereNode, ExprNode, ExprNode] }; + +/** A compiled `set`: the new value of each changed column, every right side evaluated against the row before. */ +export type SetNode = { readonly [column: string]: ExprNode }; + +/** The comparison operators of a `WhereNode`. */ +export type Comparison = "eq" | "ne" | "lt" | "lte" | "gt" | "gte"; + +/** The operations of an `ExprNode`. */ +export type Operation = "add" | "sub" | "max" | "min" | "concat" | "coalesce" | "replacePrefix" | "when"; + +/** A value expression as a plugin builds it: `T` is its value, `K` its column type. Compiled when staged. */ +export class Expression { + readonly op: Operation | "col"; + readonly args: readonly unknown[]; + /** Type only. */ + declare readonly "~value"?: T; + declare readonly "~type"?: K; + + constructor(op: Operation | "col", args: readonly unknown[]) { + this.op = op; + this.args = args; + } +} + +/** A reference to a column of the row, as `(t) => …` callbacks get them. */ +export class ColumnRef extends Expression { + readonly column: string; + + constructor(column: string) { + super("col", [column]); + this.column = column; + } +} + +/** A condition as a plugin builds it; compiled when staged or run. */ +export class Condition { + readonly op: Comparison | "inArray" | "isNull" | "prefix" | "and" | "or" | "not"; + readonly args: readonly unknown[]; + + constructor(op: Condition["op"], args: readonly unknown[]) { + this.op = op; + this.args = args; + } +} + +/** + * A `set` of a table with columns `C` and key `K`: new values of non-key columns, each a value or an expression of + * the column's type; `undefined` clears an `opt(...)` column. + */ +export type SetOf = { + readonly [P in Exclude]?: C[P] extends TOptional + ? ValueOf | Expression, ColumnTypeOf> | undefined + : ValueOf | Expression, ColumnTypeOf>; +}; + +/** A scalar column type. */ +type ScalarType = Exclude; + +/** An operand of type `T` and column type `K`: an expression or a plain value. */ +type Operand = Expression | T; + +/** The column equals `value` (`NULL` never matches). */ +export function eq(column: ColumnRef, value: T): Condition { + return new Condition("eq", [column, value]); +} + +/** The column differs from `value` (`NULL` never matches). */ +export function ne(column: ColumnRef, value: T): Condition { + return new Condition("ne", [column, value]); +} + +/** The column is below `value`. */ +export function lt(column: ColumnRef, value: T): Condition { + return new Condition("lt", [column, value]); +} + +/** The column is at most `value`. */ +export function lte(column: ColumnRef, value: T): Condition { + return new Condition("lte", [column, value]); +} + +/** The column is above `value`. */ +export function gt(column: ColumnRef, value: T): Condition { + return new Condition("gt", [column, value]); +} + +/** The column is at least `value`. */ +export function gte(column: ColumnRef, value: T): Condition { + return new Condition("gte", [column, value]); +} + +/** The column equals one of `values` (`[]` matches nothing). */ +export function inArray(column: ColumnRef, values: readonly T[]): Condition { + return new Condition("inArray", [column, values]); +} + +/** The column is `NULL` (an `opt(...)` column without a value). */ +export function isNull(column: ColumnRef): Condition { + return new Condition("isNull", [column]); +} + +/** The text column starts with `prefix`, compared by code point (`""` matches every value). */ +export function prefix(column: ColumnRef, prefix: string): Condition { + return new Condition("prefix", [column, prefix]); +} + +/** Every condition holds (`and()` always does). */ +export function and(...conditions: Condition[]): Condition { + return new Condition("and", conditions); +} + +/** Some condition holds (`or()` never does). */ +export function or(...conditions: Condition[]): Condition { + return new Condition("or", conditions); +} + +/** The condition does not hold (`NULL` stays `NULL`: it never matches). */ +export function not(condition: Condition): Condition { + return new Condition("not", [condition]); +} + +/** `a + b` over integers; a result outside ±(2^53 - 1) fails (rule 10). */ +export function add(a: Operand, b: Operand): Expression { + return new Expression("add", [a, b]); +} + +/** `a - b` over integers; a result outside ±(2^53 - 1) fails (rule 10). */ +export function sub(a: Operand, b: Operand): Expression { + return new Expression("sub", [a, b]); +} + +/** The larger of `a` and b` (`int` or `num`). */ +export function max(a: Operand, b: Operand): Expression { + return new Expression("max", [a, b]); +} + +/** The smaller of `a` and `b` (`int` or `num`). */ +export function min(a: Operand, b: Operand): Expression { + return new Expression("min", [a, b]); +} + +/** `a` followed by `b`. */ +export function concat(a: Operand, b: Operand): Expression { + return new Expression("concat", [a, b]); +} + +/** `value` with its prefix `from` replaced by `to`; a value without the prefix stays as it is. */ +export function replacePrefix(value: Operand, from: string, to: string): Expression { + return new Expression("replacePrefix", [value, from, to]); +} + +/** `a` unless it is `NULL`, else `b`. */ +export function coalesce(a: Expression, b: Operand): Expression { + return new Expression("coalesce", [a, b]); +} + +/** `a` where `condition` holds for the row, else `b`. */ +export function when( + condition: Condition, + a: Operand, + b: Operand, +): Expression { + return new Expression("when", [condition, a, b]); +} + +/** One `ColumnRef` per column of `definition`, for its `(t) => …` callbacks. */ +export function columnRefs(columns: readonly Column[]): { readonly [column: string]: ColumnRef } { + return Object.fromEntries(columns.map(({ name }) => [name, new ColumnRef(name)])); +} + +/** + * Compiles a plugin's condition over `table`'s columns (a trust boundary: plugin values): known scalar columns, + * operands of their type made storable (`invalid/args` otherwise, `undefined` included: ask with `isNull`, rule 16). + */ +export function compileWhere(table: TableDefinition, condition: unknown, what: string): WhereNode { + if (!(condition instanceof Condition)) throw invalid("args", `${table.name}: ${what}: expected a condition`); + const { op, args } = condition; + switch (op) { + case "and": + case "or": { + const nodes = args.map((each) => compileWhere(table, each, what)); + return op === "and" ? { and: nodes } : { or: nodes }; + } + case "not": + return { not: compileWhere(table, args[0], what) }; + case "isNull": + return { isNull: scalarColumn(table, args[0], what).name }; + case "prefix": { + const column = scalarColumn(table, args[0], what); + if (column.type !== "str") throw invalid("args", `${table.name}: ${what}: prefix needs a str() column`); + return { prefix: [column.name, operand(column, args[1], what) as string] }; + } + case "inArray": { + const column = scalarColumn(table, args[0], what); + const values = args[1]; + if (!Array.isArray(values)) throw invalid("args", `${table.name}: ${what}: inArray needs an array`); + return { inArray: [column.name, values.map((value) => operand(column, value, what))] }; + } + default: { + const column = scalarColumn(table, args[0], what); + // One of the six comparisons: the node's shape is `{ [op]: [column, value] }`. + return { [op]: [column.name, operand(column, args[1], what)] as const } as WhereNode; + } + } +} + +/** + * Compiles a plugin's `set` (plain values or a callback's result) for `table` (a trust boundary): known non-key + * columns (rule 13), at least one (rule 14), `undefined` clearing an `opt(...)` column as the explicit `null` (a + * `null` value is a `json(...)` cell's JSON `null`, refused for other columns), an + * expression whose result may be `NULL` only for an `opt(...)` column; integer arithmetic only on `int` columns. + */ +export function compileSet(table: TableDefinition, set: unknown, what: string): SetNode { + if (typeof set !== "object" || set === null || Array.isArray(set)) { + throw invalid("args", `${table.name}: ${what}: expected an object of column values`); + } + const entries = Object.entries(set); + if (entries.length === 0) throw invalid("args", `${table.name}: ${what}: the set is empty`); + const node: { [column: string]: ExprNode } = {}; + for (const [name, value] of entries) { + const column = table.byName.get(name); + if (column === undefined) throw invalid("args", `${table.name}: ${what}: no column "${name}"`); + if (table.key.includes(name)) { + throw invalid("args", `${table.name}: ${what}: key column "${name}" cannot change (delete and put)`); + } + const where = `${what}: column "${name}"`; + if (value === undefined) { + if (!column.optional) throw invalid("args", `${table.name}: ${where} is not opt(...) and cannot be cleared`); + node[name] = null; + } else if (value === null && column.type !== "json") { + // A row never holds an explicit `null` (decision "Nullable columns and the set form"); a JSON `null` is a value. + throw invalid("args", `${table.name}: ${where}: null is no ${column.type} value (undefined clears)`); + } else if (value instanceof Expression) { + if (column.type === "json") + throw invalid("args", `${table.name}: ${where}: json(...) columns take plain values`); + const compiled = compileExpr(table, value, column.type, where); + if (compiled.nullable && !column.optional) { + throw invalid("args", `${table.name}: ${where}: the expression may be NULL; the column is not opt(...)`); + } + node[name] = compiled.node; + } else if (column.type === "json") { + node[name] = { json: encodeCell(column, value, `${table.name}: ${where}`).stored }; + } else node[name] = checkScalar(column.type, value, `${table.name}: ${where}`); + } + return node; +} + +/** A compiled expression and whether its result may be `NULL`. */ +type Compiled = { readonly node: ExprNode; readonly nullable: boolean }; + +function compileExpr(table: TableDefinition, value: unknown, type: ScalarType, what: string): Compiled { + if (!(value instanceof Expression)) { + if (value === undefined || value === null) return { node: null, nullable: true }; + return { node: checkScalar(type, value, `${table.name}: ${what}`), nullable: false }; + } + const { op, args } = value; + const refuse = (message: string): never => { + throw invalid("args", `${table.name}: ${what}: ${message}`); + }; + switch (op) { + case "col": { + const column = table.byName.get(args[0] as string) ?? refuse(`no column "${String(args[0])}"`); + if (column.type !== type) refuse(`column "${column.name}" is ${column.type}, not ${type}`); + return { node: { col: column.name }, nullable: column.optional }; + } + case "add": + case "sub": + case "max": + case "min": + case "concat": { + const types: readonly ScalarType[] = + op === "concat" ? ["str"] : op === "add" || op === "sub" ? ["int"] : ["int", "num"]; + if (!types.includes(type)) refuse(`${op} gives ${types.join(" or ")}, the column is ${type}`); + const [a, b] = [compileExpr(table, args[0], type, what), compileExpr(table, args[1], type, what)]; + // The operation's shape is `{ [op]: [a, b] }`. + return { node: { [op]: [a.node, b.node] } as ExprNode, nullable: a.nullable || b.nullable }; + } + case "coalesce": { + const [a, b] = [compileExpr(table, args[0], type, what), compileExpr(table, args[1], type, what)]; + return { node: { coalesce: [a.node, b.node] }, nullable: a.nullable && b.nullable }; + } + case "replacePrefix": { + if (type !== "str") refuse(`replacePrefix gives str, the column is ${type}`); + const subject = compileExpr(table, args[0], type, what); + const [from, to] = [args[1], args[2]].map( + (each) => checkScalar("str", each, `${table.name}: ${what}`) as string, + ); + return { node: { replacePrefix: [subject.node, from!, to!] }, nullable: subject.nullable }; + } + case "when": { + const condition = compileWhere(table, args[0], what); + const [a, b] = [compileExpr(table, args[1], type, what), compileExpr(table, args[2], type, what)]; + return { node: { when: [condition, a.node, b.node] }, nullable: a.nullable || b.nullable }; + } + } +} + +/** The column a condition compares: a reference to a known scalar column. */ +function scalarColumn(table: TableDefinition, ref: unknown, what: string): Column { + if (!(ref instanceof ColumnRef)) throw invalid("args", `${table.name}: ${what}: expected a column reference`); + const column = table.byName.get(ref.column); + if (column === undefined) throw invalid("args", `${table.name}: ${what}: no column "${ref.column}"`); + if (column.type === "json") + throw invalid("args", `${table.name}: ${what}: json(...) column "${ref.column}" is not queryable`); + return column; +} + +/** An operand compared with `column`, made storable; `undefined` and `null` are refused (rule 16). */ +function operand(column: Column, value: unknown, what: string): Scalar { + if (value === undefined || value === null) { + throw invalid("args", `${what}: column "${column.name}" compared with ${String(value)}; ask with isNull`); + } + // `scalarColumn` refused json columns. + return checkScalar(column.type as ScalarType, value, `${what}: column "${column.name}"`); +} diff --git a/packages/chord/src/next/store/import.ts b/packages/chord/src/next/store/import.ts new file mode 100644 index 000000000..c53ca4e77 --- /dev/null +++ b/packages/chord/src/next/store/import.ts @@ -0,0 +1,236 @@ +// `importStore(backend, parts, ctx, { outbox? })` (slice 6a §13, rules 12, 24, 25): a store made from an archive, a +// trust boundary. Every part is checked against the archive schema, its place in the archive's order and its content +// (names, rows against their table's recorded columns, ids and seqs, one live record per address, versions, +// duplicates, the `end` count, the manifest); everything is applied in one backend transaction with the applier, so a +// refused import leaves the backend as it was. An input that stalls is cancelled by `ctx`. + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import type { SqliteDatabase, SqliteExecutor } from "../../sqlite/index.ts"; +import { ChordError } from "../data/index.ts"; +import { invalid } from "../data/internal.ts"; +import { applyWrite } from "./applier.ts"; +import { ARCHIVE_FORMAT, type ArchiveManifest, type ArchivePart, checkPart } from "./archive.ts"; +import type { Backend } from "./backend.ts"; +import { readCatalog, writeCatalog } from "./catalog.ts"; +import { applyStoreSchema, type MetaRow, STORE_SCHEMA, TABLES } from "./schema.ts"; +import { Step } from "./step.ts"; +import { ChordStore, type Store, type StoreOptions } from "./store.ts"; +import { storedText } from "./text.ts"; + +/** + * A store holding the archive `parts` on `backend`, which must not hold a store yet: at the manifest's `lastSeq`, its + * outbox (when `options.outbox`) starting after it. A refused archive, a stalled input cancelled by `context` or a + * backend error rejects; the backend is closed then (rule 12). + */ +export async function importStore( + backend: Backend, + parts: AsyncIterable | Iterable, + context: Context, + options: StoreOptions = {}, +): Promise { + const { db } = backend; + try { + context.abortSignal?.throwIfAborted(); + const existing = await db.get(`SELECT name FROM sqlite_master WHERE name = ?`, TABLES.meta); + if (existing !== undefined) throw invalid("args", "import: the backend holds a store already"); + const meta = await db.transaction(async (executor) => { + const meta = await applyStoreSchema(inTransaction(executor)); + const archive = new ArchiveImport(executor, context); + await each(parts, context, (part) => archive.take(part)); + return archive.finish(meta, options.outbox === true); + }); + return new ChordStore(db, meta, await readCatalog(db)); + } catch (error) { + try { + await db.close(); + } catch { + // The import's own failure is the one to report. + } + throw error; + } +} + +/** The store's schema migrations run inside the import's backend transaction (they open none of their own). */ +function inTransaction(executor: SqliteExecutor): SqliteDatabase { + return { + exec: (sql) => executor.exec(sql), + run: (sql, ...params) => executor.run(sql, ...params), + runCounted: (sql, ...params) => executor.runCounted(sql, ...params), + get: (sql, ...params) => executor.get(sql, ...params), + all: (sql, ...params) => executor.all(sql, ...params), + transaction: (callback) => callback(executor), + close: () => Promise.reject(new Error("import: the backend closes after the import")), + }; +} + +/** A place in the archive's order: before the manifest, in a table group's puts or indexes, the documents, the end. */ +type Place = "manifest" | "puts" | "indexes" | "documents" | "ended"; + +/** + * One archive being imported: the place reached in its order (the manifest, then table groups of a `table` write, its + * `put`s and its `index` writes, then the documents, then `end`), what was seen (tables, ids, live addresses, the + * write count) and the step its writes are applied in. + */ +class ArchiveImport { + readonly #executor: SqliteExecutor; + readonly #context: Context; + #position = 0; + #manifest: ArchiveManifest | undefined; + #step: Step | undefined; + /** Where the next part may go. */ + #place: Place = "manifest"; + /** The table whose group is open. */ + #table: string | undefined; + readonly #tables = new Set(); + readonly #ids = new Set(); + readonly #addresses = new Set(); + #writes = 0; + + constructor(executor: SqliteExecutor, context: Context) { + this.#executor = executor; + this.#context = context; + } + + /** Checks the next part and applies it. */ + async take(value: unknown): Promise { + const position = this.#position++; + const part = checkPart(value, position, this.#context); + const what = `import: part ${position} (${part.kind})`; + if (this.#place === "ended") throw invalid("args", `${what}: after the end`); + if (part.kind === "manifest" || this.#step === undefined) { + this.#start(part, what); + return; + } + if (part.kind === "end") { + if (part.writes !== this.#writes) { + throw invalid("args", `${what}: counts ${part.writes} writes, the archive has ${this.#writes}`); + } + this.#place = "ended"; + return; + } + this.#place = this.#placeOf(part, what); + this.#writes++; + await applyWrite(this.#step, part, what); + } + + /** The stored metadata once the archive ended: its `lastSeq`, next document id and the outbox setting. */ + async finish(meta: MetaRow, outbox: boolean): Promise { + if (this.#place !== "ended" || this.#step === undefined || this.#manifest === undefined) { + throw invalid("args", "import: the archive has no end part"); + } + const { lastSeq, nextDocumentId } = this.#manifest; + await writeCatalog(this.#step); + await this.#executor.run( + `UPDATE ${TABLES.meta} SET last_seq = ?, next_document_id = ?, outbox = ?, outbox_floor = ? WHERE singleton = 1`, + lastSeq, + nextDocumentId, + outbox ? 1 : 0, + lastSeq, + ); + return { + ...meta, + last_seq: lastSeq, + next_document_id: nextDocumentId, + outbox: outbox ? 1 : 0, + outbox_floor: lastSeq, + }; + } + + /** The manifest, first: a format and schema this code reads. */ + #start(part: ArchivePart, what: string): void { + if (part.kind !== "manifest" || this.#step !== undefined) + throw invalid("args", `${what}: the manifest comes first, once`); + if (part.format !== ARCHIVE_FORMAT) + throw invalid("args", `${what}: format ${part.format} is not ${ARCHIVE_FORMAT}`); + if (part.schema > STORE_SCHEMA) + throw invalid("args", `${what}: schema ${part.schema} is newer than ${STORE_SCHEMA}`); + this.#manifest = part; + this.#step = new Step(this.#executor, part.lastSeq, part.nextDocumentId, new Map()); + } + + /** Where `part` puts the archive, refused when out of order, a duplicate, or outside the manifest's ranges. */ + #placeOf(part: Exclude, what: string): Place { + const place = this.#place; + switch (part.kind) { + case "table": + if (place === "documents") throw invalid("args", `${what}: tables come before the documents`); + if (this.#tables.has(part.table)) throw invalid("args", `${what}: table ${part.table} occurs twice`); + this.#tables.add(part.table); + this.#table = part.table; + return "puts"; + case "put": + if (place !== "puts" || part.table !== this.#table) { + throw invalid("args", `${what}: a put follows its table's table write, before its indexes`); + } + return "puts"; + case "index": + if ((place !== "puts" && place !== "indexes") || part.table !== this.#table) { + throw invalid("args", `${what}: an index follows its table's table write and puts`); + } + return "indexes"; + case "document.create": + this.#checkDocument(part, what); + return "documents"; + } + } + + /** + * A record within the manifest's ids and seqs, unique, alone at its address, its base at its version; its address + * and meta well-formed text without NUL (SQLite would cut a NUL off, slice 6a §8). + */ + #checkDocument(part: Extract, what: string): void { + const { record, base } = part; + const { lastSeq, nextDocumentId } = this.#manifest!; + if (record.id >= nextDocumentId) throw invalid("args", `${what}: id ${record.id} is not below ${nextDocumentId}`); + if (this.#ids.has(record.id)) throw invalid("args", `${what}: id ${record.id} occurs twice`); + if (record.createdAt > lastSeq) throw invalid("args", `${what}: created after the manifest's lastSeq`); + if (base.version !== record.version || base.fingerprint !== record.fingerprint) { + throw invalid("args", `${what}: the base is not at the record's version and fingerprint`); + } + const { namespace, owner, kind, meta } = record; + // A singleton's record has no key; its column keeps `""`. + const key = record.key ?? ""; + for (const text of [namespace, owner, kind, key]) { + if (storedText(text, what) !== text) throw invalid("args", `${what}: an address is not well-formed text`); + } + if (meta !== undefined && storedText(meta, what) !== meta) { + throw invalid("args", `${what}: the meta is not well-formed text`); + } + const address = JSON.stringify([namespace, owner, kind, key]); + if (this.#addresses.has(address)) throw invalid("args", `${what}: a second live record at one address`); + this.#ids.add(record.id); + this.#addresses.add(address); + } +} + +/** Runs `take` on each part in turn; a wait for the next part ends when `context` aborts (the input is closed). */ +async function each( + parts: AsyncIterable | Iterable, + context: Context, + take: (part: unknown) => Promise, +): Promise { + const iterator = Symbol.asyncIterator in parts ? parts[Symbol.asyncIterator]() : parts[Symbol.iterator](); + let done = false; + try { + for (;;) { + let result: IteratorResult; + try { + result = await awaitWithContext(Promise.resolve(iterator.next()), context); + } catch (error) { + if (!context.abortSignal?.aborted) throw error; + throw new ChordError( + { code: "aborted", message: "import: aborted" }, + { cause: context.abortSignal.reason }, + ); + } + if (result.done) { + done = true; + return; + } + await take(result.value); + } + } finally { + // A stalled input is not awaited: the backend transaction ends now. + if (!done) void Promise.resolve(iterator.return?.()).catch(() => {}); + } +} diff --git a/packages/chord/src/next/store/index.ts b/packages/chord/src/next/store/index.ts new file mode 100644 index 000000000..57ee09a63 --- /dev/null +++ b/packages/chord/src/next/store/index.ts @@ -0,0 +1,49 @@ +// The store for apps: opening it over a backend, tables with their queries, writes and migrations, export and +// import of archives (with their JSONL encoding), the outbox, text order. + +export type { ArchiveManifest, ArchivePart } from "./archive.ts"; +export { type Backend, sqliteBackend } from "./backend.ts"; +export type { Columns, PutRowOf, RowOf } from "./columns.ts"; +export { + add, + and, + type ColumnRef, + type Condition, + coalesce, + concat, + type Expression, + eq, + gt, + gte, + inArray, + isNull, + lt, + lte, + max, + min, + ne, + not, + or, + prefix, + replacePrefix, + type SetOf, + sub, + when, +} from "./expression.ts"; +export { importStore } from "./import.ts"; +export { type JsonColumnOptions, type JsonColumnSchema, json } from "./json-column.ts"; +export { decodeJsonl, encodeJsonl } from "./jsonl.ts"; +export type { Outbox, OutboxCommit } from "./outbox.ts"; +export { type ColumnRefs, type Query, query } from "./query.ts"; +export { openStore, type Store, type StoreOptions } from "./store.ts"; +export { defineTable, type KeyOf, type TableDefinition, type TableOptions } from "./table.ts"; +export { + type AddedColumn, + evolveTable, + type PatchOf, + type TableEditor, + type TableTailEditor, +} from "./table-evolve.ts"; +export type { Page, PageOptions, Table, TableOf } from "./table-handle.ts"; +export { compareText } from "./text.ts"; +export type { StoreWrite } from "./writes.ts"; diff --git a/packages/chord/src/next/store/json-column.ts b/packages/chord/src/next/store/json-column.ts new file mode 100644 index 000000000..8e8a0a1c7 --- /dev/null +++ b/packages/chord/src/next/store/json-column.ts @@ -0,0 +1,78 @@ +// `json(schema, { version, migrate? })` table columns (decision "The `json(...)` column helper", W11): one JSON `TEXT` +// cell with its own version, typed when read, migrated on access (the result checked once, R8), never walked on write +// (R9). + +import Type, { type TUnsafe } from "typebox"; +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { Static, TSchema } from "../contracts/index.ts"; +import { fingerprintOf } from "../contracts/internal.ts"; +import { snapshotSchema } from "../contracts/snapshot.ts"; +import { ChordError } from "../data/index.ts"; +import { validatorFor } from "../data/internal.ts"; + +/** What `json(...)` takes besides the schema. */ +export type JsonColumnOptions = { + /** The version of the cells' shape; a cell stores the version it was written at. */ + readonly version: number; + /** Maps a cell of an older version `from` to this version's shape; pure. Without it older cells are refused. */ + readonly migrate?: (old: JsonValue, from: number) => T; +}; + +/** A `json(...)` column as `columns` takes it: a schema of the cells' type carrying the column. */ +export type JsonColumnSchema = TUnsafe & { readonly json: JsonColumn }; + +/** A column holding one JSON value of `schema` per row, at `version`; wrap it in `opt(...)` to make it nullable. */ +export function json(schema: S, options: JsonColumnOptions>): JsonColumnSchema> { + const column = new JsonColumn(schema, options.version, options.migrate); + // A TypeBox schema (so `opt(...)` and `Static` take it) whose enumerable `json` field `opt`'s clone keeps. + return Type.Unsafe>({ json: column }) as JsonColumnSchema>; +} + +/** A `json(...)` column's definition: its schema snapshot, version and fingerprint, and the read of a stored cell. */ +export class JsonColumn { + readonly schema: TSchema; + readonly version: number; + /** The fingerprint of the schema's normal form, recorded with the table: the same version with another is `broken`. */ + readonly fingerprint: string; + readonly #migrate: ((old: JsonValue, from: number) => unknown) | undefined; + + constructor(schema: TSchema, version: number, migrate: ((old: JsonValue, from: number) => unknown) | undefined) { + // Declaration policy: JavaScript callers reach this check. + if (!Number.isSafeInteger(version) || version < 1) + throw new Error(`json(...): version ${version} is not a positive integer`); + const snapshot = snapshotSchema(schema, "json(...)"); + this.schema = snapshot.schema; + this.version = version; + this.fingerprint = fingerprintOf(snapshot.normal); + this.#migrate = migrate; + } + + /** + * The value of a stored cell written at `version`: parsed; an older one migrated and checked once (R8, counted). A + * newer version, a missing or throwing `migrate` and an invalid result throw `unavailable` (`what` names the cell); + * the caller fails the owner (slice 6a §1.4), the data stays. + */ + read(text: string, version: number, what: string, context: Context): unknown { + const value = JSON.parse(text) as JsonValue; + if (version === this.version) return value; + if (version > this.version) throw refused(`${what}: stored at version ${version}, newer than ${this.version}`); + if (this.#migrate === undefined) throw refused(`${what}: no migrate from version ${version} to ${this.version}`); + let migrated: unknown; + try { + migrated = this.#migrate(value, version); + } catch (error) { + throw refused(`${what}: migrate from version ${version} threw`, error); + } + context.telemetry.metrics?.counter("chord.data.validations").add(1); + if (!validatorFor(this, context).Check(migrated)) { + throw refused(`${what}: migrate from version ${version} does not match the schema`); + } + return migrated; + } +} + +/** The error of a cell the code cannot read: the owner fails `broken` with it. */ +function refused(message: string, cause?: unknown): ChordError { + return new ChordError({ code: "unavailable", message }, cause === undefined ? undefined : { cause }); +} diff --git a/packages/chord/src/next/store/jsonl.ts b/packages/chord/src/next/store/jsonl.ts new file mode 100644 index 000000000..302f43dce --- /dev/null +++ b/packages/chord/src/next/store/jsonl.ts @@ -0,0 +1,60 @@ +// Plain JSONL for archives (slice 6a §13): one part per line, UTF-8. The store yields and takes parts; this is the +// encoding it ships (others, compressed ones, come later without touching the applier). + +import { invalid } from "../data/internal.ts"; +import type { ArchivePart } from "./archive.ts"; + +/** The bytes a chunk of encoded lines collects before it is yielded. */ +const CHUNK = 64 * 1024; + +/** `parts` as JSONL bytes, in chunks of whole lines. */ +export async function* encodeJsonl(parts: AsyncIterable): AsyncIterable { + const encoder = new TextEncoder(); + let lines = ""; + for await (const part of parts) { + lines += `${JSON.stringify(part)}\n`; + if (lines.length >= CHUNK) { + yield encoder.encode(lines); + lines = ""; + } + } + if (lines.length > 0) yield encoder.encode(lines); +} + +/** + * The values of JSONL `bytes`, one per nonempty line, for `importStore` to check (a trust boundary: bytes that are + * no UTF-8 and a line that is no JSON are `invalid/args`). + */ +export async function* decodeJsonl(bytes: AsyncIterable): AsyncIterable { + const decoder = new TextDecoder("utf-8", { fatal: true }); + let rest = ""; + let line = 0; + const parse = (text: string): unknown => { + try { + return JSON.parse(text); + } catch { + throw invalid("args", `jsonl: line ${line} is not JSON`); + } + }; + const decode = (chunk?: Uint8Array): string => { + try { + return chunk === undefined ? decoder.decode() : decoder.decode(chunk, { stream: true }); + } catch { + throw invalid("args", `jsonl: line ${line + 1} is not UTF-8`); + } + }; + for await (const chunk of bytes) { + rest += decode(chunk); + let start = 0; + for (let end = rest.indexOf("\n"); end >= 0; end = rest.indexOf("\n", start)) { + const text = rest.slice(start, end); + start = end + 1; + line++; + if (text.trim().length > 0) yield parse(text); + } + rest = rest.slice(start); + } + rest += decode(); + line++; + if (rest.trim().length > 0) yield parse(rest); +} diff --git a/packages/chord/src/next/store/outbox.ts b/packages/chord/src/next/store/outbox.ts new file mode 100644 index 000000000..642feba73 --- /dev/null +++ b/packages/chord/src/next/store/outbox.ts @@ -0,0 +1,100 @@ +// The outbox (slice 6a §2; decisions "The outbox is optional, internal and versioned", small items 3-4): the store's +// commits as the sync layer reads them, one `{ seq, writes }` per commit from its floor on, in the current format. + +import type { Context } from "../../context/index.ts"; +import { ChordError } from "../data/index.ts"; +import { invalid } from "../data/internal.ts"; +import { TABLES } from "./schema.ts"; +import type { ChordStore } from "./store.ts"; +import type { StoreWrite } from "./writes.ts"; + +/** The write vocabulary's version an outbox commit is stored in; a newer one is refused, data intact. */ +export const OUTBOX_FORMAT = 1; + +/** One commit as the outbox gives it. */ +export type OutboxCommit = { readonly seq: number; readonly writes: readonly StoreWrite[] }; + +const NOTHING = (): void => {}; + +/** A store's outbox: its commits after the floor (the seq it was turned on at, advanced by `trim`). */ +export class Outbox { + readonly #store: ChordStore; + #floor: number; + /** + * The outbox's reads and trims, one at a time: a read checks the floor and reads with no trim running (a trim's + * commit may have deleted rows before its floor is set), a trim waits for the reads in flight. + */ + #queue: Promise = Promise.resolve(); + + constructor(store: ChordStore, floor: number) { + this.#store = store; + this.#floor = floor; + } + + /** The seq the outbox starts after: `read` takes any `afterSeq` from it to `lastSeq`. */ + get floor(): number { + return this.#floor; + } + + /** Up to `limit` commits after `afterSeq`, in seq order; `afterSeq` outside floor..lastSeq is `invalid/args`. */ + async read(afterSeq: number, limit: number, context: Context): Promise { + if (!Number.isSafeInteger(afterSeq)) throw invalid("args", `outbox: read after ${afterSeq}, not an integer`); + if (!Number.isSafeInteger(limit) || limit < 1) throw invalid("args", "outbox: limit is a positive integer"); + return this.#queued(() => this.#read(afterSeq, limit, context)); + } + + /** + * Drops the commits through `throughSeq` and advances the floor; idempotent; beyond `lastSeq` is `invalid/args`. + * Trims run one at a time, so the stored floor only rises and equals the one in memory. + */ + trim(throughSeq: number, context: Context): Promise { + try { + context.abortSignal?.throwIfAborted(); + if (!Number.isSafeInteger(throughSeq) || throughSeq > this.#store.lastSeq) { + throw invalid("args", `outbox: trim through ${throughSeq}, beyond ${this.#store.lastSeq}`); + } + } catch (error) { + return Promise.reject(error); + } + return this.#queued(() => this.#trim(throughSeq)); + } + + /** Runs `operation` once the outbox operations before it settled. */ + #queued(operation: () => Promise): Promise { + const run = this.#queue.then(operation); + this.#queue = run.then(NOTHING, NOTHING); + return run; + } + + /** `read`, its turn come: the floor as no trim moves it. */ + async #read(afterSeq: number, limit: number, context: Context): Promise { + if (afterSeq < this.#floor || afterSeq > this.#store.lastSeq) { + throw invalid("args", `outbox: read after ${afterSeq}, outside ${this.#floor}..${this.#store.lastSeq}`); + } + const rows = await this.#store.read( + (executor) => + executor.all<{ readonly seq: number; readonly format: number; readonly writes: string }>( + `SELECT seq, format, writes FROM ${TABLES.outbox} WHERE seq > ? ORDER BY seq LIMIT ?`, + afterSeq, + limit, + ), + context, + ); + return rows.map(({ seq, format, writes }) => { + if (format !== OUTBOX_FORMAT) { + const message = `outbox: commit ${seq} is in format ${format}, newer than ${OUTBOX_FORMAT}`; + throw new ChordError({ code: "unavailable", message }); + } + return { seq, writes: JSON.parse(writes) as StoreWrite[] }; + }); + } + + async #trim(throughSeq: number): Promise { + if (throughSeq <= this.#floor) return; + await this.#store.maintain(async (executor) => { + await executor.run(`DELETE FROM ${TABLES.outbox} WHERE seq <= ?`, throughSeq); + await executor.run(`UPDATE ${TABLES.meta} SET outbox_floor = ? WHERE singleton = 1`, throughSeq); + }); + this.#floor = throughSeq; + } +} diff --git a/packages/chord/src/next/store/query.ts b/packages/chord/src/next/store/query.ts new file mode 100644 index 000000000..e4e9c7f81 --- /dev/null +++ b/packages/chord/src/next/store/query.ts @@ -0,0 +1,118 @@ +// Queries (decision "Query runner"; slice 6a §6, rule 17): `query(Table).where(…).orderBy(…)` as immutable data +// (the condition compiled when given, the order with the key appended as the tie-break), and the opaque cursors that +// continue a page, bound to the table, the order columns and the direction. + +import type { SqliteValue } from "../../sqlite/index.ts"; +import { invalid } from "../data/internal.ts"; +import type { Columns, ColumnTypeOf, ValueOf } from "./columns.ts"; +import { type ColumnRef, type Condition, compileWhere, type WhereNode } from "./expression.ts"; +import { checkScalar } from "./rows.ts"; +import type { Order } from "./sql.ts"; +import type { TableDefinition } from "./table.ts"; + +/** The column references a table's callbacks get: `e.path`, `e.at`. */ +export type ColumnRefs = { + readonly [K in keyof C & string]: ColumnRef, ColumnTypeOf>; +}; + +/** A query of `table`: its rows, in key order. */ +export function query(table: TableDefinition): Query { + return new Query(table, undefined, [], undefined); +} + +/** + * An immutable query of one table: an optional condition (several `where`s are `and`ed) and an order (`orderBy` + * columns, one direction, the key appended unless already there). + */ +export class Query { + readonly table: TableDefinition; + readonly condition: WhereNode | undefined; + readonly #orderBy: readonly string[]; + readonly #direction: "asc" | "desc" | undefined; + + constructor( + table: TableDefinition, + condition: WhereNode | undefined, + orderBy: readonly string[], + direction: "asc" | "desc" | undefined, + ) { + this.table = table; + this.condition = condition; + this.#orderBy = orderBy; + this.#direction = direction; + } + + /** This query with `fn`'s condition too. */ + where(fn: (row: ColumnRefs) => Condition): Query { + // The refs are the table's columns, typed by `C`. + const compiled = compileWhere(this.table, fn(this.table.refs as ColumnRefs), "where"); + const condition = this.condition === undefined ? compiled : { and: [this.condition, compiled] }; + return new Query(this.table, condition, this.#orderBy, this.#direction); + } + + /** + * This query ordered by `fn`'s column next (a non-`opt` `str` or `int` one), in `direction`: one direction per + * query, no column twice (`invalid/args`). + */ + orderBy( + fn: (row: ColumnRefs) => ColumnRef, + direction: "asc" | "desc" = "asc", + ): Query { + const table = this.table; + // The refs are `C`'s columns; what the plugin's callback returns is checked below. + const ref = fn(table.refs as ColumnRefs) as ColumnRef | undefined; + const column = table.byName.get(ref?.column ?? ""); + if (column === undefined || column.optional || (column.type !== "str" && column.type !== "int")) { + throw invalid("args", `${table.name}: orderBy: expected a non-opt str() or int() column`); + } + if (this.#orderBy.includes(column.name)) throw invalid("args", `${table.name}: orderBy: "${column.name}" twice`); + if (this.#direction !== undefined && this.#direction !== direction) { + throw invalid("args", `${table.name}: orderBy: one direction per query`); + } + return new Query(table, this.condition, [...this.#orderBy, column.name], direction); + } + + /** The order of its rows: the `orderBy` columns, then the key columns not among them. */ + get order(): Order { + const columns = [...this.#orderBy, ...this.table.key.filter((column) => !this.#orderBy.includes(column))]; + return { columns, direction: this.#direction ?? "asc" }; + } +} + +/** The cursor of the position after a row with the order values `values` in the table named `table` (its SQL name). */ +export function encodeCursor(table: string, order: Order, values: readonly SqliteValue[]): string { + const json = JSON.stringify({ t: table, o: order.columns, d: order.direction, v: values }); + let binary = ""; + for (const byte of new TextEncoder().encode(json)) binary += String.fromCharCode(byte); + return btoa(binary).replaceAll("+", "-").replaceAll("/", "_").replace(/=+$/, ""); +} + +/** + * The order values of `cursor` (a trust boundary: an app hands it back): one made for `table`'s `order` (any + * condition) in the table named `sqlName` (another object's table of the same definition is another table), each value + * of its column's type; `invalid/args` otherwise. + */ +export function decodeCursor(table: TableDefinition, sqlName: string, order: Order, cursor: unknown): SqliteValue[] { + const refuse = (): never => { + throw invalid("args", `${table.name}: the cursor is not one of this table's order`); + }; + if (typeof cursor !== "string") return refuse(); + let decoded: unknown; + try { + const binary = atob(cursor.replaceAll("-", "+").replaceAll("_", "/")); + decoded = JSON.parse( + new TextDecoder("utf-8", { fatal: true }).decode(Uint8Array.from(binary, (c) => c.charCodeAt(0))), + ); + } catch { + return refuse(); + } + // Wire input: every field is checked below. + const { t, o, d, v } = (decoded ?? {}) as { t?: unknown; o?: unknown; d?: unknown; v?: unknown }; + if (t !== sqlName || d !== order.direction || JSON.stringify(o) !== JSON.stringify(order.columns)) refuse(); + if (!Array.isArray(v) || v.length !== order.columns.length) return refuse(); + return order.columns.map((name, index) => { + const column = table.byName.get(name)!; + // Order columns are non-opt `str` or `int` (checked when the query was built). + return checkScalar(column.type as "str" | "int", v[index], `${table.name}: cursor`) as string | number; + }); +} diff --git a/packages/chord/src/next/store/rows.ts b/packages/chord/src/next/store/rows.ts new file mode 100644 index 000000000..da1152f0d --- /dev/null +++ b/packages/chord/src/next/store/rows.ts @@ -0,0 +1,213 @@ +// Rows (slice 6a §5 "Values at write", §9): one compiled codec per table version between a plugin's rows and keys +// (a trust boundary, checked closed) and what the store binds and reads; `json(...)` cells serialized at staging +// (rule 8), never walked (R9), and migrated on access when read. + +import type { Context } from "../../context/index.ts"; +import type { JsonValue } from "../../json.ts"; +import type { SqliteValue } from "../../sqlite/index.ts"; +import { invalid } from "../data/internal.ts"; +import type { Column, ColumnType } from "./columns.ts"; +import { storedText } from "./text.ts"; + +/** A scalar cell as the store keeps it in rows, writes and operands: a well-formed string, a number or a boolean. */ +export type Scalar = string | number | boolean; + +/** A row as the outbox records it: the stored values, `opt(...)` columns absent when `NULL`. */ +export type StoredRow = { readonly [column: string]: Scalar | JsonValue }; + +/** A `put` ready for the step: its parameters in `physical` order and its record. */ +export type EncodedRow = { + readonly params: readonly SqliteValue[]; + readonly row: StoredRow; + /** Each written `json(...)` column's version. */ + readonly columns: { readonly [column: string]: number }; + /** The row's key identity, as `EncodedKey.id`. */ + readonly id: string; +}; + +/** A key ready for the store: its parameters in key order, its record and its identity within one table. */ +export type EncodedKey = { + readonly params: readonly SqliteValue[]; + readonly key: { readonly [column: string]: Scalar }; + /** `JSON.stringify` of the parameters: equal keys, equal ids. */ + readonly id: string; +}; + +/** The physical name of a `json(...)` column's version column, and of the row's eager-step version. */ +export function versionColumn(column: string): string { + return `${column}$v`; +} + +/** The row's eager-step version column (W11): set by `put` and eager migration steps only. */ +export const ROW_VERSION = "$v"; + +/** The physical column order of a stored row: the declared columns, each `json(...)` column's version, then `$v`. */ +export function physicalColumns(columns: readonly { readonly name: string; readonly type: ColumnType }[]): string[] { + return [ + ...columns.map(({ name }) => name), + ...columns.filter(({ type }) => type === "json").map(({ name }) => versionColumn(name)), + ROW_VERSION, + ]; +} + +/** A scalar as SQLite binds it: booleans as 0 and 1. */ +export function toSqlite(scalar: Scalar): string | number { + return scalar === true ? 1 : scalar === false ? 0 : scalar; +} + +/** + * A table version's codec: `put` rows and keys checked and encoded, stored rows decoded. `physical` is the column + * order of an encoded row (`physicalColumns`). + */ +export class RowCodec { + readonly physical: readonly string[]; + readonly #columns: readonly Column[]; + readonly #key: readonly Column[]; + readonly #json: readonly Extract[]; + readonly #names: ReadonlySet; + readonly #keyNames: ReadonlySet; + /** The positions of the key columns in `physical`. */ + readonly #keyPositions: readonly number[]; + readonly #table: string; + + constructor(columns: readonly Column[], key: readonly string[], table: string) { + this.#columns = columns; + this.#key = key.map((name) => columns.find((column) => column.name === name)!); + this.#json = columns.filter( + (column): column is Extract => column.type === "json", + ); + this.#names = new Set(columns.map(({ name }) => name)); + this.#keyNames = new Set(key); + this.#keyPositions = key.map((name) => columns.findIndex((column) => column.name === name)); + this.#table = table; + this.physical = physicalColumns(columns); + } + + /** + * Checks and encodes a `put` row of table version `version` (a trust boundary): a plain object with every + * non-`opt` column (a defaulted one filled in) and no other field; `invalid/args` otherwise. + */ + put(row: unknown, version: number): EncodedRow { + const what = `${this.#table}: put`; + const fields = this.#closed(row, this.#names, what); + const params: SqliteValue[] = []; + const stored: { [column: string]: Scalar | JsonValue } = {}; + const versions: { [column: string]: number } = {}; + for (const column of this.#columns) { + let value = fields[column.name]; + if (value === undefined && column.type !== "json") value = column.default; + if (value === undefined) { + if (!column.optional) throw invalid("args", `${what}: column "${column.name}" is missing`); + params.push(null); + continue; + } + const cell = encodeCell(column, value, `${what}: column "${column.name}"`); + params.push(cell.param); + stored[column.name] = cell.stored; + } + for (const column of this.#json) { + const present = Object.hasOwn(stored, column.name); + if (present) versions[column.name] = column.json.version; + params.push(present ? column.json.version : null); + } + params.push(version); + const id = JSON.stringify(this.#keyPositions.map((position) => params[position])); + return { params, row: stored, columns: versions, id }; + } + + /** Checks and encodes a key (a trust boundary): an object with exactly the key columns; `invalid/args` otherwise. */ + key(key: unknown, what: string): EncodedKey { + const fields = this.#closed(key, this.#keyNames, `${this.#table}: ${what}: the key`); + const params: SqliteValue[] = []; + const stored: { [column: string]: Scalar } = {}; + for (const column of this.#key) { + const value = fields[column.name]; + const where = `${this.#table}: ${what}: key column "${column.name}"`; + if (value === undefined) throw invalid("args", `${where} is missing`); + const cell = encodeScalar(column, value, where); + params.push(cell); + stored[column.name] = cell; + } + return { params, key: stored, id: JSON.stringify(params) }; + } + + /** + * A stored row as reads return it: booleans from 0 and 1, `opt(...)` columns absent when `NULL`, `json(...)` cells + * parsed and migrated on access (a cell the code cannot read throws, see `JsonColumn.read`). + */ + decode(stored: { readonly [column: string]: SqliteValue }, context: Context): { [column: string]: unknown } { + const row: { [column: string]: unknown } = {}; + for (const column of this.#columns) { + const value = stored[column.name]; + if (value === null || value === undefined) continue; + if (column.type === "bool") row[column.name] = value === 1; + else if (column.type === "json") { + const version = stored[versionColumn(column.name)] as number; + row[column.name] = column.json.read(value as string, version, `${this.#table}.${column.name}`, context); + } else row[column.name] = value; + } + return row; + } + + /** `value`'s own fields when it is a plain object with only fields of `names`; `invalid/args` otherwise. */ + #closed(value: unknown, names: ReadonlySet, what: string): { readonly [field: string]: unknown } { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + throw invalid("args", `${what}: expected an object`); + } + for (const field of Object.keys(value)) { + if (!names.has(field)) throw invalid("args", `${what}: unknown column "${field}"`); + } + // Checked above: a plain object with known fields. + return value as { readonly [field: string]: unknown }; + } +} + +/** A checked cell: what is bound and what is recorded. */ +type Cell = { readonly param: SqliteValue; readonly stored: Scalar | JsonValue }; + +/** Checks and encodes a value of `column`; a `json(...)` value is serialized, not walked (R9). */ +export function encodeCell(column: Column, value: unknown, what: string): Cell { + if (column.type !== "json") { + const stored = checkScalar(column.type, value, what); + return { param: toSqlite(stored), stored }; + } + let text: string | undefined; + try { + text = JSON.stringify(value); + } catch (error) { + throw invalid("args", `${what}: not JSON (${error instanceof Error ? error.message : String(error)})`); + } + if (text === undefined) throw invalid("args", `${what}: not JSON`); + // `JSON.stringify` accepted it; the recorded value is the one serialized. It is kept, not copied or parsed back: + // same-host values are immutable by convention, so the outbox serializing it again in the step records this text + // (a plugin mutating a value after `tx.put` breaks that convention, as for any other value it hands over). + return { param: text, stored: value as JsonValue }; +} + +/** Checks a scalar value of `column` and returns what is bound (booleans as 0 and 1). */ +function encodeScalar(column: Column, value: unknown, what: string): string | number { + if (column.type === "json") throw invalid("args", `${what}: a json(...) column has no scalar value`); + return toSqlite(checkScalar(column.type, value, what)); +} + +/** + * A value of a scalar column type as the store keeps it (a trust boundary): `str` well-formed without NUL, `int` a + * safe integer, `num` finite, `bool` a boolean; `invalid/args` otherwise. + */ +export function checkScalar(type: "str" | "int" | "num" | "bool", value: unknown, what: string): Scalar { + switch (type) { + case "str": + if (typeof value === "string") return storedText(value, what); + break; + case "int": + if (Number.isSafeInteger(value)) return value as number; + break; + case "num": + if (typeof value === "number" && Number.isFinite(value)) return value; + break; + case "bool": + if (typeof value === "boolean") return value; + break; + } + throw invalid("args", `${what}: ${JSON.stringify(value) ?? String(value)} is not a ${type} value`); +} diff --git a/packages/chord/src/next/store/schema.ts b/packages/chord/src/next/store/schema.ts new file mode 100644 index 000000000..c528360ce --- /dev/null +++ b/packages/chord/src/next/store/schema.ts @@ -0,0 +1,89 @@ +// The store's own SQL schema (slice 6a §2): its metadata, the recorded table definitions, the documents engine's +// records and revisions, and the outbox; versioned by `applySqliteMigrations`. Declared tables are named +// `"::

"`, so no name here can meet one of them. + +import { applySqliteMigrations, type SqliteDatabase, type SqliteMigration } from "../../sqlite/index.ts"; + +/** The store's tables. */ +export const TABLES = { + meta: "chord_meta", + tables: "chord_tables", + documents: "chord_documents", + revisions: "chord_revisions", + outbox: "chord_outbox", +} as const; + +/** + * The schema history; never change an entry, append one. `chord_meta` is one row: `last_seq`, the next document id + * (S10), and the outbox setting with its floor (the seq it starts after, advanced by `trim`). A document record keeps + * its current base's version and fingerprint and a `revision` counter for the conditional `UPDATE`; the partial + * unique index refuses two live records at one address. + */ +const MIGRATIONS: readonly SqliteMigration[] = [ + { + version: 1, + statements: [ + `CREATE TABLE ${TABLES.meta} ( + singleton INTEGER PRIMARY KEY CHECK (singleton = 1), + last_seq INTEGER NOT NULL, + next_document_id INTEGER NOT NULL, + outbox INTEGER NOT NULL CHECK (outbox IN (0, 1)), + outbox_floor INTEGER NOT NULL + ) STRICT`, + `INSERT INTO ${TABLES.meta} (singleton, last_seq, next_document_id, outbox, outbox_floor) VALUES (1, 0, 1, 0, 0)`, + `CREATE TABLE ${TABLES.tables} (name TEXT PRIMARY KEY, definition TEXT NOT NULL) STRICT, WITHOUT ROWID`, + `CREATE TABLE ${TABLES.documents} ( + id INTEGER PRIMARY KEY, + namespace TEXT NOT NULL, + owner TEXT NOT NULL, + kind TEXT NOT NULL, + key TEXT NOT NULL, + history TEXT NOT NULL CHECK (history IN ('latest', 'rewindable')), + created_at INTEGER NOT NULL, + retired_at INTEGER, + meta TEXT, + revision INTEGER NOT NULL, + version INTEGER NOT NULL, + fingerprint TEXT NOT NULL + ) STRICT`, + `CREATE UNIQUE INDEX chord_documents_live ON ${TABLES.documents} (namespace, owner, kind, key) + WHERE retired_at IS NULL`, + `CREATE TABLE ${TABLES.revisions} ( + document_id INTEGER NOT NULL, + seq INTEGER NOT NULL, + kind TEXT NOT NULL CHECK (kind IN ('base', 'delta')), + version INTEGER NOT NULL, + content TEXT NOT NULL, + PRIMARY KEY (document_id, seq) + ) STRICT, WITHOUT ROWID`, + `CREATE TABLE ${TABLES.outbox} (seq INTEGER PRIMARY KEY, format INTEGER NOT NULL, writes TEXT NOT NULL) STRICT`, + ], + }, +]; + +/** The store's schema version: the newest entry of its history (archives carry it). */ +export const STORE_SCHEMA = MIGRATIONS.at(-1)!.version; + +/** The store's metadata row, as `openStore` reads it once. */ +export type MetaRow = { + readonly last_seq: number; + readonly next_document_id: number; + readonly outbox: number; + readonly outbox_floor: number; +}; + +/** + * Brings `db` to the store's newest schema (one transaction) and checks that it stores text as UTF-8 (the store + * compares text by its bytes); returns the metadata row. + */ +export async function applyStoreSchema(db: SqliteDatabase): Promise { + const encoding = await db.get<{ readonly encoding: string }>("PRAGMA encoding"); + if (encoding?.encoding !== "UTF-8") + throw new Error(`store: the database encoding is ${encoding?.encoding}, not UTF-8`); + await applySqliteMigrations(db, MIGRATIONS, { table: "chord_store_schema" }); + const meta = await db.get( + `SELECT last_seq, next_document_id, outbox, outbox_floor FROM ${TABLES.meta} WHERE singleton = 1`, + ); + if (meta === undefined) throw new Error("store: the metadata row is missing"); + return meta; +} diff --git a/packages/chord/src/next/store/sql.ts b/packages/chord/src/next/store/sql.ts new file mode 100644 index 000000000..95529eb55 --- /dev/null +++ b/packages/chord/src/next/store/sql.ts @@ -0,0 +1,266 @@ +// SQL of declared tables (slice 6a §5, §6, §7): the DDL a table definition renders, and the statements of its reads +// and writes, generated from the compiled vocabulary with every value bound and every identifier quoted. Names were +// checked at declaration (and at import), so no `"` occurs in them and quoting is always safe. + +import type { SqliteValue } from "../../sqlite/index.ts"; +import type { Column, ColumnRecord } from "./columns.ts"; +import type { ExprNode, SetNode, WhereNode } from "./expression.ts"; +import { physicalColumns, ROW_VERSION, toSqlite, versionColumn } from "./rows.ts"; +import { prefixUpperBound } from "./text.ts"; + +/** A statement and its parameters. */ +export type Statement = { readonly sql: string; readonly params: SqliteValue[] }; + +/** An order of rows: the order columns (the key appended as the tie-break) and one direction. */ +export type Order = { readonly columns: readonly string[]; readonly direction: "asc" | "desc" }; + +/** A quoted SQL identifier. */ +export function quote(name: string): string { + return `"${name}"`; +} + +/** The SQL name of an object's table: `::
`, injective since no part contains `:`. */ +export function tableSqlName(hostKind: string, objectType: string, table: string): string { + return `${hostKind}:${objectType}:${table}`; +} + +/** + * `CREATE TABLE` for `columns` (as recorded) and `key`: `STRICT`, `WITHOUT ROWID`, the key as primary key, an `int` + * column checked within ±(2^53 - 1) (rule 10), a `bool` one within 0 and 1, a `json(...)` one as `TEXT` with its + * version column, and the row's `$v`. + */ +export function createTableSql( + table: string, + columns: { readonly [name: string]: ColumnRecord }, + key: readonly string[], +): string { + const definitions = Object.entries(columns).flatMap(([name, column]) => columnSql(name, column)); + definitions.push(`${quote(ROW_VERSION)} INTEGER NOT NULL`); + definitions.push(`PRIMARY KEY (${key.map(quote).join(", ")})`); + return `CREATE TABLE ${quote(table)} (${definitions.join(", ")}) STRICT, WITHOUT ROWID`; +} + +const SAFE = Number.MAX_SAFE_INTEGER; + +/** The definitions of one column: itself, and a `json(...)` column's version column. */ +export function columnSql(name: string, column: ColumnRecord): string[] { + const nullability = column.optional ? "" : " NOT NULL"; + const defaulted = column.default === undefined ? "" : ` DEFAULT ${literal(column.default)}`; + const id = quote(name); + switch (column.type) { + case "str": + return [`${id} TEXT${nullability}${defaulted}`]; + case "int": + return [`${id} INTEGER${nullability}${defaulted} CHECK (${id} BETWEEN ${-SAFE} AND ${SAFE})`]; + case "num": + return [`${id} REAL${nullability}${defaulted}`]; + case "bool": + return [`${id} INTEGER${nullability}${defaulted} CHECK (${id} IN (0, 1))`]; + case "json": + return [`${id} TEXT${nullability}`, `${quote(versionColumn(name))} INTEGER${nullability}`]; + } +} + +/** A default as a SQL literal: declaration data (checked), which DDL cannot bind. */ +function literal(value: string | number | boolean): string { + if (typeof value === "string") return `'${value.replaceAll("'", "''")}'`; + return typeof value === "boolean" ? (value ? "1" : "0") : String(value); +} + +/** + * The physical columns of a recorded table in the order `RowCodec.physical` uses (`physicalColumns`), and its + * whole-row `INSERT OR REPLACE` over them. + */ +export function recordedPut( + table: string, + columns: { readonly [name: string]: ColumnRecord }, +): { readonly physical: readonly string[]; readonly sql: string } { + const physical = physicalColumns(Object.entries(columns).map(([name, { type }]) => ({ name, type }))); + return { physical, sql: putSql(table, physical) }; +} + +/** `INSERT OR REPLACE` of a whole row over `physical` (`put` is an upsert). */ +function putSql(table: string, physical: readonly string[]): string { + return `INSERT OR REPLACE INTO ${quote(table)} (${physical.map(quote).join(", ")}) VALUES (${physical.map(() => "?").join(", ")})`; +} + +/** `CREATE INDEX` named `
:` over `columns`, the key appended (every index ends with it). */ +export function createIndexSql( + table: string, + index: string, + columns: readonly string[], + key: readonly string[], +): string { + const all = [...columns, ...key.filter((column) => !columns.includes(column))]; + return `CREATE INDEX ${quote(`${table}:${index}`)} ON ${quote(table)} (${all.map(quote).join(", ")})`; +} + +/** `DROP INDEX` of `
:`. */ +export function dropIndexSql(table: string, index: string): string { + return `DROP INDEX ${quote(`${table}:${index}`)}`; +} + +/** + * The statements of one table at one version, rendered once per table handle: the fixed ones as texts, the ones + * over compiled conditions and `set`s on demand. + */ +export class TableSql { + readonly table: string; + /** The columns a read selects: the declared ones and each `json(...)` column's version. */ + readonly #select: string; + readonly #byKey: string; + /** `INSERT OR REPLACE` of a whole row, in `RowCodec.physical`'s order (`put` is an upsert). */ + readonly put: string; + readonly get: string; + readonly delete: string; + + constructor(table: string, columns: readonly Column[], physical: readonly string[], key: readonly string[]) { + this.table = table; + const id = quote(table); + this.#select = [ + ...columns.map(({ name }) => quote(name)), + ...columns.filter(({ type }) => type === "json").map(({ name }) => quote(versionColumn(name))), + ].join(", "); + this.#byKey = key.map((column) => `${quote(column)} = ?`).join(" AND "); + this.put = putSql(table, physical); + this.get = `SELECT ${this.#select} FROM ${id} WHERE ${this.#byKey}`; + this.delete = `DELETE FROM ${id} WHERE ${this.#byKey}`; + } + + /** `UPDATE` of one row by key; `versions` gives each `json(...)` column's version to stamp. */ + update(set: SetNode, key: readonly SqliteValue[], versions: ReadonlyMap): Statement { + const params: SqliteValue[] = []; + const assignments = setSql(set, versions, params); + params.push(...key); + return { sql: `UPDATE ${quote(this.table)} SET ${assignments} WHERE ${this.#byKey}`, params }; + } + + /** `UPDATE` of every row matching `where`. */ + updateWhere(where: WhereNode, set: SetNode, versions: ReadonlyMap): Statement { + const params: SqliteValue[] = []; + const assignments = setSql(set, versions, params); + const condition = whereSql(where, params); + return { sql: `UPDATE ${quote(this.table)} SET ${assignments} WHERE ${condition}`, params }; + } + + /** `DELETE` of every row matching `where`. */ + deleteWhere(where: WhereNode): Statement { + const params: SqliteValue[] = []; + return { sql: `DELETE FROM ${quote(this.table)} WHERE ${whereSql(where, params)}`, params }; + } + + /** + * A page: rows matching `where` in `order`, after the order values `after` (row-value comparison; the key in the + * order makes it total), at most `limit` (bound). + */ + select( + where: WhereNode | undefined, + order: Order, + after: readonly SqliteValue[] | undefined, + limit: number, + ): Statement { + const params: SqliteValue[] = []; + const conditions: string[] = []; + if (where !== undefined) conditions.push(whereSql(where, params)); + if (after !== undefined) { + const columns = order.columns.map(quote).join(", "); + conditions.push(`(${columns}) ${order.direction === "asc" ? ">" : "<"} (${after.map(() => "?").join(", ")})`); + params.push(...after); + } + const filter = conditions.length === 0 ? "" : ` WHERE ${conditions.join(" AND ")}`; + const direction = order.direction === "asc" ? "ASC" : "DESC"; + const sorting = order.columns.map((column) => `${quote(column)} ${direction}`).join(", "); + params.push(limit); + return { sql: `SELECT ${this.#select} FROM ${quote(this.table)}${filter} ORDER BY ${sorting} LIMIT ?`, params }; + } + + /** `count(*)` of the rows matching `where`. */ + count(where: WhereNode | undefined): Statement { + const params: SqliteValue[] = []; + const filter = where === undefined ? "" : ` WHERE ${whereSql(where, params)}`; + return { sql: `SELECT count(*) AS count FROM ${quote(this.table)}${filter}`, params }; + } +} + +/** The assignments of `set`; a written `json(...)` column also stamps its version (a clear stamps `NULL`). */ +function setSql(set: SetNode, versions: ReadonlyMap, params: SqliteValue[]): string { + const assignments: string[] = []; + for (const [column, node] of Object.entries(set)) { + assignments.push(`${quote(column)} = ${exprSql(node, params)}`); + const version = versions.get(column); + if (version !== undefined) { + assignments.push(`${quote(versionColumn(column))} = ?`); + params.push(node === null ? null : version); + } + } + return assignments.join(", "); +} + +function exprSql(node: ExprNode, params: SqliteValue[]): string { + if (node === null) return "NULL"; + if (typeof node !== "object") { + params.push(toSqlite(node)); + return "?"; + } + if ("col" in node) return quote(node.col); + if ("json" in node) { + // Serialized once at staging already (rule 8): this cannot throw. + params.push(JSON.stringify(node.json)); + return "?"; + } + if ("when" in node) { + const [condition, a, b] = node.when; + return `(CASE WHEN ${whereSql(condition, params)} THEN ${exprSql(a, params)} ELSE ${exprSql(b, params)} END)`; + } + if ("replacePrefix" in node) { + const [subject, from, to] = node.replacePrefix; + const length = [...from].length; + // SQLite's `substr` counts characters of UTF-8 text: code points, as the prefix's length does. + // Parameters in the order their `?` occur in the text. + const value = exprSql(subject, params); + params.push(length, from, to); + const rest = exprSql(subject, params); + params.push(length + 1); + const unchanged = exprSql(subject, params); + return `(CASE WHEN substr(${value}, 1, ?) = ? THEN ? || substr(${rest}, ?) ELSE ${unchanged} END)`; + } + if ("coalesce" in node) + return `coalesce(${exprSql(node.coalesce[0], params)}, ${exprSql(node.coalesce[1], params)})`; + if ("add" in node) return `(${exprSql(node.add[0], params)} + ${exprSql(node.add[1], params)})`; + if ("sub" in node) return `(${exprSql(node.sub[0], params)} - ${exprSql(node.sub[1], params)})`; + if ("max" in node) return `max(${exprSql(node.max[0], params)}, ${exprSql(node.max[1], params)})`; + if ("min" in node) return `min(${exprSql(node.min[0], params)}, ${exprSql(node.min[1], params)})`; + return `(${exprSql(node.concat[0], params)} || ${exprSql(node.concat[1], params)})`; +} + +const COMPARISONS = { eq: "=", ne: "<>", lt: "<", lte: "<=", gt: ">", gte: ">=" } as const; + +/** A condition with its operands bound; SQL's `NULL` rules are the vocabulary's. */ +export function whereSql(node: WhereNode, params: SqliteValue[]): string { + if ("and" in node) + return node.and.length === 0 ? "1" : `(${node.and.map((each) => whereSql(each, params)).join(" AND ")})`; + if ("or" in node) + return node.or.length === 0 ? "0" : `(${node.or.map((each) => whereSql(each, params)).join(" OR ")})`; + if ("not" in node) return `(NOT ${whereSql(node.not, params)})`; + if ("isNull" in node) return `${quote(node.isNull)} IS NULL`; + if ("inArray" in node) { + const [column, values] = node.inArray; + params.push(JSON.stringify(values.map(toSqlite))); + return `${quote(column)} IN (SELECT value FROM json_each(?))`; + } + if ("prefix" in node) { + const [column, prefix] = node.prefix; + const upper = prefixUpperBound(prefix); + params.push(prefix); + if (upper === undefined) return `${quote(column)} >= ?`; + params.push(upper); + return `(${quote(column)} >= ? AND ${quote(column)} < ?)`; + } + // The remaining nodes are the comparisons, one entry each; `Object.entries` loses that. + const [op, [column, value]] = Object.entries(node)[0] as [ + keyof typeof COMPARISONS, + readonly [string, string | number | boolean], + ]; + params.push(toSqlite(value)); + return `${quote(column)} ${COMPARISONS[op]} ?`; +} diff --git a/packages/chord/src/next/store/step.ts b/packages/chord/src/next/store/step.ts new file mode 100644 index 000000000..8a428774a --- /dev/null +++ b/packages/chord/src/next/store/step.ts @@ -0,0 +1,69 @@ +// The commit step's working state (slice 6a §3) and the shape every write takes into it: a write staged before the +// queue (serialized, rule 8), applied inside the step in rule 9's order. + +import type { SqliteExecutor, SqliteValue } from "../../sqlite/index.ts"; +import type { TableRecord } from "./catalog.ts"; +import type { StoreWrite } from "./writes.ts"; + +/** Where a staged write goes in its commit (rule 9): retirements, creations, then everything else in order. */ +export type WriteOrder = "retire" | "create" | "write"; + +/** A write staged for the commit step: its order and what it does there, returning the write as recorded. */ +export type StagedWrite = { + readonly order: WriteOrder; + /** Applies the write inside the step's backend transaction; resolves with the write as the outbox records it. */ + apply(step: Step): Promise; +}; + +/** + * One commit step: its backend transaction, its seq, the document ids it assigns (S10, monotonic with seq), the table + * records it changed (the store's catalog takes them after `COMMIT`) and the statements it ran + * (`chord.store.statements`). + */ +export class Step { + readonly executor: SqliteExecutor; + readonly seq: number; + /** The next document id; the store adopts it after `COMMIT`. */ + nextDocumentId: number; + /** The ids of the commit's created documents, in the order of their writes. */ + readonly created: number[] = []; + /** The table records this step changed, by SQL name. */ + readonly tables = new Map(); + readonly #catalog: ReadonlyMap; + statements = 0; + + constructor( + executor: SqliteExecutor, + seq: number, + nextDocumentId: number, + catalog: ReadonlyMap, + ) { + this.executor = executor; + this.seq = seq; + this.nextDocumentId = nextDocumentId; + this.#catalog = catalog; + } + + /** The record of table `name` as this step leaves it so far. */ + table(name: string): TableRecord | undefined { + return this.tables.get(name) ?? this.#catalog.get(name); + } + + /** Runs one statement. */ + run(sql: string, ...params: SqliteValue[]): Promise { + this.statements++; + return this.executor.run(sql, ...params); + } + + /** Runs one statement and resolves with the rows it changed. */ + runCounted(sql: string, ...params: SqliteValue[]): Promise { + this.statements++; + return this.executor.runCounted(sql, ...params); + } + + /** Runs one query. */ + all(sql: string, ...params: SqliteValue[]): Promise { + this.statements++; + return this.executor.all(sql, ...params); + } +} diff --git a/packages/chord/src/next/store/store.ts b/packages/chord/src/next/store/store.ts new file mode 100644 index 000000000..f24f2d51e --- /dev/null +++ b/packages/chord/src/next/store/store.ts @@ -0,0 +1,347 @@ +// The store (slice 6a §2, §3): opened over a backend, one commit queue that serializes only the commit step (the seal +// fence's admission check, `BEGIN`, the writes in rule 9's order, the outbox commit, `lastSeq`, `COMMIT`, then the +// owner's adoption), reads guarded as Durable's `GuardedStorage` guards them, and the stop after a backend error. + +import { awaitWithContext, type Context } from "../../context/index.ts"; +import type { SqliteDatabase, SqliteExecutor } from "../../sqlite/index.ts"; +import { ChordError } from "../data/index.ts"; +import type { ArchivePart } from "./archive.ts"; +import type { Backend } from "./backend.ts"; +import { readCatalog, type TableRecord, writeCatalog } from "./catalog.ts"; +import { StoreExport } from "./export.ts"; +import { OUTBOX_FORMAT, Outbox } from "./outbox.ts"; +import { applyStoreSchema, type MetaRow, TABLES } from "./schema.ts"; +import { type StagedWrite, Step, type WriteOrder } from "./step.ts"; +import type { StoreWrite } from "./writes.ts"; + +/** A store as apps hold it: opened by `openStore`, owned and closed by the app. */ +export interface Store { + /** The seq of the store's last commit (0 for a new store). */ + readonly lastSeq: number; + /** The outbox the sync layer reads; `undefined` without one. */ + readonly outbox: Outbox | undefined; + /** + * The store's state after its `lastSeq` as archive parts (slice 6a §13), holding the commit queue from the first + * `next()` until the iteration ends, throws, is closed or `context` aborts. + */ + export(context: Context): AsyncIterableIterator; + /** + * Closes the store: waits for a running commit step, rejects queued commits and reads `released`, closes the + * backend. Idempotent, also after `storage_failed`; `context` ends only this caller's wait (the close goes on). + */ + close(context: Context): Promise; +} + +/** What `openStore` takes besides the backend. */ +export type StoreOptions = { + /** Turns the outbox on (`true`) or off (`false`); omitted, the persisted setting stays (a new store: off). */ + readonly outbox?: boolean; +}; + +/** + * Opens a store over `backend`: the store's schema migrations, the UTF-8 check, the metadata read once, the outbox + * switched when asked. A failure closes the backend and rejects (rule 12). + */ +export async function openStore(backend: Backend, context: Context, options: StoreOptions = {}): Promise { + const { db } = backend; + try { + context.abortSignal?.throwIfAborted(); + const meta = await switchOutbox(db, await applyStoreSchema(db), options.outbox); + return new ChordStore(db, meta, await readCatalog(db)); + } catch (error) { + try { + await db.close(); + } catch { + // The open's own failure is the one to report. + } + throw error; + } +} + +/** Turns the outbox on (from the current `lastSeq`) or off (its commits dropped) when `outbox` asks for the other. */ +async function switchOutbox(db: SqliteDatabase, meta: MetaRow, outbox: boolean | undefined): Promise { + if (outbox === undefined || outbox === (meta.outbox === 1)) return meta; + const floor = outbox ? meta.last_seq : 0; + await db.transaction(async (executor) => { + await executor.run( + `UPDATE ${TABLES.meta} SET outbox = ?, outbox_floor = ? WHERE singleton = 1`, + outbox ? 1 : 0, + floor, + ); + if (!outbox) await executor.run(`DELETE FROM ${TABLES.outbox}`); + }); + return { ...meta, outbox: outbox ? 1 : 0, outbox_floor: floor }; +} + +/** The seal fence's two checks of a commit (§1.3), as the commit step calls them. */ +export type CommitFence = { + /** At the step's entry: `false` rejects the commit `released` and writes nothing. */ + admit(): boolean; + /** After `COMMIT`: adopts the commit, `ids` being its created documents' ids in the order of their writes. */ + adopt(seq: number, ids: readonly number[]): void; +}; + +/** Where a store stands: open, stopped by a backend error, or closed. */ +type StoreState = + | { readonly kind: "open" } + | { readonly kind: "failed"; readonly error: unknown } + | { readonly kind: "closed" }; + +const ORDER: readonly WriteOrder[] = ["retire", "create", "write"]; + +const NOTHING = (): void => {}; + +/** The store behind `Store`, with what Chord's layers use: commits, guarded reads, the host's failure listener. */ +export class ChordStore implements Store { + readonly #db: SqliteDatabase; + #lastSeq: number; + #nextDocumentId: number; + #outbox: Outbox | undefined; + #state: StoreState = { kind: "open" }; + /** The commit queue's tail: each step starts after the one before it settled. */ + #tail: Promise = Promise.resolve(); + /** Reads underway, which `close` waits for. */ + readonly #underway = new Set>(); + /** The open host's listener for `storage_failed`, told once; one host per store (rule 7). */ + #listener: ((error: unknown) => void) | undefined; + #closing: Promise | undefined; + /** The recorded tables, by SQL name; changed only after a step's `COMMIT`. */ + readonly #catalog: Map; + /** The ends of the exports holding the queue or waiting for it, which closing the store calls. */ + readonly #held = new Set<() => void>(); + + constructor(db: SqliteDatabase, meta: MetaRow, catalog: Map) { + this.#db = db; + this.#catalog = catalog; + this.#lastSeq = meta.last_seq; + this.#nextDocumentId = meta.next_document_id; + this.#outbox = meta.outbox === 1 ? new Outbox(this, meta.outbox_floor) : undefined; + } + + get lastSeq(): number { + return this.#lastSeq; + } + + get outbox(): Outbox | undefined { + return this.#outbox; + } + + /** The id the next created document gets. */ + get nextDocumentId(): number { + return this.#nextDocumentId; + } + + /** Every recorded table, by SQL name, as of the last commit. */ + get tables(): ReadonlyMap { + return this.#catalog; + } + + export(context: Context): AsyncIterableIterator { + this.assertUsable(); + return new StoreExport(this, context); + } + + /** + * Takes the commit queue (an export): resolves once every earlier commit settled, with the release that lets the + * later ones run (idempotent). Closing the store calls `end` (from the call on, also while waiting, and at once for a + * store closed before the call took the queue), which must release. + */ + take(end: () => void): Promise<() => void> { + const held = Promise.withResolvers(); + const release = (): void => { + this.#held.delete(end); + held.resolve(); + }; + this.#held.add(end); + const taken = this.#tail.then(() => { + // A store closed meanwhile (also before the export's first `next()`) holds nothing: the export ends. + if (this.#state.kind === "closed") { + release(); + end(); + } + return release; + }); + this.#tail = taken.then(() => held.promise); + return taken; + } + + /** The recorded table `name` (its SQL name), as of the last commit. */ + table(name: string): TableRecord | undefined { + return this.#catalog.get(name); + } + + /** Attaches the open host's `storage_failed` listener; a store attached to an open host is refused (rule 7). */ + attach(listener: (error: unknown) => void): void { + if (this.#listener !== undefined) { + throw new ChordError({ code: "invalid", reason: "args", message: "store: already attached to an open host" }); + } + this.#listener = listener; + } + + /** The host closed: the store may serve another one. */ + detach(): void { + this.#listener = undefined; + } + + /** + * Commits `writes` as one seq when the commit queue reaches it: `fence.admit()` at the step's entry, the writes in + * rule 9's order (retirements, creations, the rest), the outbox commit, `lastSeq`, `COMMIT`, then `fence.adopt` + * before the next step starts. Resolves with the seq. A backend error or a throwing `adopt` stops the store + * (`storage_failed`); a refused admission rejects `released`. `context` counts the step (no abort once queued). + */ + commit(writes: readonly StagedWrite[], fence: CommitFence, context: Context): Promise { + try { + this.assertUsable(); + } catch (error) { + return Promise.reject(error); + } + const step = this.#tail.then(() => this.#step(writes, fence, context)); + this.#tail = step.then(NOTHING, NOTHING); + return step; + } + + /** + * Runs `read` on the backend under the guard (rule 11, Durable's `GuardedStorage`): refused once the store stopped + * or closed; a backend error fails the store unless `context` was aborted (then `aborted`); a read settling after + * the store stopped rejects `unavailable/storage_failed`. + */ + async read(read: (executor: SqliteExecutor) => Promise, context: Context): Promise { + this.assertUsable(); + const call = (async () => read(this.#db))(); + this.#underway.add(call); + let result: T; + try { + result = await call; + } catch (error) { + this.assertUsable(); + const signal = context.abortSignal; + if (signal?.aborted) + throw new ChordError({ code: "aborted", message: "store: read aborted" }, { cause: signal.reason }); + throw this.#fail(error); + } finally { + this.#underway.delete(call); + } + this.assertUsable(); + return result; + } + + /** + * Runs `write` on the backend outside the commit queue (the outbox's `trim`), in a backend transaction, guarded as + * a read without a context: any error stops the store. + */ + async maintain(write: (executor: SqliteExecutor) => Promise): Promise { + this.assertUsable(); + const call = this.#db.transaction(write); + this.#underway.add(call); + try { + await call; + } catch (error) { + this.assertUsable(); + throw this.#fail(error); + } finally { + this.#underway.delete(call); + } + } + + close(context: Context): Promise { + this.#closing ??= this.#close(); + return awaitWithContext(this.#closing, context); + } + + async #close(): Promise { + this.#state = { kind: "closed" }; + for (const end of [...this.#held]) end(); + await this.#tail; + await Promise.allSettled([...this.#underway]); + await this.#db.close(); + } + + async #step(writes: readonly StagedWrite[], fence: CommitFence, context: Context): Promise { + this.assertUsable(); + if (!fence.admit()) { + throw new ChordError({ code: "released", reason: "discarded", message: "store: the commit's attempt ended" }); + } + if (writes.length === 0) { + // An empty commit takes no seq; a throwing `adopt` stops the store as for any commit. + try { + fence.adopt(this.#lastSeq, []); + } catch (error) { + throw this.#fail(error); + } + return this.#lastSeq; + } + const seq = this.#lastSeq + 1; + // Past the safe range seqs stop counting exactly (a later one would repeat): the store cannot number further. + if (seq > Number.MAX_SAFE_INTEGER) throw this.#fail(new Error("store: no seq past Number.MAX_SAFE_INTEGER")); + const outbox = this.#outbox !== undefined; + let step: Step; + try { + step = await this.#db.transaction(async (executor) => { + const step = new Step(executor, seq, this.#nextDocumentId, this.#catalog); + const recorded: StoreWrite[] | undefined = outbox ? [] : undefined; + for (const order of ORDER) { + for (const write of writes) { + if (write.order !== order) continue; + const done = await write.apply(step); + recorded?.push(done); + } + } + await writeCatalog(step); + if (recorded !== undefined) { + await step.run( + `INSERT INTO ${TABLES.outbox} (seq, format, writes) VALUES (?, ?, ?)`, + seq, + OUTBOX_FORMAT, + JSON.stringify(recorded), + ); + } + await step.run( + `UPDATE ${TABLES.meta} SET last_seq = ?, next_document_id = ? WHERE singleton = 1`, + seq, + step.nextDocumentId, + ); + return step; + }); + } catch (error) { + throw this.#fail(error); + } + // `COMMIT` returned: the commit is durable, and memory follows. + this.#lastSeq = seq; + this.#nextDocumentId = step.nextDocumentId; + for (const [name, record] of step.tables) this.#catalog.set(name, record); + const metrics = context.telemetry.metrics; + metrics?.counter("chord.store.commits").add(1); + metrics?.counter("chord.store.statements").add(step.statements); + try { + fence.adopt(seq, step.created); + } catch (error) { + throw this.#fail(error); + } + return seq; + } + + /** Throws `unavailable/storage_failed` once the store stopped, `released` once it closed. */ + assertUsable(): void { + const state = this.#state; + if (state.kind === "failed") throw storageFailed(state.error); + if (state.kind === "closed") + throw new ChordError({ code: "released", reason: "closed", message: "store: closed" }); + } + + /** Stops the store after `error` (once; its host is told once); returns the error the caller rejects with. */ + #fail(error: unknown): ChordError { + if (this.#state.kind === "open") { + this.#state = { kind: "failed", error }; + this.#listener?.(error); + } + return storageFailed(error); + } +} + +/** The `unavailable/storage_failed` error of a stopped store, its backend error as the cause. */ +function storageFailed(error: unknown): ChordError { + return new ChordError( + { code: "unavailable", reason: "storage_failed", message: "store: storage failed" }, + { cause: error }, + ); +} diff --git a/packages/chord/src/next/store/table-changes.ts b/packages/chord/src/next/store/table-changes.ts new file mode 100644 index 000000000..faa66cd32 --- /dev/null +++ b/packages/chord/src/next/store/table-changes.ts @@ -0,0 +1,147 @@ +// A transaction's writes to one table (slice 6a §6, §7; W10): checked and staged synchronously like a draft's ops (no +// I/O, a bad row throws at its own call), kept in staging order for the commit step, and overlaid on committed rows +// for `tx.get` until a bulk write makes that impossible (`invalid/read_after_bulk`). + +import type { Context } from "../../context/index.ts"; +import { invalid } from "../data/internal.ts"; +import { applySet, type EvaluatedRow } from "./evaluate.ts"; +import { compileSet, compileWhere, type SetNode } from "./expression.ts"; +import type { StagedWrite } from "./step.ts"; +import type { TableHandle } from "./table-handle.ts"; +import { + jsonVersions, + recordedVersions, + stageDelete, + stageDeleteWhere, + stagePut, + stageUpdate, + stageUpdateWhere, +} from "./table-writes.ts"; + +/** One staged write of one key, as the overlay replays it. */ +type KeyWrite = + | { readonly kind: "put"; readonly row: EvaluatedRow } + | { readonly kind: "update"; readonly set: SetNode } + | { readonly kind: "delete" }; + +/** + * The writes of one transaction to one table: each appended to the transaction's staged list (`staged`, one order + * across tables) and, for the single-key ones, to its key's overlay. + */ +export class TableChanges { + readonly handle: TableHandle; + readonly #staged: StagedWrite[]; + readonly #byKey = new Map(); + /** An `updateWhere` or `deleteWhere` was staged: `get` is refused from then on (W10). */ + #bulk = false; + + constructor(handle: TableHandle, staged: StagedWrite[]) { + this.handle = handle; + this.#staged = staged; + } + + /** Stages a `put` of `row` (an upsert). */ + put(row: unknown): void { + const { definition, sql } = this.handle; + const encoded = definition.rows.put(row, definition.version); + this.#staged.push(stagePut(sql.put, sql.table, encoded, definition.version)); + this.#write(encoded.id, { kind: "put", row: encoded.row }); + } + + /** Stages an `update` of the row with `key` (`set`: values, or a callback of column references). */ + update(key: unknown, set: unknown): void { + const { definition, sql } = this.handle; + const encoded = definition.rows.key(key, "update"); + const compiled = this.#set(set, "update"); + const versions = jsonVersions(this.handle.definition, compiled); + this.#staged.push( + stageUpdate( + sql.update(compiled, encoded.params, versions), + sql.table, + encoded.key, + compiled, + recordedVersions(compiled, versions), + ), + ); + this.#write(encoded.id, { kind: "update", set: compiled }); + } + + /** Stages a `delete` of the row with `key`. */ + delete(key: unknown): void { + const { definition, sql } = this.handle; + const encoded = definition.rows.key(key, "delete"); + this.#staged.push(stageDelete({ sql: sql.delete, params: [...encoded.params] }, sql.table, encoded.key)); + this.#write(encoded.id, { kind: "delete" }); + } + + /** Stages an `updateWhere`: every row matching `where` gets `set`. */ + updateWhere(where: unknown, set: unknown): void { + const { definition, sql } = this.handle; + const condition = compileWhere( + definition, + typeof where === "function" ? where(definition.refs) : where, + "updateWhere", + ); + const compiled = this.#set(set, "updateWhere"); + const versions = jsonVersions(this.handle.definition, compiled); + const statement = sql.updateWhere(condition, compiled, versions); + this.#staged.push( + stageUpdateWhere(statement, sql.table, condition, compiled, recordedVersions(compiled, versions)), + ); + this.#bulk = true; + } + + /** Stages a `deleteWhere`: every row matching `where` goes. */ + deleteWhere(where: unknown): void { + const { definition, sql } = this.handle; + const condition = compileWhere( + definition, + typeof where === "function" ? where(definition.refs) : where, + "deleteWhere", + ); + this.#staged.push(stageDeleteWhere(sql.deleteWhere(condition), sql.table, condition)); + this.#bulk = true; + } + + /** + * The row with `key` as this transaction leaves it: the committed row with this transaction's `put`, `update` and + * `delete` of the key replayed; refused after a bulk write (W10). + */ + async get(key: unknown, context: Context): Promise { + const { definition } = this.handle; + if (this.#bulk) { + throw invalid( + "read_after_bulk", + `${definition.name}: tx.get after an updateWhere or deleteWhere on the table`, + ); + } + const encoded = definition.rows.key(key, "get"); + const writes = this.#byKey.get(encoded.id) ?? []; + let start = writes.length; + while (start > 0 && writes[start - 1]!.kind === "update") start--; + const from = writes[start - 1]; + // A committed row is decoded by the table's columns: scalars and JSON, what the evaluator takes. + let row: EvaluatedRow | undefined = + from === undefined + ? ((await this.handle.get(encoded.key, context)) as EvaluatedRow | undefined) + : from.kind === "put" + ? from.row + : undefined; + for (const write of writes.slice(start)) { + // An `update` of an absent row changes nothing. + if (row !== undefined && write.kind === "update") row = applySet(write.set, row, definition.name); + } + return row; + } + + #set(set: unknown, what: string): SetNode { + const { definition } = this.handle; + return compileSet(definition, typeof set === "function" ? set(definition.refs) : set, what); + } + + #write(id: string, write: KeyWrite): void { + const writes = this.#byKey.get(id); + if (writes === undefined) this.#byKey.set(id, [write]); + else writes.push(write); + } +} diff --git a/packages/chord/src/next/store/table-evolve.ts b/packages/chord/src/next/store/table-evolve.ts new file mode 100644 index 000000000..3e31ff82e --- /dev/null +++ b/packages/chord/src/next/store/table-evolve.ts @@ -0,0 +1,306 @@ +// `evolveTable(previous, (t) => t…)` (slice 6a §9; decision "Second spec review: the majority"): the next version of a +// table from the edits of one migration. The editor's types order the edits (`rename`, `add` and `json` before the +// data step, `drop`, `index` and `dropIndex` after it) and track the columns and key through the chain; at runtime one +// editor records the edits, checks each as it is made, and materializes the new version at the end. + +import type { TOptional } from "typebox"; +import { opt, type TSchema } from "../contracts/index.ts"; +import { editWith, type Increment } from "../contracts/internal.ts"; +import { + type ColumnSchema, + type Columns, + type ColumnTypeOf, + checkColumnName, + columnRecord, + describeColumns, + type OrderColumn, + type RowOf, + type ValueOf, +} from "./columns.ts"; +import { type Condition, compileSet, compileWhere, type SetOf } from "./expression.ts"; +import type { ColumnRefs } from "./query.ts"; +import { INDEX_NAME, orderColumns, TableDefinition, type TableOptions } from "./table.ts"; +import type { DataStep, TableEdit } from "./table-history.ts"; + +/** The edits after the data step: `drop`, `index` and `dropIndex`; `C` and `K` are the columns and key so far. */ +export interface TableTailEditor { + /** Drops a non-key column; its indexes are dropped first, in the same commit. */ + drop>(name: N): TableTailEditor, K>; + /** Adds an index (`[a-z0-9_]+`) over non-`opt` `str` or `int` columns; it ends with the key. */ + index(name: string, columns: readonly OrderColumn[]): TableTailEditor; + /** Drops an index. */ + dropIndex(name: string): TableTailEditor; + /** Type only: the columns and key so far. */ + readonly "~columns"?: C; + readonly "~key"?: K; +} + +/** The editor `evolveTable`'s edit gets: every edit; the column edits only before the data step. */ +export interface TableEditor extends TableTailEditor { + /** Renames a column; the key, the indexes and a `json(...)` column's version column follow. */ + rename( + from: F, + to: T, + ): TableEditor, RenamedKey>; + /** Adds a column: `opt(...)`, or a scalar one with a `default` (existing rows get it). */ + add(name: N, column: S): TableEditor, K>; + /** A `json(...)` column's new version (its nullability kept): no DDL, no data step, cells migrate on access. */ + json, S extends ColumnSchema>(name: N, column: S): TableEditor, K>; + /** + * The data step: `patch` gets each row as the columns stand here and returns new values of non-key columns (`{}` + * stamps the row only); synchronous and pure, run in chunks of 100 in key order. + */ + rows(patch: (row: RowOf) => PatchOf): TableTailEditor; + /** The data step as one `updateWhere` over the rows not migrated yet. */ + update(step: { + readonly where?: (row: ColumnRefs) => Condition; + readonly set: SetOf | ((row: ColumnRefs) => SetOf); + }): TableTailEditor; +} + +/** A column `add` takes: `opt(...)` or one with a `default`. */ +export type AddedColumn = ColumnSchema & (TOptional | { readonly default: unknown }); + +/** A `rows` patch: new values of non-key columns; `undefined` clears an `opt(...)` one. */ +export type PatchOf = { + readonly [P in Exclude]?: ValueOf; +}; + +type Renamed = { + readonly [P in keyof C as P extends F ? T : P]: C[P]; +}; + +type RenamedKey = { + readonly [I in keyof K]: K[I] extends F ? T : K[I]; +}; + +type With = { + readonly [P in keyof C | N]: P extends N ? S : C[P & keyof C]; +}; + +type Without = { readonly [P in Exclude]: C[P] }; + +type JsonNames = { [P in keyof C]: ColumnTypeOf extends "json" ? P : never }[keyof C] & string; + +type WithJson = With< + C, + N, + C[N] extends TOptional ? TOptional : S +>; + +/** `V + 1` for a literal version (contracts' `Increment`); `number` stays `number`. */ +type Next = number extends V + ? number + : `${Increment<`${V}`>}` extends `${infer N extends number}` + ? N + : number; + +/** + * Appends the next version (`previous.version + 1`) with the edits `edit` makes; `edit` must return the editor it was + * given. Every version is materialized and fingerprinted here; the object serves the newest, and its activation + * migrates a store at an older version through each one. + */ +export function evolveTable< + C extends Columns, + K extends readonly string[], + V extends number, + E extends { readonly "~columns"?: Columns; readonly "~key"?: readonly string[] }, +>( + previous: TableDefinition, + edit: (t: TableEditor) => E, +): TableDefinition, NonNullable, Next> { + // The runtime editor works on untyped definitions; the types live in the editor interfaces. + const editor = new Editor(previous as unknown as TableDefinition); + // The editor interfaces track the edits in the types; at runtime every edit returns the same editor. + editWith(editor as unknown as TableEditor, edit, `${previous.name}: evolveTable`); + return editor.finish() as unknown as TableDefinition, NonNullable, Next>; +} + +/** + * The editor at runtime: the columns, key and indexes as the edits leave them, the edits before and after the data + * step, and the step. Each edit is checked as it is made (declaration policy: JavaScript callers reach the checks). + */ +class Editor { + readonly #previous: TableDefinition; + readonly #what: string; + #schemas: { [name: string]: ColumnSchema }; + #key: string[]; + readonly #indexes: Map; + /** `before` until the data step or the first `drop`, `index` or `dropIndex`. */ + #phase: "before" | "after" = "before"; + readonly #before: TableEdit[] = []; + #step: DataStep | undefined; + readonly #after: TableEdit[] = []; + + constructor(previous: TableDefinition) { + this.#previous = previous; + this.#what = `${previous.name} version ${previous.version + 1}`; + this.#schemas = { ...previous.schemas }; + this.#key = [...previous.key]; + this.#indexes = new Map(previous.indexes); + } + + rename(from: string, to: string): this { + this.#assertBefore("rename"); + this.#assertColumn(from, "rename"); + // Checked here, not by the final definition: a later edit may rename it again, but this one runs as DDL. + checkColumnName(to, `${this.#what}: rename`); + this.#assertNewName(to, "rename", from); + const rename = (column: string): string => (column === from ? to : column); + this.#schemas = Object.fromEntries(Object.entries(this.#schemas).map(([name, schema]) => [rename(name), schema])); + this.#key = this.#key.map(rename); + for (const [name, columns] of this.#indexes) this.#indexes.set(name, columns.map(rename)); + this.#before.push({ kind: "rename", from, to }); + return this; + } + + add(name: string, column: ColumnSchema): this { + this.#assertBefore("add"); + this.#assertNewName(name, "add"); + const [described] = describeColumns({ [name]: column }, `${this.#what}: add`); + if (!described!.optional && (described!.type === "json" || described!.default === undefined)) { + throw new Error(`${this.#what}: add: column "${name}" needs opt(...) or a default (existing rows get it)`); + } + this.#schemas[name] = column; + this.#before.push({ kind: "add", name, column: columnRecord(described!) }); + return this; + } + + json(name: string, column: ColumnSchema): this { + this.#assertBefore("json"); + const existing = describeColumns({ [name]: this.#assertColumn(name, "json") }, this.#what)[0]!; + if (existing.type !== "json") throw new Error(`${this.#what}: json: column "${name}" is not a json(...) column`); + if (Object.hasOwn(column, "~optional")) { + throw new Error(`${this.#what}: json: pass json(...) itself; the column keeps its nullability`); + } + const schema = existing.optional ? opt(column) : column; + const described = describeColumns({ [name]: schema }, `${this.#what}: json`)[0]!; + if (described.type !== "json") throw new Error(`${this.#what}: json: expected a json(...) column`); + const [before, after] = [existing.json, described.json]; + // D9 across table versions: a cell's version says which schema it was written with. + if (after.version < before.version) { + throw new Error(`${this.#what}: json: column "${name}" goes back from version ${before.version}`); + } + if (after.version === before.version && after.fingerprint !== before.fingerprint) { + throw new Error( + `${this.#what}: json: column "${name}" changes its schema at version ${before.version} (raise its version)`, + ); + } + this.#schemas[name] = schema; + this.#before.push({ kind: "json", name, column: columnRecord(described) }); + return this; + } + + rows(patch: (row: { readonly [column: string]: unknown }) => unknown): this { + this.#assertBefore("rows"); + if (typeof patch !== "function") throw new Error(`${this.#what}: rows: expected a function`); + this.#step = { kind: "rows", table: this.#current(), patch }; + this.#phase = "after"; + return this; + } + + update(step: { readonly where?: unknown; readonly set: unknown }): this { + this.#assertBefore("update"); + const table = this.#current(); + const what = `version ${table.version} update`; + const where = typeof step.where === "function" ? step.where(table.refs) : step.where; + const set = typeof step.set === "function" ? step.set(table.refs) : step.set; + this.#step = { + kind: "update", + table, + where: where === undefined ? undefined : compileWhere(table, where, what), + set: compileSet(table, set, what), + }; + this.#phase = "after"; + return this; + } + + drop(name: string): this { + this.#phase = "after"; + this.#assertColumn(name, "drop"); + if (this.#key.includes(name)) throw new Error(`${this.#what}: drop: "${name}" is a key column`); + for (const [index, columns] of this.#indexes) { + if (!columns.includes(name)) continue; + this.#indexes.delete(index); + this.#after.push({ kind: "dropIndex", name: index }); + } + const { [name]: _dropped, ...schemas } = this.#schemas; + this.#schemas = schemas; + this.#after.push({ kind: "drop", name }); + return this; + } + + index(name: string, columns: readonly string[]): this { + this.#phase = "after"; + const what = `${this.#what}: index "${name}"`; + if (!INDEX_NAME.test(name)) throw new Error(`${what}: expected [a-z0-9_]+`); + if (this.#indexes.has(name)) throw new Error(`${what} exists`); + // Checked against the columns at this edit: the migration creates the index here, whatever later edits do. + const byName = new Map(describeColumns(this.#schemas, this.#what).map((column) => [column.name, column])); + const checked = orderColumns(byName, columns, what); + this.#indexes.set(name, checked); + this.#after.push({ kind: "index", name, columns: checked }); + return this; + } + + dropIndex(name: string): this { + this.#phase = "after"; + if (!this.#indexes.delete(name)) throw new Error(`${this.#what}: dropIndex: no index "${name}"`); + this.#after.push({ kind: "dropIndex", name }); + return this; + } + + /** The new version, materialized and fingerprinted, linked to the previous one and the migration from it. */ + finish(): TableDefinition { + const migration = { before: this.#before, step: this.#step, after: this.#after }; + return new TableDefinition(this.#previous.name, this.#options(), { previous: this.#previous, migration }); + } + + /** The table as the edits so far leave it, at the new version: what a data step reads and writes. */ + #current(): TableDefinition { + return new TableDefinition(this.#previous.name, this.#options()); + } + + #options(): TableOptions { + const version = this.#previous.version + 1; + const options = { + version, + columns: { ...this.#schemas }, + key: [...this.#key], + indexes: Object.fromEntries(this.#indexes), + }; + // The key and indexes are names the edits kept; the constructor checks them as it checks a base's. + return options as unknown as TableOptions; + } + + #assertBefore(edit: string): void { + if (this.#phase === "after") { + throw new Error(`${this.#what}: ${edit} comes before the data step and before drop, index and dropIndex`); + } + } + + /** + * `name` names no column yet, also ignoring ASCII case (SQLite's column names do): checked per edit, since a later + * `drop` cannot take back an `add` the migration runs first. A rename may change only the case of `from`. + */ + #assertNewName(name: string, edit: string, from?: string): void { + const folded = foldAscii(name); + for (const existing of Object.keys(this.#schemas)) { + if (existing !== from && foldAscii(existing) === folded) { + throw new Error( + `${this.#what}: ${edit}: column "${existing}" exists (names differing only in case collide)`, + ); + } + } + } + + #assertColumn(name: string, edit: string): ColumnSchema { + if (!Object.hasOwn(this.#schemas, name)) throw new Error(`${this.#what}: ${edit}: no column "${name}"`); + return this.#schemas[name]!; + } +} + +/** `name` with ASCII letters lowercased and nothing else changed, as SQLite folds identifiers. */ +function foldAscii(name: string): string { + return name.replace(/[A-Z]+/g, (letters) => letters.toLowerCase()); +} diff --git a/packages/chord/src/next/store/table-handle.ts b/packages/chord/src/next/store/table-handle.ts new file mode 100644 index 000000000..65e9dcde1 --- /dev/null +++ b/packages/chord/src/next/store/table-handle.ts @@ -0,0 +1,149 @@ +// Table handles (slice 6a §6; decision "Query runner"): what `self.table(Def)` returns, one per attempt and table: reads +// of committed rows (`page`, `rows`, `get`, `count`) through the store's guarded reads, and what a transaction's +// writes to the table are staged with. + +import type { Context } from "../../context/index.ts"; +import type { SqliteValue } from "../../sqlite/index.ts"; +import { ChordError } from "../data/index.ts"; +import { invalid } from "../data/internal.ts"; +import type { Columns, RowOf } from "./columns.ts"; +import { decodeCursor, encodeCursor, type Query } from "./query.ts"; +import { TableSql } from "./sql.ts"; +import type { ChordStore } from "./store.ts"; +import type { KeyOf, TableDefinition } from "./table.ts"; + +/** One page of rows (`items`, as Durable's pages) and, when more follow, the cursor to continue after it. */ +export type Page = { readonly items: T[]; readonly next?: string }; + +/** What `page` takes besides the query, the limit and the context. */ +export type PageOptions = { + /** A cursor from an earlier page of the same order. */ + readonly after?: string; +}; + +/** A table of this object: reads of committed state, the context last. Writes go through `self.tx`. */ +export interface Table { + /** Up to `limit` rows of `q` (after `options.after`), and the cursor of the next page when there is one. */ + page(q: Query, limit: number, context: Context, options?: PageOptions): Promise>>; + /** Every row of `q`, read a page of 100 at a time. */ + rows(q: Query, context: Context): AsyncIterable>; + /** The row with `key`, `undefined` when absent. */ + get(key: KeyOf, context: Context): Promise | undefined>; + /** The number of rows of `q`. */ + count(q: Query, context: Context): Promise; +} + +/** The handle type of table definition `D`: what `self.table(D)` returns. */ +export type TableOf = D extends TableDefinition ? Table : never; + +/** Who owns a table handle: its object (whose transactions alone write it), liveness check and live failure. */ +export type TableOwner = { + readonly label: string; + /** The object's documents: a transaction of another object refuses the handle. */ + readonly scope: object; + /** Throws `released` once the object ended. */ + assertLive(): void; + /** A row the code cannot read (a `json(...)` cell, slice 6a §9) fails the owner `broken` (§1.4). */ + fail(error: ChordError): void; +}; + +/** The rows `rows()` reads per page (rule 17). */ +export const ROWS_PAGE = 100; + +/** A table handle: the definition the attempt declared, the table's SQL name and statements, the store. */ +export class TableHandle implements Table { + readonly definition: TableDefinition; + readonly sql: TableSql; + readonly store: ChordStore; + readonly owner: TableOwner; + + constructor(definition: TableDefinition, sqlName: string, store: ChordStore, owner: TableOwner) { + this.definition = definition; + this.sql = new TableSql(sqlName, definition.columns, definition.rows.physical, definition.key); + this.store = store; + this.owner = owner; + } + + async page(q: Query, limit: number, context: Context, options: PageOptions = {}): Promise>> { + this.owner.assertLive(); + this.#assertQuery(q); + if (!Number.isSafeInteger(limit) || limit < 1) + throw invalid("args", `${this.definition.name}: limit is a positive integer`); + const order = q.order; + const after = + options.after === undefined ? undefined : decodeCursor(this.definition, this.sql.table, order, options.after); + const statement = this.sql.select(q.condition, order, after, limit + 1); + const stored = await this.store.read( + (executor) => executor.all<{ [column: string]: SqliteValue }>(statement.sql, ...statement.params), + context, + ); + const more = stored.length > limit; + if (more) stored.length = limit; + const items = stored.map((row) => this.decode(row, context)); + if (!more) return { items }; + const last = stored[limit - 1]!; + return { + items, + next: encodeCursor( + this.sql.table, + order, + order.columns.map((column) => last[column]!), + ), + }; + } + + async *rows(q: Query, context: Context): AsyncIterable> { + let after: string | undefined; + do { + const page = await this.page(q, ROWS_PAGE, context, after === undefined ? {} : { after }); + yield* page.items; + after = page.next; + } while (after !== undefined); + } + + async get(key: unknown, context: Context): Promise | undefined> { + this.owner.assertLive(); + const encoded = this.definition.rows.key(key, "get"); + const stored = await this.store.read( + (executor) => executor.get<{ [column: string]: SqliteValue }>(this.sql.get, ...encoded.params), + context, + ); + return stored === undefined ? undefined : this.decode(stored, context); + } + + async count(q: Query, context: Context): Promise { + this.owner.assertLive(); + this.#assertQuery(q); + const statement = this.sql.count(q.condition); + const row = await this.store.read( + (executor) => executor.get<{ readonly count: number }>(statement.sql, ...statement.params), + context, + ); + return row?.count ?? 0; + } + + /** A stored row as reads return it; one the code cannot read fails the owner and throws. */ + decode(stored: { readonly [column: string]: SqliteValue }, context: Context): RowOf { + try { + // The codec decodes by the definition's columns: a row of them. + return this.definition.rows.decode(stored, context) as RowOf; + } catch (error) { + const failure = + error instanceof ChordError + ? error + : new ChordError({ code: "unavailable", message: String(error) }, { cause: error }); + this.owner.fail(failure); + throw failure; + } + } + + /** A query of another table is refused (`invalid/args`). */ + #assertQuery(q: Query): void { + if (q.table !== this.definition) { + throw invalid( + "args", + `${this.owner.label}: a query of table "${q.table.name}" run on "${this.definition.name}"`, + ); + } + } +} diff --git a/packages/chord/src/next/store/table-history.ts b/packages/chord/src/next/store/table-history.ts new file mode 100644 index 000000000..639cbedd2 --- /dev/null +++ b/packages/chord/src/next/store/table-history.ts @@ -0,0 +1,98 @@ +// A table's history (slice 6a §9; decision "Second spec review: the majority"): the versions from a base +// `defineTable` through each `evolveTable`, the migration each version recorded, and what an activation does with the +// version the store recorded: create the table, use it, migrate it from there, or refuse it. + +import type { TableRecord } from "./catalog.ts"; +import type { ColumnRecord } from "./columns.ts"; +import type { SetNode, WhereNode } from "./expression.ts"; +import type { TableDefinition } from "./table.ts"; + +/** One edit of a migration as its writes record it (`alter`, `index`, `dropIndex`); the table is named when applied. */ +export type TableEdit = + | { readonly kind: "rename"; readonly from: string; readonly to: string } + | { readonly kind: "add"; readonly name: string; readonly column: ColumnRecord } + | { readonly kind: "drop"; readonly name: string } + | { readonly kind: "json"; readonly name: string; readonly column: ColumnRecord } + | { readonly kind: "index"; readonly name: string; readonly columns: readonly string[] } + | { readonly kind: "dropIndex"; readonly name: string }; + +/** The column edits an `alter` write records. */ +export type AlterEdit = Extract; + +/** + * A version's data step over `table`, the columns between the edits before it and after it: `rows` patches each row + * (synchronous and pure), `update` is one `updateWhere`. + */ +export type DataStep = + | { + readonly kind: "rows"; + readonly table: TableDefinition; + readonly patch: (row: { readonly [column: string]: unknown }) => unknown; + } + | { + readonly kind: "update"; + readonly table: TableDefinition; + readonly where: WhereNode | undefined; + readonly set: SetNode; + }; + +/** What one `evolveTable` changed: the edits before the data step, the step, the edits after it. */ +export type Migration = { + readonly before: readonly TableEdit[]; + readonly step: DataStep | undefined; + readonly after: readonly TableEdit[]; +}; + +/** + * What an activation does with a declared table: create it at the newest version, use it as recorded, or migrate it + * through `versions` (oldest first), the first one from its start (`begin`) or, recorded `migrating`, after its (A) + * commit (`resume`). + */ +export type TablePlan = + | { readonly kind: "create" } + | { readonly kind: "ready" } + | { + readonly kind: "migrate"; + readonly versions: readonly TableDefinition[]; + readonly first: "begin" | "resume"; + }; + +/** + * The plan for `definition` (the code's newest version) against `recorded`. A recorded version newer than the newest, + * older than the history's base, or the same version with another fingerprint (a history edited in place) throws: + * the object fails `broken`, the data stays. + */ +export function planTable(definition: TableDefinition, recorded: TableRecord | undefined, what: string): TablePlan { + if (recorded === undefined) return { kind: "create" }; + const { version, state, fingerprint } = recorded; + if (version > definition.version) { + throw new Error(`${what}: the store has version ${version}, newer than the code's ${definition.version}`); + } + const versions = historyOf(definition); + const at = versions.find((each) => each.version === version); + if (at === undefined) { + throw new Error( + `${what}: the store has version ${version}, older than the history's base ${versions[0]!.version}`, + ); + } + if (at.fingerprint !== fingerprint) { + throw new Error(`${what}: version ${version} is recorded with another definition (a history edited in place)`); + } + if (state === "ready" && version === definition.version) return { kind: "ready" }; + if (state === "migrating" && at.evolution?.migration.step === undefined) { + throw new Error(`${what}: version ${version} is recorded migrating, but has no data step to resume`); + } + if (state === "migrating") { + return { kind: "migrate", versions: versions.filter((each) => each.version >= version), first: "resume" }; + } + return { kind: "migrate", versions: versions.filter((each) => each.version > version), first: "begin" }; +} + +/** Every version of `definition`'s history, the base first. */ +function historyOf(definition: TableDefinition): TableDefinition[] { + const versions: TableDefinition[] = []; + for (let at: TableDefinition | undefined = definition; at !== undefined; at = at.evolution?.previous) { + versions.unshift(at); + } + return versions; +} diff --git a/packages/chord/src/next/store/table-migration.ts b/packages/chord/src/next/store/table-migration.ts new file mode 100644 index 000000000..8fef8fa9a --- /dev/null +++ b/packages/chord/src/next/store/table-migration.ts @@ -0,0 +1,245 @@ +// Declared tables brought to their version at activation (slice 6a §9): a table the store does not have is created at +// the newest version with its indexes (no replay); an older one is migrated eagerly, one version at a time, in its +// commits. Per version N: (A) the edits before the data step and `table { version: N, state: "migrating" }`; the data +// step's chunks; (Z) the last chunk (or the `update`) with the edits after the step and `state: "ready"`. Without a +// data step (A) and (Z) are one commit. A version recorded `migrating` resumes after (A), skipping the rows already +// stamped `$v >= N`. Its record holds the columns during the step but version N's fingerprint, not the fingerprint of +// those columns: the fingerprint names the version being reached, which resume, export and import check. + +import { type Context, withoutAbortSignal } from "../../context/index.ts"; +import type { SqliteValue } from "../../sqlite/index.ts"; +import type { Column } from "./columns.ts"; +import type { SetNode, WhereNode } from "./expression.ts"; +import { encodeCell, ROW_VERSION, type Scalar } from "./rows.ts"; +import { type Order, TableSql } from "./sql.ts"; +import type { StagedWrite } from "./step.ts"; +import type { ChordStore, CommitFence } from "./store.ts"; +import type { TableDefinition } from "./table.ts"; +import { type DataStep, planTable, type TablePlan } from "./table-history.ts"; +import { + jsonVersions, + recordedVersions, + stageEdit, + stageIndex, + stageTable, + stageUpdate, + stageUpdateWhere, +} from "./table-writes.ts"; + +/** A declared table: its newest definition and its SQL name. */ +export type DeclaredTable = { readonly definition: TableDefinition; readonly table: string }; + +/** Where a migration runs: the store, the table's SQL name and the commits' fence. */ +type MigrationTarget = { readonly store: ChordStore; readonly table: string; readonly fence: CommitFence }; + +/** + * Brings `tables` to their definitions: the ones the store does not have created in one commit, then each older one + * migrated. A table the code cannot use (see `planTable`) throws before anything is written; `what` names the owner. + */ +export async function prepareTables( + tables: readonly DeclaredTable[], + store: ChordStore, + fence: CommitFence, + what: string, + context: Context, +): Promise { + const created: StagedWrite[] = []; + const migrations: { readonly table: string; readonly plan: Extract }[] = []; + for (const { definition, table } of tables) { + const plan = planTable(definition, store.table(table), `${what}: table "${definition.name}"`); + if (plan.kind === "create") created.push(...creation(definition, table)); + else if (plan.kind === "migrate") migrations.push({ table, plan }); + } + if (created.length > 0) await commit(created, { store, fence }, context); + for (const { table, plan } of migrations) { + await migrateTable(plan, { store, table, fence }, context); + } +} + +/** The writes creating `definition` fresh as `table`: at its version, `ready`, with its indexes. */ +function creation(definition: TableDefinition, table: string): StagedWrite[] { + const { version, record: columns, key, fingerprint } = definition; + return [ + stageTable({ kind: "table", table, version, state: "ready", columns, key, fingerprint }), + ...[...definition.indexes].map(([name, columns]) => stageIndex({ kind: "index", table, name, columns })), + ]; +} + +/** + * Migrates the table through the plan's versions (oldest first, each with its migration), the first resumed after its + * (A) commit when the plan says so. `context`'s abort is checked between chunks (a chunk admitted to the step completes); a data + * step that throws, or returns a patch the version cannot take, rejects: the table stays `migrating` and the committed + * chunks stay. + */ +async function migrateTable( + plan: Extract, + target: MigrationTarget, + context: Context, +): Promise { + for (const [index, version] of plan.versions.entries()) { + await migrateVersion(version, index === 0 ? plan.first : "begin", target, context); + } +} + +/** The size of a `rows` chunk: one commit each. */ +export const MIGRATION_CHUNK = 100; + +/** Migrates to `definition`'s version from its (A) commit (`begin`) or after it (`resume`). */ +async function migrateVersion( + definition: TableDefinition, + from: "begin" | "resume", + target: MigrationTarget, + context: Context, +): Promise { + const { before, step, after } = definition.evolution!.migration; + const { table } = target; + const edits = (list: typeof before): StagedWrite[] => list.map((edit) => stageEdit(table, definition.version, edit)); + const ready = [...edits(after), recordOf(definition, definition, "ready", table)]; + if (step === undefined) return commit([...edits(before), ...ready], target, context); + if (from === "begin") + await commit([...edits(before), recordOf(step.table, definition, "migrating", table)], target, context); + if (step.kind === "update") return commit([updateStep(step, definition.version, table), ...ready], target, context); + await rowsStep(step, definition.version, ready, target, context); +} + +/** The `table` write of `table`'s columns and key, with `definition`'s version and fingerprint, in `state`. */ +function recordOf( + columns: TableDefinition, + definition: TableDefinition, + state: "ready" | "migrating", + table: string, +): StagedWrite { + const { version, fingerprint } = definition; + return stageTable({ kind: "table", table, version, state, columns: columns.record, key: columns.key, fingerprint }); +} + +/** The `update` step: one `updateWhere` over the rows below version `v`, stamping it, so it never applies twice. */ +function updateStep(step: Extract, v: number, table: string): StagedWrite { + const sql = sqlOf(step.table, table); + const below: WhereNode = { lt: [ROW_VERSION, v] }; + const where: WhereNode = step.where === undefined ? below : { and: [step.where, below] }; + const versions = jsonVersions(step.table, step.set); + const statement = sql.updateWhere(where, { ...step.set, [ROW_VERSION]: v }, versions); + return stageUpdateWhere(statement, table, where, step.set, recordedVersions(step.set, versions), v); +} + +/** + * The `rows` step: chunks of `MIGRATION_CHUNK` rows below version `v` in key order, paging forward by the last key; + * each patched row stamped `v`; the last chunk commits with `ready`. + */ +async function rowsStep( + step: Extract, + v: number, + ready: readonly StagedWrite[], + target: MigrationTarget, + context: Context, +): Promise { + const { table: columns } = step; + const sql = sqlOf(columns, target.table); + const order: Order = { columns: columns.key, direction: "asc" }; + const below: WhereNode = { lt: [ROW_VERSION, v] }; + let after: SqliteValue[] | undefined; + // The fallback yield between chunks (`macrotask`), made by the first yield that needs it, closed at the end. + let channel: InstanceType | undefined; + try { + for (;;) { + const statement = sql.select(below, order, after, MIGRATION_CHUNK + 1); + const stored = await target.store.read( + (executor) => executor.all<{ [column: string]: SqliteValue }>(statement.sql, ...statement.params), + context, + ); + const last = stored.length <= MIGRATION_CHUNK; + if (!last) stored.length = MIGRATION_CHUNK; + const writes = stored.map((row) => patchRow(step, row, v, sql, target.table, context)); + if (last) return await commit([...writes, ...ready], target, context); + await commit(writes, target, context); + after = columns.key.map((name) => stored[MIGRATION_CHUNK - 1]![name]!); + await macrotask(() => { + channel ??= new MessageChannel(); + return channel; + }); + // An abandoned activation stops here. + context.abortSignal?.throwIfAborted(); + } + } finally { + channel?.port1.close(); + } +} + +/** + * Resolves after one macrotask (other commits and timers run between chunks) without the 1 ms clamp of + * `setTimeout(0)`: `setImmediate` where the runtime has it (Node), else a message through the channel `channel()` + * gives (browsers, Workers). In Node a message costs 0.27 ms per yield against `setImmediate`'s 13 µs (the 10k-row + * migration bench: 100 ms against 70 ms). + */ +function macrotask(channel: () => InstanceType): Promise { + return new Promise((resolve) => { + // A runtime feature check: Workers and browsers have no `setImmediate`. + if ("setImmediate" in globalThis) { + setImmediate(resolve); + return; + } + const { port1, port2 } = channel(); + port1.addEventListener("message", () => resolve(), { once: true }); + // A port listened to with `addEventListener` delivers only once started (browsers, Workers). + port1.start(); + port2.postMessage(null); + }); +} + +/** + * The write of one row's patch: `patch` run on the decoded row, its result checked (an object of non-key columns of + * the version, values of their types) and stamped `v` through the internal form of `update`. + */ +function patchRow( + step: Extract, + stored: { readonly [column: string]: SqliteValue }, + v: number, + sql: TableSql, + table: string, + context: Context, +): StagedWrite { + const { table: definition } = step; + const patch = step.patch(definition.rows.decode(stored, context)); + const what = `${definition.name} version ${v}: rows`; + if (typeof patch !== "object" || patch === null || Array.isArray(patch)) { + throw new Error(`${what}: expected an object of column values`); + } + const set: { [column: string]: SetNode[string] } = {}; + for (const [name, value] of Object.entries(patch)) { + const column = definition.byName.get(name); + if (column === undefined) throw new Error(`${what}: no column "${name}"`); + if (definition.key.includes(name)) throw new Error(`${what}: key column "${name}" cannot change`); + set[name] = cellOf(column, value, `${what}: column "${name}"`); + } + const key: { [column: string]: Scalar } = {}; + // Key columns are non-opt `str` or `int`: stored scalars. + for (const name of definition.key) key[name] = stored[name] as Scalar; + const versions = jsonVersions(definition, set); + const params = definition.key.map((name) => stored[name]!); + const statement = sql.update({ ...set, [ROW_VERSION]: v }, params, versions); + return stageUpdate(statement, table, key, set, recordedVersions(set, versions), v); +} + +/** + * A patch value as a `set` node: `undefined` clears an `opt(...)` column, `json(...)` cells as JSON literals (a JSON + * `null` too); a scalar `null` is refused by `encodeCell`. + */ +function cellOf(column: Column, value: unknown, what: string): SetNode[string] { + if (value === undefined) { + if (!column.optional) throw new Error(`${what} is not opt(...) and cannot be cleared`); + return null; + } + const cell = encodeCell(column, value, what); + // `encodeCell` returns a scalar for a scalar column. + return column.type === "json" ? { json: cell.stored } : (cell.stored as Scalar); +} + +function sqlOf(definition: TableDefinition, table: string): TableSql { + return new TableSql(table, definition.columns, definition.rows.physical, definition.key); +} + +/** Commits `writes` without the activation's abort: a commit admitted to the step completes (the fence refuses one). */ +async function commit(writes: StagedWrite[], target: Omit, context: Context): Promise { + await target.store.commit(writes, target.fence, withoutAbortSignal(context)); +} diff --git a/packages/chord/src/next/store/table-writes.ts b/packages/chord/src/next/store/table-writes.ts new file mode 100644 index 000000000..101c7f8b3 --- /dev/null +++ b/packages/chord/src/next/store/table-writes.ts @@ -0,0 +1,260 @@ +// Table writes (slice 6a §5, §7, §9; S17; decisions "Writes carry their versions", "Index definitions are recorded"): +// the recorded form of every table write, the staged writes a transaction or an activation hands the commit step +// (their SQL rendered at staging, rule 8), and the appliers of the schema writes, which render DDL from the record. + +import type { TableRecord } from "./catalog.ts"; +import type { ColumnRecord } from "./columns.ts"; +import type { SetNode, WhereNode } from "./expression.ts"; +import type { EncodedRow, Scalar, StoredRow } from "./rows.ts"; +import { versionColumn } from "./rows.ts"; +import { columnSql, createIndexSql, createTableSql, dropIndexSql, quote, type Statement } from "./sql.ts"; +import type { StagedWrite, Step } from "./step.ts"; +import type { TableDefinition } from "./table.ts"; +import type { AlterEdit, TableEdit } from "./table-history.ts"; + +/** A key as writes record it: the key columns' stored values. */ +export type StoredKey = { readonly [column: string]: Scalar }; + +/** `json(...)` columns' versions written by a write, by column. */ +export type ColumnVersions = { readonly [column: string]: number }; + +/** The table writes as the outbox and archives record them. */ +export type TableWrite = + | { + readonly kind: "table"; + readonly table: string; + readonly version: number; + readonly state: "ready" | "migrating"; + readonly columns: { readonly [name: string]: ColumnRecord }; + readonly key: readonly string[]; + readonly fingerprint: string; + } + | { readonly kind: "index"; readonly table: string; readonly name: string; readonly columns: readonly string[] } + | { readonly kind: "dropIndex"; readonly table: string; readonly name: string } + /** A column edit of the migration to `version`; DDL rendered from it. */ + | { readonly kind: "alter"; readonly table: string; readonly version: number; readonly edit: AlterEdit } + | { + readonly kind: "put"; + readonly table: string; + readonly row: StoredRow; + /** The row's eager-step version (the table's version at the `put`). */ + readonly version: number; + readonly columns: ColumnVersions; + } + | { + readonly kind: "update"; + readonly table: string; + readonly key: StoredKey; + readonly set: SetNode; + readonly columns: ColumnVersions; + readonly count: number; + /** A migration's data step: the version it stamps as the row's `$v` (a plugin's `update` never does). */ + readonly version?: number; + } + | { readonly kind: "delete"; readonly table: string; readonly key: StoredKey; readonly count: number } + | { + readonly kind: "updateWhere"; + readonly table: string; + readonly where: WhereNode; + readonly set: SetNode; + readonly columns: ColumnVersions; + readonly count: number; + /** A migration's `update` step: the version it stamps as the rows' `$v`. */ + readonly version?: number; + } + | { readonly kind: "deleteWhere"; readonly table: string; readonly where: WhereNode; readonly count: number }; + +/** The versions of the `json(...)` columns of `definition` that `set` writes, which its statement stamps. */ +export function jsonVersions(definition: TableDefinition, set: SetNode): ReadonlyMap { + const versions = new Map(); + for (const name of Object.keys(set)) { + const column = definition.byName.get(name)!; + if (column.type === "json") versions.set(name, column.json.version); + } + return versions; +} + +/** The `columns` a write records: the versions of the `json(...)` columns it sets to a value (a clear has none). */ +export function recordedVersions(set: SetNode, versions: ReadonlyMap): ColumnVersions { + const columns: { [column: string]: number } = {}; + for (const [column, version] of versions) if (set[column] !== null) columns[column] = version; + return columns; +} + +/** The `table` write. */ +export type TableDefinitionWrite = Extract; + +/** Stages a `table` write: the table created when the store does not have it yet, its record replaced otherwise. */ +export function stageTable(write: TableDefinitionWrite): StagedWrite { + return { order: "write", apply: (step) => applyTable(step, write) }; +} + +/** Stages an `index` write. */ +export function stageIndex(write: Extract): StagedWrite { + return { order: "write", apply: (step) => applyIndex(step, write) }; +} + +/** Stages one edit of the migration of `table` to `version` as its `alter`, `index` or `dropIndex` write. */ +export function stageEdit(table: string, version: number, edit: TableEdit): StagedWrite { + switch (edit.kind) { + case "index": + return stageIndex({ kind: "index", table, name: edit.name, columns: edit.columns }); + case "dropIndex": { + const write = { kind: "dropIndex", table, name: edit.name } as const; + return { order: "write", apply: (step) => applyDropIndex(step, write) }; + } + default: { + const write = { kind: "alter", table, version, edit } as const; + return { order: "write", apply: (step) => applyAlter(step, write) }; + } + } +} + +/** Stages a `put` of an encoded row with its `INSERT OR REPLACE` (an upsert: the whole row replaced). */ +export function stagePut(sql: string, table: string, row: EncodedRow, version: number): StagedWrite { + const write: TableWrite = { kind: "put", table, row: row.row, version, columns: row.columns }; + return { + order: "write", + apply: async (step) => { + await step.run(sql, ...row.params); + return write; + }, + }; +} + +/** + * Stages an `update` of one row by key; its count (0 for an absent key) is recorded, and `version` when the statement stamps + * the row's `$v` (a migration's `rows` step). + */ +export function stageUpdate( + statement: Statement, + table: string, + key: StoredKey, + set: SetNode, + columns: ColumnVersions, + version?: number, +): StagedWrite { + const stamped = version === undefined ? {} : { version }; + return counted(statement, (count) => ({ kind: "update", table, key, set, columns, count, ...stamped })); +} + +/** Stages a `delete` of one row by key; its count is recorded. */ +export function stageDelete(statement: Statement, table: string, key: StoredKey): StagedWrite { + return counted(statement, (count) => ({ kind: "delete", table, key, count })); +} + +/** + * Stages an `updateWhere`: one statement, its count recorded (a replica affecting another count stops, S17), and `version` + * when it stamps the rows' `$v` (a migration's `update` step). + */ +export function stageUpdateWhere( + statement: Statement, + table: string, + where: WhereNode, + set: SetNode, + columns: ColumnVersions, + version?: number, +): StagedWrite { + const stamped = version === undefined ? {} : { version }; + return counted(statement, (count) => ({ kind: "updateWhere", table, where, set, columns, count, ...stamped })); +} + +/** Stages a `deleteWhere`: one statement, its count recorded. */ +export function stageDeleteWhere(statement: Statement, table: string, where: WhereNode): StagedWrite { + return counted(statement, (count) => ({ kind: "deleteWhere", table, where, count })); +} + +function counted(statement: Statement, record: (count: number) => TableWrite): StagedWrite { + return { + order: "write", + apply: async (step) => record(await step.runCounted(statement.sql, ...statement.params)), + }; +} + +/** Applies a `table` write: `CREATE TABLE` when the store does not have the table yet, then its record. */ +export async function applyTable(step: Step, write: TableDefinitionWrite): Promise { + const existing = step.table(write.table); + if (existing === undefined) await step.run(createTableSql(write.table, write.columns, write.key)); + const { table, version, state, columns, key, fingerprint } = write; + step.tables.set(table, { table, version, state, columns, key, fingerprint, indexes: existing?.indexes ?? {} }); + return write; +} + +/** Applies an `index` write: `CREATE INDEX` and the record's index. */ +export async function applyIndex( + step: Step, + write: Extract, +): Promise { + const existing = recorded(step, write.table); + await step.run(createIndexSql(write.table, write.name, write.columns, existing.key)); + step.tables.set(write.table, { ...existing, indexes: { ...existing.indexes, [write.name]: write.columns } }); + return write; +} + +/** Applies a `dropIndex` write: `DROP INDEX` and the record without it. */ +export async function applyDropIndex( + step: Step, + write: Extract, +): Promise { + const existing = recorded(step, write.table); + await step.run(dropIndexSql(write.table, write.name)); + const { [write.name]: _dropped, ...indexes } = existing.indexes; + step.tables.set(write.table, { ...existing, indexes }); + return write; +} + +/** Applies an `alter` write: its DDL (a `json(...)` column's version column follows it) and the record's change. */ +export async function applyAlter( + step: Step, + write: Extract, +): Promise { + const existing = recorded(step, write.table); + const { edit } = write; + const id = quote(write.table); + switch (edit.kind) { + case "rename": { + const json = existing.columns[edit.from]?.type === "json"; + await step.run(`ALTER TABLE ${id} RENAME COLUMN ${quote(edit.from)} TO ${quote(edit.to)}`); + if (json) { + const [from, to] = [versionColumn(edit.from), versionColumn(edit.to)]; + await step.run(`ALTER TABLE ${id} RENAME COLUMN ${quote(from)} TO ${quote(to)}`); + } + const rename = (column: string): string => (column === edit.from ? edit.to : column); + const columns = Object.entries(existing.columns).map(([name, column]) => [rename(name), column] as const); + const indexes = Object.entries(existing.indexes).map(([name, on]) => [name, on.map(rename)] as const); + step.tables.set(write.table, { + ...existing, + columns: Object.fromEntries(columns), + key: existing.key.map(rename), + indexes: Object.fromEntries(indexes), + }); + break; + } + case "add": + for (const definition of columnSql(edit.name, edit.column)) { + await step.run(`ALTER TABLE ${id} ADD COLUMN ${definition}`); + } + step.tables.set(write.table, { ...existing, columns: { ...existing.columns, [edit.name]: edit.column } }); + break; + case "drop": { + const json = existing.columns[edit.name]?.type === "json"; + await step.run(`ALTER TABLE ${id} DROP COLUMN ${quote(edit.name)}`); + if (json) await step.run(`ALTER TABLE ${id} DROP COLUMN ${quote(versionColumn(edit.name))}`); + const { [edit.name]: _dropped, ...columns } = existing.columns; + step.tables.set(write.table, { ...existing, columns }); + break; + } + case "json": + step.tables.set(write.table, { ...existing, columns: { ...existing.columns, [edit.name]: edit.column } }); + break; + } + return write; +} + +/** The record of `table` in `step`; a write for a table the store does not have fails the step. */ +function recorded(step: Step, table: string): TableRecord { + const record = step.table(table); + // Memory and disk disagree: the step fails, which stops the store. + if (record === undefined) throw new Error(`table ${table}: not in the store`); + return record; +} diff --git a/packages/chord/src/next/store/table.ts b/packages/chord/src/next/store/table.ts new file mode 100644 index 000000000..d86d2fa16 --- /dev/null +++ b/packages/chord/src/next/store/table.ts @@ -0,0 +1,157 @@ +// Table definitions (slice 6a §5, §9; decisions "Flat columns", "Restricted names", "Index definitions are recorded"): +// `defineTable(name, { version, columns, key, indexes? })`, checked and materialized once at declaration: its columns, +// its key, its indexes, its fingerprint and the compiled row codec its writes and reads use. A version `evolveTable` +// appends links to the one before it and to the migration between them. + +import { checkName, hash64 } from "../contracts/internal.ts"; +import { + type Column, + type ColumnRecord, + type Columns, + columnRecord, + describeColumns, + type OrderColumn, +} from "./columns.ts"; +import { type ColumnRef, columnRefs } from "./expression.ts"; +import { RowCodec } from "./rows.ts"; +import type { Migration } from "./table-history.ts"; + +/** What `defineTable` takes. */ +export type TableOptions[], V extends number = number> = { + /** The table's version: a positive integer, raised by every change (`evolveTable`). */ + readonly version: V; + readonly columns: C; + /** The primary key's columns, non-`opt` `str` or `int` ones. */ + readonly key: K; + /** Secondary indexes by name (`[a-z0-9_]+`); each ends with the primary key. */ + readonly indexes?: { readonly [name: string]: readonly OrderColumn[] }; +}; + +/** The key of a row of a table with columns `C` and key `K`: an object with exactly the key columns. */ +export type KeyOf = { + readonly [P in K[number]]: P extends keyof C + ? C[P] extends { readonly "~kind": "Integer" } + ? number + : string + : never; +}; + +/** + * Declares a table: a base version written in full, the first of its history. Later versions come from + * `evolveTable`; a new base starts a new history. + */ +export function defineTable[], const V extends number>( + name: string, + options: TableOptions, +): TableDefinition { + return new TableDefinition(name, options); +} + +/** The table form's version: the fingerprints of table definitions are persisted, so their input is versioned. */ +export const TABLE_FORM_VERSION = 1; + +/** The index name rule. */ +export const INDEX_NAME = /^[a-z0-9_]+$/; + +/** How a version came about: the version before it and the migration from there, for one `evolveTable`. */ +export type Evolution = { readonly previous: TableDefinition; readonly migration: Migration }; + +/** + * A declared table at one version: its columns in declaration order, key, indexes (declared columns; the key is + * appended when they are rendered), the fingerprint of all of it and the row codec. `C` and `K` type its rows and keys, + * `V` its version. + */ +export class TableDefinition< + C extends Columns = Columns, + K extends readonly string[] = readonly string[], + V extends number = number, +> { + readonly name: string; + readonly version: V; + /** The column schemas as declared, which `evolveTable` edits into the next version's. */ + readonly schemas: Columns; + readonly columns: readonly Column[]; + readonly byName: ReadonlyMap; + readonly key: readonly string[]; + readonly indexes: ReadonlyMap; + /** `hash64` of the columns, key and indexes in `TABLE_FORM_VERSION`'s form. */ + readonly fingerprint: string; + /** The columns as the `table` write records them. */ + readonly record: { readonly [name: string]: ColumnRecord }; + readonly rows: RowCodec; + /** One reference per column, for the `(t) => …` callbacks of queries and writes. */ + readonly refs: { readonly [column: string]: ColumnRef }; + /** The version before this one and the migration from it; `undefined` for a base. */ + readonly evolution: Evolution | undefined; + /** Type only: the columns and key. */ + declare readonly "~columns"?: C; + declare readonly "~key"?: K; + + constructor(name: string, options: TableOptions[], V>, evolution?: Evolution) { + checkName(name, "table"); + const { version, columns, key, indexes = {} } = options; + // Declaration policy: JavaScript callers reach these checks. + if (!Number.isSafeInteger(version) || version < 1) + throw new Error(`${name}: version ${version} is not a positive integer`); + this.name = name; + this.version = version; + this.schemas = columns; + this.columns = describeColumns(columns, name); + this.byName = new Map(this.columns.map((column) => [column.name, column])); + this.key = orderColumns(this.byName, key, `${name}: the key`); + if (this.key.length === 0) throw new Error(`${name}: the key has no column`); + this.indexes = new Map( + Object.entries(indexes).map(([index, columns]) => { + if (!INDEX_NAME.test(index)) throw new Error(`${name}: index "${index}": expected [a-z0-9_]+`); + return [index, orderColumns(this.byName, columns, `${name}: index "${index}"`)] as const; + }), + ); + this.record = Object.fromEntries(this.columns.map((column) => [column.name, columnRecord(column)])); + this.fingerprint = tableFingerprint(this.record, this.key, this.indexes); + this.rows = new RowCodec(this.columns, this.key, name); + this.refs = columnRefs(this.columns); + this.evolution = evolution; + } + + /** The column `name`; an unknown one throws (`what` names the use). */ + column(name: string, what: string): Column { + const column = this.byName.get(name); + if (column === undefined) throw new Error(`${this.name}: ${what}: no column "${name}"`); + return column; + } +} + +/** + * Checks `names` as key, index or order columns of a table whose columns are `byName`: existing, non-`opt` `str` or + * `int`, no column twice (declaration policy: JavaScript callers reach it); `what` names the use in errors. + */ +export function orderColumns( + byName: ReadonlyMap, + names: readonly string[], + what: string, +): readonly string[] { + if (!Array.isArray(names)) throw new Error(`${what}: expected an array of column names`); + if (new Set(names).size !== names.length) throw new Error(`${what}: a column occurs twice`); + for (const name of names) { + const column = byName.get(name); + if (column === undefined) throw new Error(`${what}: no column "${name}"`); + if (column.optional || (column.type !== "str" && column.type !== "int")) { + throw new Error(`${what}: column "${name}" is not a non-opt str() or int() column`); + } + } + return [...names]; +} + +/** The fingerprint of a table's columns, key and indexes; column and index names sorted (their order is no change). */ +export function tableFingerprint( + columns: { readonly [name: string]: ColumnRecord }, + key: readonly string[], + indexes: ReadonlyMap, +): string { + const value = { columns: sorted(Object.entries(columns)), key, indexes: sorted([...indexes]) }; + return hash64(JSON.stringify({ tableFormVersion: TABLE_FORM_VERSION, value })); +} + +function sorted(entries: [string, T][]): { [name: string]: T } { + return Object.fromEntries(entries.sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0))); +} diff --git a/packages/chord/src/next/store/text.ts b/packages/chord/src/next/store/text.ts new file mode 100644 index 000000000..e526e27b9 --- /dev/null +++ b/packages/chord/src/next/store/text.ts @@ -0,0 +1,51 @@ +// Text in the store (decision "Text is plain UTF-8, in code point order"): Chord compares strings by code point, as +// SQLite's BINARY collation compares their UTF-8 bytes, never by JavaScript's UTF-16 code units; every stored string +// is well-formed and free of NUL, and a prefix becomes a range by its upper bound. + +import { invalid } from "../data/internal.ts"; + +/** Compares `a` and `b` by code point: negative, zero or positive. */ +export function compareText(a: string, b: string): number { + const length = Math.min(a.length, b.length); + for (let i = 0; i < length; i++) { + const x = a.charCodeAt(i); + const y = b.charCodeAt(i); + if (x !== y) return codePointRank(x) - codePointRank(y); + } + return a.length - b.length; +} + +/** + * A UTF-16 code unit's rank in code point order: a surrogate starts a code point above U+FFFF, so it ranks above every + * unit from U+E000 on. + */ +function codePointRank(unit: number): number { + if (unit < 0xd800) return unit; + return unit < 0xe000 ? unit + 0x2000 : unit - 0x800; +} + +/** + * `text` as the store keeps it: a lone surrogate replaced by U+FFFD (copied only when one occurs), NUL refused + * `invalid/args` (`what` names the value). Plugin strings are a trust boundary. + */ +export function storedText(text: string, what: string): string { + if (text.includes("\0")) throw invalid("args", `${what}: text may not contain NUL`); + return text.isWellFormed() ? text : text.toWellFormed(); +} + +/** + * The smallest string above every string starting with `prefix`, in code point order: the next code point after its + * last, carrying past U+10FFFF; `undefined` when there is none (`""` and all-U+10FFFF prefixes). `prefix` is + * well-formed. + */ +export function prefixUpperBound(prefix: string): string | undefined { + const points = [...prefix]; + while (points.length > 0) { + const last = points.pop()!.codePointAt(0)!; + if (last === 0x10ffff) continue; + // The surrogates are no code points of well-formed text: after U+D7FF comes U+E000. + const next = last === 0xd7ff ? 0xe000 : last + 1; + return points.join("") + String.fromCodePoint(next); + } + return undefined; +} diff --git a/packages/chord/src/next/store/writes.ts b/packages/chord/src/next/store/writes.ts new file mode 100644 index 000000000..8b10b8797 --- /dev/null +++ b/packages/chord/src/next/store/writes.ts @@ -0,0 +1,8 @@ +// The store's write vocabulary (`StoreWrite`, slice 6a §7, §8; decision "Writes carry their versions"): one change of +// the store each, as the outbox and archives record it. + +import type { DocumentWrite } from "./engine.ts"; +import type { TableWrite } from "./table-writes.ts"; + +/** One change of the store, tagged by `kind`. */ +export type StoreWrite = DocumentWrite | TableWrite; diff --git a/packages/chord/test/next-node/compression.test.ts b/packages/chord/test/next-node/compression.test.ts new file mode 100644 index 000000000..00f50adf4 --- /dev/null +++ b/packages/chord/test/next-node/compression.test.ts @@ -0,0 +1,39 @@ +// Tests of nodeCompression: raw deflate and the bounded inflate with its tagged outcomes, a limit of 0 included. + +import { describe, expect, it } from "vitest"; +import type { Inflated } from "../../src/next/link/part.ts"; +import { nodeCompression } from "../../src/next-node/compression.ts"; + +const UTF8 = new TextEncoder(); + +const bytesOf = (inflated: Inflated) => (inflated.kind === "bytes" ? new Uint8Array(inflated.bytes) : inflated); + +describe("nodeCompression", () => { + it("deflates raw at level 1 and inflates back", () => { + const bytes = UTF8.encode(JSON.stringify({ text: "lorem ipsum ".repeat(5000) })); + const deflated = nodeCompression.deflate(bytes); + expect(deflated.byteLength).toBeLessThan(bytes.byteLength / 10); + expect(bytesOf(nodeCompression.inflate(deflated, bytes.byteLength))).toEqual(bytes); + expect(bytesOf(nodeCompression.inflate(deflated, Number.POSITIVE_INFINITY))).toEqual(bytes); + }); + + it("gives too_large past the limit, without inflating the rest (a bomb)", () => { + const bomb = nodeCompression.deflate(new Uint8Array(256 * 1024 * 1024)); + expect(nodeCompression.inflate(bomb, 1024 * 1024)).toEqual({ kind: "too_large" }); + }); + + it("takes a limit of 0: an empty stream inflates, anything else is too_large (never zlib's range error)", () => { + expect(bytesOf(nodeCompression.inflate(nodeCompression.deflate(new Uint8Array(0)), 0))).toEqual( + new Uint8Array(0), + ); + expect(nodeCompression.inflate(nodeCompression.deflate(new Uint8Array(1)), 0)).toEqual({ kind: "too_large" }); + expect(nodeCompression.inflate(nodeCompression.deflate(new Uint8Array(100)), 0)).toEqual({ kind: "too_large" }); + expect(bytesOf(nodeCompression.inflate(nodeCompression.deflate(new Uint8Array(1)), 1))).toEqual( + new Uint8Array(1), + ); + }); + + it("gives corrupt for data that is no raw deflate stream", () => { + expect(nodeCompression.inflate(new Uint8Array([255, 1, 2, 3]), 100)).toMatchObject({ kind: "corrupt" }); + }); +}); diff --git a/packages/chord/test/next/bench/compression.ts b/packages/chord/test/next/bench/compression.ts new file mode 100644 index 000000000..7b74ad8f8 --- /dev/null +++ b/packages/chord/test/next/bench/compression.ts @@ -0,0 +1,155 @@ +// Light benchmark of compression at the origin (slice 4b): a 500 KB snapshot and a 1 MiB call result, uncompressed and +// deflated (level 1) by the server; their bytes on the wire, the deflate at the origin, the inflate at the owner +// (synchronous `nodeCompression`, asynchronous `DecompressionStream`), and end to end over a memory link. +// Run by hand from packages/chord (not in CI): +// node --conditions=source --experimental-strip-types --expose-gc test/next/bench/compression.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { type Compression, type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { DeflatedPart, inflateAsync } from "../../../src/next/link/internal.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "owner" } | { kind: "server" }; + +const Entry = obj({ id: int(), text: str(), author: str(), at: int() }); + +const Store = local("bench.store", { version: 1, params: {}, members: { doc: obj({ entries: arr(Entry) }) } }); + +/** A tab's view of the 500 KB document, and a method returning 1 MiB. */ +const View = remote("bench.view", { + version: 1, + params: {}, + variants: { owner: { state: { doc: obj({ entries: arr(Entry) }) }, methods: { export: method(undefined, str()) } } }, +}); + +/** Words that make text compress like prose and code (about 3x at level 1), not like a repeated string. */ +const WORDS = "the a model tool call file read write test error result agent session note code run build".split(" "); + +/** About `length` characters of words, from a fixed pseudo-random sequence starting at `seed`. */ +function prose(seed: number, length: number): string { + let state = seed * 7919 + 1; + const words: string[] = []; + for (let total = 0; total < length; total += words.at(-1)!.length + 1) { + state = (state * 1103515245 + 12345) % 2147483648; + const word = WORDS[state % WORDS.length]!; + words.push(state % 5 === 0 ? `${word}${state % 1000}` : word); + } + return words.join(" ").slice(0, length); +} + +/** The 500 KB document: 500 entries of 960 characters, as in the slice 3a benchmark, with prose-like text. */ +const entries = Array.from({ length: 500 }, (_, id) => ({ + id, + text: prose(id, 960), + author: "mario", + at: 1_700_000_000 + id, +})); + +/** The 1 MiB result. */ +const RESULT = prose(1, 1024 * 1024); + +const Doc = defineState("doc", { schema: obj({ entries: arr(Entry) }), version: 1, initial: () => ({ entries }) }); + +const facet = defineFacet

((host) => { + host.serve(Store, (_, self) => ({ doc: self.state(Doc) })); + host.serve(View, (_, _principal, self) => ({ + variant: "owner", + members: { doc: self.get(Store, {}).doc, export: async () => RESULT }, + })); +}); + +/** A link end counting the bytes its host receives. */ +function counted(link: Link, bytes: { received: number }): Link { + return { + send: (message) => link.send(message), + onMessage: (listener) => + link.onMessage((message) => { + bytes.received += message.byteLength; + listener(message); + }), + close: () => link.close(), + onClose: (listener) => link.onClose(listener), + }; +} + +/** The mean milliseconds of `runs` runs of `work`, the fastest of three such loops. */ +async function mean(runs: number, work: () => unknown): Promise { + let best = Number.POSITIVE_INFINITY; + for (let loop = 0; loop < 3; loop++) { + const started = performance.now(); + for (let run = 0; run < runs; run++) await work(); + best = Math.min(best, (performance.now() - started) / runs); + } + return best; +} + +const UTF8 = new TextEncoder(); + +const kib = (bytes: number) => `${(bytes / 1024).toFixed(1)} KiB`; + +const ms = (value: number) => `${value.toFixed(3)} ms`; + +console.log( + "| Payload | JSON | deflated (level 1) | deflate at the origin | inflate, sync (`nodeCompression`) | inflate, async (`DecompressionStream`) |", +); +console.log("| --- | --- | --- | --- | --- | --- |"); +for (const [name, json] of [ + ["500 KB snapshot", JSON.stringify({ entries })], + ["1 MiB result", JSON.stringify(RESULT)], +] as const) { + const bytes = UTF8.encode(json); + const deflated = nodeCompression.deflate(bytes); + const deflate = await mean(20, () => nodeCompression.deflate(bytes)); + const inflate = await mean(20, () => nodeCompression.inflate(deflated, Number.POSITIVE_INFINITY)); + const inflateAsyncMs = await mean(20, () => + inflateAsync(new DeflatedPart(deflated), Number.POSITIVE_INFINITY, new AbortController().signal, context), + ); + console.log( + `| ${name} | ${kib(bytes.byteLength)} | ${kib(deflated.byteLength)} | ${ms(deflate)} | ${ms(inflate)} | ${ms(inflateAsyncMs)} |`, + ); +} + +/** A server compressing with `server` and a client with `client` (none: it inflates asynchronously). */ +async function linked(server: Compression | undefined, client: Compression | undefined) { + const serverHost = createHost

({ principal: { kind: "server" }, compression: server, onError: console.error }); + await serverHost.load([{ name: "@bench/app", facet }], context); + const clientHost = createHost

({ compression: client, onError: console.error }); + await clientHost.load([], context); + const [a, b] = memoryLinkPair(); + const bytes = { received: 0 }; + serverHost.accept(a, { principal: { kind: "owner" } }, context); + clientHost.connect(counted(b, bytes), { principal: { kind: "server" } }, context); + return { server: serverHost, client: clientHost, bytes }; +} + +console.log(); +console.log( + "| End to end over a memory link (binary codec) | snapshot: bytes received | hold to live (mean of 20) | 1 MiB result: bytes received | call to result (mean of 20) |", +); +console.log("| --- | --- | --- | --- | --- |"); +for (const [label, server, client] of [ + ["uncompressed", undefined, undefined], + ["deflated, inflated synchronously (Node client)", nodeCompression, nodeCompression], + ["deflated, inflated asynchronously (browser client)", nodeCompression, undefined], +] as const) { + const hosts = await linked(server, client); + // Warm up: the store and the view activate once. + (await hosts.client.get(View, {}, context)).release(); + const before = hosts.bytes.received; + const view = await hosts.client.get(View, {}, context); + const snapshotBytes = hosts.bytes.received - before; + view.release(); + const hold = await mean(20, async () => (await hosts.client.get(View, {}, context)).release()); + const held = await hosts.client.get(View, {}, context); + const beforeCall = hosts.bytes.received; + await held.export(context); + const resultBytes = hosts.bytes.received - beforeCall; + const call = await mean(20, () => held.export(context)); + console.log(`| ${label} | ${kib(snapshotBytes)} | ${ms(hold)} | ${kib(resultBytes)} | ${ms(call)} |`); + await hosts.client.close(context); + await hosts.server.close(context); +} diff --git a/packages/chord/test/next/bench/derives.ts b/packages/chord/test/next/bench/derives.ts new file mode 100644 index 000000000..7c2785173 --- /dev/null +++ b/packages/chord/test/next/bench/derives.ts @@ -0,0 +1,73 @@ +// Benchmark of a retirement through derives (slice 6a rule 21): a singleton under a diamond of 40 derives (13 layers of +// two derives joined by a third, 2^13 paths from the top to the bottom), its bottom watched; each step retires the +// document and creates it again in one transaction, so every step carries the `retired` flag through the diamond. +// Median and p95 per step over several bursts. The flag reaches each derive once per retirement, not once per path. +// Run by hand from packages/chord (not in CI): +// node --conditions=source --experimental-strip-types --expose-gc test/next/bench/derives.ts +import { performance } from "node:perf_hooks"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Self } from "../../../src/next/host/index.ts"; +import type { Watchable } from "../../../src/watch.ts"; + +const STEPS = 1_000; +const RUNS = 7; +const LAYERS = 13; + +const List = defineState("bench.list", { + schema: obj({ items: arr(str()) }), + version: 1, + initial: () => ({ items: [] }), +}); +const Diamond = local("bench.diamond", { version: 1, params: {}, members: { bottom: int(), touch: method() } }); + +/** The diamond over `list`: its bottom derive. */ +function diamond(self: Self, list: SingletonHandle<{ items: string[] }>): Watchable { + let previous: Watchable = self.derive([list], ([value]) => value.items.length); + for (let layer = 0; layer < LAYERS; layer++) { + const left = self.derive([previous], ([n]) => n + 1); + const right = self.derive([previous], ([n]) => n + 2); + previous = self.derive([left, right], ([a, b]) => (a + b) % 1_000); + } + return previous; +} + +/** Per-step times (ms) of `RUNS` bursts of `STEPS` retire-and-recreate transactions, after one warm-up burst. */ +async function bursts(): Promise<{ readonly median: number; readonly p95: number }> { + const times: number[] = []; + for (let run = -1; run < RUNS; run++) { + let list: SingletonHandle<{ items: string[] }> | undefined; + let self: Self | undefined; + const host = createHost({ onError: () => {} }); + const facet = defineFacet((h) => + h.serve(Diamond, (_, declared) => { + self = declared; + list = declared.state(List); + return { bottom: diamond(declared, list), touch: async () => null }; + }), + ); + await host.load([{ name: "@bench/derives", facet }], context); + const reference = await host.get(Diamond, {}, context); + const watch = await reference.bottom.watch(context); + watch.start(async () => {}); + for (let step = 0; step < STEPS; step++) { + const before = performance.now(); + await self!.tx(async (tx) => { + await tx.retire(list!); + (await tx.draft(list!)).items.push(`${step}`); + }, context); + if (run >= 0) times.push(performance.now() - before); + } + await watch.stop(); + reference.release(); + await host.close(context); + } + times.sort((a, b) => a - b); + return { median: times[Math.floor(times.length / 2)]!, p95: times[Math.floor(times.length * 0.95)]! }; +} + +const { median, p95 } = await bursts(); +console.log( + `retire + recreate under a ${1 + 3 * LAYERS}-derive diamond: ${(median * 1000).toFixed(1)} µs median, ${(p95 * 1000).toFixed(1)} µs p95`, +); diff --git a/packages/chord/test/next/bench/links.ts b/packages/chord/test/next/bench/links.ts new file mode 100644 index 000000000..3bbbf6d31 --- /dev/null +++ b/packages/chord/test/next/bench/links.ts @@ -0,0 +1,469 @@ +// Light benchmarks of remote objects over one memory link (slices 3a, 3b), per codec side by side: wire bytes per +// message; compute (encode and decode time per message, commit-to-frame end to end); memory (heap allocated per message +// while encoding and decoding, retained per hold); drops (3b: reconnects re-holding 1 and 100 references, 1,000 calls +// parked while the link is down). Payloads are modelled on p2: a notes list, a 500 KB conversation document, a reply +// streamed token by token, calls with and without bytes. +// Run by hand from packages/chord (not in CI): +// node --conditions=source --experimental-strip-types --expose-gc test/next/bench/links.ts +import { GCProfiler } from "node:v8"; +import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; +import { BACKGROUND_CONTEXT, type Context, withTelemetry } from "../../../src/context/index.ts"; +import { arr, bytes, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host, type RefState } from "../../../src/next/host/index.ts"; +import { type Codec, type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { binaryCodec, type Message, textCodec } from "../../../src/next/link/internal.ts"; +import type { NativeWatchable } from "../../../src/watch.ts"; + +type P = { kind: "owner" } | { kind: "server" }; + +const Entry = obj({ id: int(), text: str(), author: str(), at: int() }); +const Shape = obj({ entries: arr(Entry) }); + +/** The shared store: one document of entries, and the writes the benchmark makes. */ +const Store = local("bench.store", { + version: 1, + params: { name: str() }, + members: { + doc: Shape, + append: method(Entry), + edit: method(obj({ index: int(), text: str() })), + remove: method(int()), + stream: method(str()), + }, +}); + +/** The remote view a client holds: the store's document as it is. */ +const View = remote("bench.view", { + version: 1, + params: { name: str() }, + variants: { owner: { state: { doc: Shape } } }, +}); + +/** A file service: a call with a small argument, and one with bytes. */ +const Files = remote("bench.files", { + version: 1, + params: {}, + variants: { + owner: { methods: { ping: method(obj({ n: int() }), int()), put: method(obj({ data: bytes() }), int()) } }, + }, +}); + +const Doc = defineState("doc", { schema: Shape, version: 1, initial: () => ({ entries: [] }) }); + +/** The `n` of every `ping` in the order the provider ran them. */ +const pings: number[] = []; + +const facet = defineFacet

((host) => { + host.serve(Store, (_, self) => { + const doc = self.state(Doc); + const change = ( + fn: (draft: { entries: { id: number; text: string; author: string; at: number }[] }) => void, + ctx: Context, + ) => doc.change((draft) => void fn(draft), ctx).then(() => null); + return { + doc, + append: (entry, ctx) => change((d) => void d.entries.push(entry), ctx), + edit: ({ index, text }, ctx) => + change((d) => { + d.entries[index]!.text = text; + }, ctx), + remove: (index, ctx) => change((d) => void d.entries.splice(index, 1), ctx), + stream: (token, ctx) => + change((d) => { + d.entries[d.entries.length - 1]!.text += token; + }, ctx), + }; + }); + host.serve(View, ({ name }, _principal, self) => ({ + variant: "owner", + members: { doc: self.get(Store, { name }).doc }, + })); + host.serve(Files, () => ({ + variant: "owner", + members: { + ping: async ({ n }) => { + pings.push(n); + return n + 1; + }, + put: async ({ data }) => data.byteLength, + }, + })); +}); + +/** A link end counting what it sends and keeping the last message. */ +function counted(link: Link): Link & { bytes: number; messages: number; last: Uint8Array | undefined } { + const counting = { + bytes: 0, + messages: 0, + last: undefined as Uint8Array | undefined, + send(message: Uint8Array) { + counting.bytes += message.byteLength; + counting.messages++; + counting.last = message; + link.send(message); + }, + onMessage: (listener: (message: Uint8Array) => void) => link.onMessage(listener), + close: () => link.close(), + onClose: (listener: () => void) => link.onClose(listener), + }; + return counting; +} + +const telemetry = new InMemoryTelemetryContext(); +const span = await telemetry.startSpan({ name: "bench" }, (s) => s); +const counting = withTelemetry(span, BACKGROUND_CONTEXT); + +/** The encodes counted so far (the in-memory adapter records each add). */ +const encodes = (): number => + telemetry + .getMetrics() + .filter(({ name }) => name === "chord.link.encodes") + .reduce((sum, { value }) => sum + value, 0); + +type Entry = { id: number; text: string; author: string; at: number }; + +function entry(id: number, length: number): Entry { + return { id, text: "x".repeat(length), author: "mario", at: 1_700_000_000_000 + id }; +} + +const notes = (): Entry[] => Array.from({ length: 1_000 }, (_, id) => entry(id, 80)); + +const document = (): Entry[] => Array.from({ length: 500 }, (_, id) => entry(id, 960)); + +/** + * A server with `entries` in store `name`, a client connected over a counted link in `codec`. `metered: false` leaves + * telemetry out (the in-memory adapter keeps every record, which a memory measurement would count). + */ +async function setup(name: string, entries: Entry[], codec: Codec, metered = true) { + const context = metered ? counting : BACKGROUND_CONTEXT; + const server = createHost

({ + principal: { kind: "server" }, + telemetry: metered ? telemetry : undefined, + onError: console.error, + }); + await server.load([{ name: "@bench/app", facet }], context); + const store = await server.get(Store, { name }, context); + for (const item of entries) await store.append(item, context); + const [a, b] = memoryLinkPair(); + const down = counted(a); + const up = counted(b); + server.accept(down, { principal: { kind: "owner" }, codec }, context); + const client = createHost

({ onError: console.error }); + client.connect(up, { principal: { kind: "server" }, codec }, context); + await client.load([], context); + const close = async () => { + await client.close(context); + await server.close(context); + }; + return { server, client, store, down, up, close, context }; +} + +const gc = (globalThis as { gc?: () => void }).gc!; + +/** Heap and `ArrayBuffer` bytes: the message bytes themselves live outside the heap. */ +type Memory = { readonly heap: number; readonly arrayBuffers: number }; + +/** Waits until queued deliveries ran and collects garbage, repeatedly, then returns the memory in use. */ +async function settledMemory(): Promise { + for (let i = 0; i < 5; i++) { + await new Promise((resolve) => setTimeout(resolve, 0)); + gc(); + } + const { heapUsed, arrayBuffers } = process.memoryUsage(); + return { heap: heapUsed, arrayBuffers }; +} + +/** Heap bytes `work` allocated: the growth plus what the collections it caused freed. */ +function allocated(work: () => void): number { + gc(); + const before = process.memoryUsage().heapUsed; + const profiler = new GCProfiler(); + profiler.start(); + work(); + const after = process.memoryUsage().heapUsed; + let freed = 0; + for (const { beforeGC, afterGC } of profiler.stop().statistics) { + freed += beforeGC.heapStatistics.usedHeapSize - afterGC.heapStatistics.usedHeapSize; + } + return after - before + freed; +} + +/** + * `ArrayBuffer` bytes one encoded message keeps alive (its backing store, which may be larger than the message), over + * `runs` messages kept at once. Freed buffers are not reported reliably during a loop, so this measures what is kept. + */ +async function retainedBuffers(codec: Codec, message: Message, runs: number): Promise { + const before = (await settledMemory()).arrayBuffers; + const kept = Array.from({ length: runs }, () => codec.encode(message)); + const after = (await settledMemory()).arrayBuffers; + return kept.length === runs ? (after - before) / runs : 0; +} + +/** + * One message through a codec, `runs` times: wire bytes, encode and decode time per message, and heap allocated per + * message while encoding and decoding. + */ +function codecRun(codec: Codec, message: Message, runs: number) { + const bytes = codec.encode(message); + for (let i = 0; i < 100; i++) codec.decode(codec.encode(message)); + // The fastest of five loops: one loop of a microsecond-sized message is at the mercy of the collector and the JIT. + let encode = Number.POSITIVE_INFINITY; + let decode = Number.POSITIVE_INFINITY; + for (let repeat = 0; repeat < 5; repeat++) { + const t0 = performance.now(); + for (let i = 0; i < runs; i++) codec.encode(message); + const t1 = performance.now(); + for (let i = 0; i < runs; i++) codec.decode(bytes); + const t2 = performance.now(); + encode = Math.min(encode, (t1 - t0) / runs); + decode = Math.min(decode, (t2 - t1) / runs); + } + const heap = allocated(() => { + for (let i = 0; i < runs; i++) codec.decode(codec.encode(message)); + }); + return { bytes: bytes.byteLength, encode, decode, heap: heap / runs }; +} + +const CODECS = [binaryCodec, textCodec]; + +/** Results by row label, one value per codec. */ +const table = new Map(); + +const record = (label: string, codec: Codec, value: string): void => { + const row = table.get(label) ?? []; + row[CODECS.indexOf(codec)] = value; + table.set(label, row); +}; + +const ms = (value: number) => `${value.toFixed(3)} ms`; +const us = (value: number) => `${(value * 1000).toFixed(1)} µs`; +const kb = (value: number) => `${(value / 1024).toFixed(1)} KiB`; +const b = (value: number) => (value >= 10_240 ? kb(value) : `${value.toFixed(0)} B`); + +/** Records a captured message's codec run under `label`. */ +async function codecRows(label: string, codec: Codec, message: Message, runs: number): Promise { + const run = codecRun(codec, message, runs); + record(`${label}: wire bytes`, codec, b(run.bytes)); + record(`${label}: encode / decode`, codec, `${us(run.encode)} / ${us(run.decode)}`); + record(`${label}: heap per encode + decode`, codec, b(run.heap)); + record(`${label}: array buffer kept per encoded message`, codec, b(await retainedBuffers(codec, message, 50))); +} + +/** A snapshot over a link: hold to live, then the captured `bound` through the codec alone. */ +async function snapshot(label: string, codec: Codec, entries: Entry[]): Promise { + const { client, down, close } = await setup(label, entries, codec); + let start = performance.now(); + // The first hold also activates the view; it stays held, so later holds measure the snapshot path alone. + await client.get(View, { name: label }, counting); + record(`${label}: first hold to live (activation, one sample)`, codec, ms(performance.now() - start)); + const runs = 100; + start = performance.now(); + for (let i = 0; i < runs; i++) (await client.get(View, { name: label }, counting)).release(); + record(`${label}: hold to live (two hops, mean of ${runs})`, codec, ms((performance.now() - start) / runs)); + await codecRows(`${label}: snapshot`, codec, codec.decode(down.last!), 1_000); + await close(); +} + +/** One commit per run: commit-to-frame, bytes and messages per commit, encodes per hold; then its message alone. */ +async function commits( + label: string, + codec: Codec, + write: (store: Awaited>["store"], i: number) => Promise, +): Promise { + const { client, store, down, close } = await setup(label, notes(), codec); + const view = await client.get(View, { name: label }, counting); + const watch = await view.doc.watch(counting); + let frame: () => void = () => {}; + watch.start(async () => frame()); + const runs = 1_000; + const [messages0, encodes0] = [down.messages, encodes()]; + let total = 0; + for (let i = 0; i < runs; i++) { + const arrived = new Promise((resolve) => { + frame = resolve; + }); + const start = performance.now(); + await write(store, i); + await arrived; + total += performance.now() - start; + } + record(`${label}: commit to frame`, codec, ms(total / runs)); + record( + `${label}: messages per commit, encodes per hold`, + codec, + `${(down.messages - messages0) / runs}, ${(encodes() - encodes0) / runs}`, + ); + await codecRows(`${label}: state message`, codec, codec.decode(down.last!), 10_000); + await close(); +} + +/** A streamed reply: 2,000 small text appends to one entry. */ +async function stream(codec: Codec): Promise { + const { client, store, down, close } = await setup("stream", [entry(0, 0)], codec); + const view = await client.get(View, { name: "stream" }, counting); + const watch = await view.doc.watch(counting); + let frames = 0; + const done = Promise.withResolvers(); + watch.start(async () => { + if (++frames === 2_000) done.resolve(); + }); + const bytes0 = down.bytes; + const start = performance.now(); + for (let i = 0; i < 2_000; i++) await store.stream(" tok", counting); + await done.promise; + record("streamed reply, 2,000 appends: time per append", codec, ms((performance.now() - start) / 2_000)); + record( + "streamed reply: bytes per append, total", + codec, + `${((down.bytes - bytes0) / 2_000).toFixed(0)} B, ${kb(down.bytes - bytes0)}`, + ); + await codecRows("streamed token", codec, codec.decode(down.last!), 10_000); + await close(); +} + +/** One trial: the memory retained per hold and the messages of one commit, with `holds` holds on one link. */ +async function holdTrial( + codec: Codec, + holds: number, +): Promise<{ readonly perHold: Memory; readonly messages: number }> { + const name = `holds ${holds}`; + const { client, store, down, close } = await setup(name, notes(), codec, false); + const before = await settledMemory(); + const views = []; + for (let i = 0; i < holds; i++) views.push(await client.get(View, { name }, BACKGROUND_CONTEXT)); + await store.edit({ index: 1, text: "warm" }, BACKGROUND_CONTEXT); + const after = await settledMemory(); + const messages0 = down.messages; + await store.edit({ index: 2, text: "changed" }, BACKGROUND_CONTEXT); + await new Promise((resolve) => setTimeout(resolve, 0)); + const messages = down.messages - messages0; + await close(); + const perHold = { + heap: (after.heap - before.heap) / holds, + arrayBuffers: (after.arrayBuffers - before.arrayBuffers) / holds, + }; + return { perHold, messages }; +} + +/** Calls: a small argument, and a 1 MiB `bytes()` argument; round trips, then the call and result messages alone. */ +async function calls(codec: Codec): Promise { + const { client, down, up, close } = await setup("calls", [], codec); + const files = await client.get(Files, {}, counting); + let start = performance.now(); + for (let i = 0; i < 1_000; i++) await files.ping({ n: i }, counting); + record("call, small argument: round trip", codec, ms((performance.now() - start) / 1_000)); + await codecRows("call, small argument: call message", codec, codec.decode(up.last!), 10_000); + await codecRows("call, small argument: result message", codec, codec.decode(down.last!), 10_000); + const data = new Uint8Array(1024 * 1024); + start = performance.now(); + for (let i = 0; i < 50; i++) await files.put({ data }, counting); + record("call, 1 MiB bytes argument: round trip", codec, ms((performance.now() - start) / 50)); + await codecRows("call, 1 MiB bytes argument: call message", codec, codec.decode(up.last!), 1_000); + await close(); +} + +/** + * Drops the client's parent link (the server's end closes it) and waits until the client saw the close; then connects + * a new counted link and returns it with the time it was connected. + */ +async function relink(server: Host

, client: Host

, down: Link, codec: Codec) { + down.close(); + await new Promise((resolve) => setTimeout(resolve, 0)); + const [a, b] = memoryLinkPair(); + const next = { down: counted(a), up: counted(b) }; + const start = performance.now(); + server.accept(next.down, { principal: { kind: "owner" }, codec }, BACKGROUND_CONTEXT); + client.connect(next.up, { principal: { kind: "server" }, codec }, BACKGROUND_CONTEXT); + return { ...next, start }; +} + +/** Resolves when `state` is live. */ +async function live(state: NativeWatchable): Promise { + if (state.value.kind === "live") return; + const watch = await state.watch(BACKGROUND_CONTEXT); + const done = Promise.withResolvers(); + watch.start(async (call) => { + if (call.kind === "value" && call.value.kind === "live") done.resolve(); + }); + await done.promise; + await watch.stop(); +} + +/** + * Reconnects with `references` held references of the 1,000-item view (one provider object, a hold each): time from + * `connect` to every reference live again, and the bytes of the re-holds both ways; the mean of `runs` reconnects. + */ +async function reconnects(codec: Codec, references: number, runs: number): Promise { + const name = `reconnect ${references}`; + const { server, client, down, close } = await setup(name, notes(), codec, false); + const views = []; + for (let i = 0; i < references; i++) views.push(await client.get(View, { name }, BACKGROUND_CONTEXT)); + let link = down; + let time = 0; + let bytes = { down: 0, up: 0 }; + for (let run = 0; run < runs; run++) { + const next = await relink(server, client, link, codec); + await Promise.all(views.map((view) => live(view.state))); + time += performance.now() - next.start; + bytes = { down: next.down.bytes, up: next.up.bytes }; + link = next.down; + } + const label = `reconnect, ${references} reference(s) of 1,000 items`; + record(`${label}: time to all live (mean of ${runs})`, codec, ms(time / runs)); + record(`${label}: bytes down / up`, codec, `${b(bytes.down)} / ${b(bytes.up)}`); + await close(); +} + +/** 1,000 calls made while the link is down, flushed on reconnect: time from `connect` to the last result, and order. */ +async function parkedCalls(codec: Codec): Promise { + const { server, client, down, close } = await setup("parked", [], codec, false); + const files = await client.get(Files, {}, BACKGROUND_CONTEXT); + let link = down; + let time = 0; + let ordered = true; + const runs = 10; + for (let run = 0; run < runs; run++) { + pings.length = 0; + const next = await relink(server, client, link, codec); + // The link is connected but not yet live: every call below parks until the re-hold is bound. + const results = await Promise.all(Array.from({ length: 1_000 }, (_, n) => files.ping({ n }, BACKGROUND_CONTEXT))); + time += performance.now() - next.start; + ordered &&= pings.every((n, index) => n === index) && results.every((result, index) => result === index + 1); + link = next.down; + } + record("1,000 calls parked while down, flushed on reconnect: time to the last result", codec, ms(time / runs)); + record("1,000 parked calls: run in the order they were made", codec, ordered ? "yes" : "no"); + await close(); +} + +for (const codec of CODECS) { + // Warm-up: the first run pays for compilation. + await snapshot("warm-up", codec, notes()); + await snapshot("notes list of 1,000 items", codec, notes()); + await snapshot("500 KB document", codec, document()); + await commits("one appended item", codec, (store, i) => store.append(entry(10_000 + i, 80), counting)); + await commits("one changed field", codec, (store, i) => store.edit({ index: 500, text: `edit ${i}` }, counting)); + await commits("one removed item", codec, (store) => store.remove(0, counting)); + await stream(codec); + for (const holds of [1, 10]) { + // The median of seven trials: a single hold is small next to the collector's noise. + const trials: Awaited>[] = []; + for (let i = 0; i < 7; i++) trials.push(await holdTrial(codec, holds)); + const median = (of: (memory: Memory) => number) => + trials.map(({ perHold }) => of(perHold)).sort((a, b) => a - b)[3]!; + const retained = `${kb(median(({ heap }) => heap))} / ${b(median(({ arrayBuffers }) => arrayBuffers))}`; + record(`${holds} hold(s) of 1,000 items: heap / array buffers retained per hold (both ends)`, codec, retained); + record(`${holds} hold(s): messages per commit`, codec, `${trials[0]!.messages}`); + } + await calls(codec); + await reconnects(codec, 1, 20); + await reconnects(codec, 100, 5); + await parkedCalls(codec); +} + +const rows = [...table].filter(([label]) => !label.startsWith("warm-up")); +const width = Math.max(...rows.map(([label]) => label.length)); +console.log(`| ${"Case".padEnd(width)} | binary | text |`); +console.log(`| ${"-".repeat(width)} | --- | --- |`); +for (const [label, [binary, text]] of rows) console.log(`| ${label.padEnd(width)} | ${binary} | ${text} |`); diff --git a/packages/chord/test/next/bench/relay.ts b/packages/chord/test/next/bench/relay.ts new file mode 100644 index 000000000..4f2c382b0 --- /dev/null +++ b/packages/chord/test/next/bench/relay.ts @@ -0,0 +1,619 @@ +// Light benchmarks of links and relays (slice 4a): commit-to-frame and wire bytes through a relay (tab ↔ server ↔ +// worker) over memory links, a real IPC link to a forked worker and a loopback WebSocket to the tab; hold-to-live +// through a relay including the `resolve` of a worker; a tab throttled to 64 KiB/s with stalls of 1, 10 and 60 s +// receiving a streamed reply into a 500 KB view, directly and through a relay (the worker folds either way: its window +// is per origin), in virtual time; 10,000 calls from one tab with another tab's latency through the relay. +// Run by hand from packages/chord (not in CI): +// node --conditions=source --experimental-strip-types --expose-gc --no-flush-bytecode test/next/bench/relay.ts +// (`--no-flush-bytecode`: V8 otherwise drops the bytecode of functions idle during a stall, which reads as freed heap.) +// The same file is the forked worker when its first argument is `worker`. +import { type ChildProcess, fork } from "node:child_process"; +import type { AddressInfo } from "node:net"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; +import { WebSocketServer } from "ws"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { type IpcPort, ipcLink, type Link, memoryLinkPair, webSocketLink } from "../../../src/next/link/index.ts"; + +type P = { kind: "owner" } | { kind: "server" } | { kind: "worker"; session: string }; + +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); + +const Entry = obj({ id: int(), text: str(), author: str(), at: int() }); +const Shape = obj({ entries: arr(Entry) }); + +type Entry = { id: number; text: string; author: string; at: number }; + +/** A worker's own store: one document of entries, and the write the benchmark makes (a streamed token). */ +const Store = local("bench.store", { + version: 1, + params: {}, + members: { doc: Shape, stream: method(str()) }, +}); + +/** What a tab holds: its session's store as it is, placed on the session's worker, and a small call. */ +const View = remote("bench.view", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { owner: { state: { doc: Shape }, methods: { stream: method(str()), ping: method(int(), int()) } } }, +}); + +/** The start data a worker gets in its start message: its session and the payload its store begins with. */ +type Start = { readonly id: string; readonly entries: "notes" | "document" }; + +function entry(id: number, length: number): Entry { + return { id, text: "x".repeat(length), author: "mario", at: 1_700_000_000_000 + id }; +} + +/** The notes list (1,000 entries of 80 characters, 134 KiB) or a 500 KB conversation document (500 of 960). */ +function entriesOf(kind: Start["entries"]): Entry[] { + return kind === "notes" + ? Array.from({ length: 1_000 }, (_, id) => entry(id, 80)) + : Array.from({ length: 500 }, (_, id) => entry(id, 960)); +} + +/** A worker's facet: its store starts with the start message's entries (the §18 core facet pattern). */ +function workerFacet(start: Start) { + const Doc = defineState("doc", { + schema: Shape, + version: 1, + initial: () => ({ entries: entriesOf(start.entries) }), + }); + return defineFacet

((host) => { + host.serve(Store, (_, self) => { + const doc = self.state(Doc); + return { + doc, + stream: (token, ctx) => + doc + .change((draft) => { + draft.entries[draft.entries.length - 1]!.text += token; + }, ctx) + .then(() => null), + }; + }); + host.serve(View, (_params, _principal, self) => { + const store = self.get(Store, {}); + return { + variant: "owner", + members: { doc: store.doc, stream: (token, ctx) => store.stream(token, ctx), ping: async (n) => n + 1 }, + }; + }); + }); +} + +/** A worker host for `start`, not loaded yet. */ +function workerHost(start: Start): Host

{ + return createHost

({ + address: Session({ id: start.id }), + principal: { kind: "worker", session: start.id }, + onError: console.error, + }); +} + +/** A worker on `link`, its parent link: connect first, then load (W6). */ +function startWorker(start: Start, link: Link): Host

{ + const worker = workerHost(start); + worker.connect(link, { principal: { kind: "server" }, trusted: true }, context); + void worker.load([{ name: "@bench/worker", facet: workerFacet(start) }], context); + return worker; +} + +/** A link end counting the bytes it sends and receives. */ +type Counted = Link & { readonly sent: number; readonly received: number }; + +function counted(link: Link): Counted { + let sent = 0; + let received = 0; + return { + get sent() { + return sent; + }, + get received() { + return received; + }, + send(message) { + sent += message.byteLength; + link.send(message); + }, + onMessage: (listener) => + link.onMessage((message) => { + received += message.byteLength; + listener(message); + }), + close: () => link.close(), + onClose: (listener) => link.onClose(listener), + }; +} + +/** A link end timing how long its host handles each received message. */ +type Timed = Link & { longest: number }; + +function timed(link: Link): Timed { + const timing: Timed = { + longest: 0, + send: (message) => link.send(message), + onMessage: (listener) => + link.onMessage((message) => { + const start = performance.now(); + listener(message); + timing.longest = Math.max(timing.longest, performance.now() - start); + }), + close: () => link.close(), + onClose: (listener) => link.onClose(listener), + }; + return timing; +} + +/** How a server reaches its workers: in this process, or forked over IPC. */ +type WorkerKind = "memory" | "ipc"; + +/** The children forked in this run, ended with it. */ +const children: ChildProcess[] = []; + +/** A server routing `Session` to workers of `kind` whose stores start with `entries`. */ +async function server(kind: WorkerKind, entries: Start["entries"]) { + const host = createHost

({ address: Server(), principal: { kind: "server" }, onError: console.error }); + /** Each worker link's server end, counted; the in-process workers and their ends, timed. */ + const workerLinks: Counted[] = []; + const workers: { readonly host: Host

; readonly link: Timed }[] = []; + host.route( + Session, + { + authorize: () => true, + resolve: async (address) => { + const start = { id: address.params.id, entries }; + const principal = { kind: "worker", session: start.id } as const; + if (kind === "memory") { + const [a, b] = memoryLinkPair(); + const end = timed(b); + workers.push({ host: startWorker(start, end), link: end }); + const link = counted(a); + workerLinks.push(link); + return { link, principal, trusted: true }; + } + const child = fork(fileURLToPath(import.meta.url), ["worker"], { + serialization: "advanced", + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + }); + children.push(child); + const link = counted(ipcLink(child)); + // The app's own start message; the link skips it. + child.send(start); + workerLinks.push(link); + return { link, principal, trusted: true }; + }, + }, + context, + ); + await host.load([], context); + return { host, workerLinks, workers }; +} + +/** A tab: a client host connected to `link`. */ +async function tab(link: Link): Promise> { + const host = createHost

({ onError: console.error }); + host.connect(link, { principal: { kind: "server" } }, context); + await host.load([], context); + return host; +} + +/** A tab on `on` over a memory link, the server's end counted. */ +async function memoryTab(on: Host

): Promise<{ readonly tab: Host

; readonly link: Counted }> { + const [a, b] = memoryLinkPair(); + const link = counted(a); + on.accept(link, { principal: { kind: "owner" } }, context); + return { tab: await tab(b), link }; +} + +/** A tab on `on` over a loopback WebSocket (`ws` serving, Node's own `WebSocket` in the tab), the server's end counted. */ +async function webSocketTab(on: Host

) { + const wss = new WebSocketServer({ host: "127.0.0.1", port: 0 }); + await new Promise((resolve) => wss.once("listening", resolve)); + const accepted = Promise.withResolvers(); + wss.on("connection", (socket) => { + const link = counted(webSocketLink(socket)); + on.accept(link, { principal: { kind: "owner" } }, context); + accepted.resolve(link); + }); + const { port } = wss.address() as AddressInfo; + const host = await tab(webSocketLink(new WebSocket(`ws://127.0.0.1:${port}`))); + const close = () => new Promise((resolve) => wss.close(resolve)); + return { tab: host, link: await accepted.promise, close }; +} + +/** A worker with `entries` served directly to a tab over one memory link (no relay), the worker's end counted. */ +async function direct(entries: Start["entries"], wrap: (link: Link) => Link = (link) => link) { + const worker = workerHost({ id: "D", entries }); + await worker.load([{ name: "@bench/worker", facet: workerFacet({ id: "D", entries }) }], context); + const [a, b] = memoryLinkPair(); + const link = counted(a); + const end = timed(link); + worker.accept(end, { principal: { kind: "owner" } }, context); + return { worker, link, end, tab: await tab(wrap(b)) }; +} + +const ms = (value: number) => `${value.toFixed(3)} ms`; +const kb = (value: number) => `${(value / 1024).toFixed(1)} KiB`; +const b = (value: number) => (value >= 10_240 ? kb(value) : `${value.toFixed(0)} B`); +const s = (value: number) => `${value.toFixed(2)} s`; +const gc = (globalThis as { gc?: () => void }).gc!; + +/** Prints a markdown table. */ +function print(title: string, head: readonly string[], rows: readonly (readonly string[])[]): void { + console.log(`\n${title}\n`); + console.log(`| ${head.join(" | ")} |`); + console.log(`| ${head.map(() => "---").join(" | ")} |`); + for (const row of rows) console.log(`| ${row.join(" | ")} |`); +} + +/** + * Commits as a tab sees them: the tab calls `stream` on its view 1,000 times, each one commit on the worker; the time + * from the call to the tab's watch frame (the call's hops down, the frame's up), and the bytes per commit on the tab + * link (sent to the tab) and the worker link (received from the worker). + */ +async function commits(holder: Host

, session: string, tabLink: Counted, workerLinks: readonly Counted[]) { + const view = await holder.get(View, { session }, context); + // Made by the route's `resolve` for the first hold (none without a relay). + const workerLink = workerLinks[0]; + const watch = await view.doc.watch(context); + let frame: () => void = () => {}; + watch.start(async () => frame()); + // Warm-up. + for (let i = 0; i < 100; i++) await view.stream(" w", context); + const runs = 1_000; + const tab0 = tabLink.sent; + const worker0 = workerLink?.received ?? 0; + let total = 0; + for (let i = 0; i < runs; i++) { + const arrived = new Promise((resolve) => { + frame = resolve; + }); + const start = performance.now(); + await Promise.all([view.stream(" tok", context), arrived]); + total += performance.now() - start; + } + await watch.stop(); + view.release(); + const workerBytes = workerLink === undefined ? "-" : b((workerLink.received - worker0) / runs); + return [ms(total / runs), `${b((tabLink.sent - tab0) / runs)} / ${workerBytes}`]; +} + +/** Commit-to-frame through a relay over each transport, and directly (one memory link, no relay) as the baseline. */ +async function transports(): Promise { + const rows: string[][] = []; + { + const { host, workerLinks } = await server("memory", "notes"); + const { tab: t, link } = await memoryTab(host); + rows.push(["memory tab, in-process worker (memory)", ...(await commits(t, "C", link, workerLinks))]); + await t.close(context); + await host.close(context); + } + { + const { host, workerLinks } = await server("ipc", "notes"); + const { tab: t, link } = await memoryTab(host); + const start = performance.now(); + (await t.get(View, { session: "C" }, context)).release(); + const first = performance.now() - start; + rows.push(["memory tab, forked worker (IPC)", ...(await commits(t, "C", link, workerLinks))]); + rows.push([" first hold to live, the fork included (one sample)", ms(first), "-"]); + await t.close(context); + await host.close(context); + } + { + const { host, workerLinks } = await server("ipc", "notes"); + const { tab: t, link, close } = await webSocketTab(host); + rows.push(["loopback WebSocket tab, forked worker (IPC)", ...(await commits(t, "C", link, workerLinks))]); + await t.close(context); + await host.close(context); + await close(); + } + { + const { worker, link, tab: t } = await direct("notes"); + rows.push(["no relay: memory tab on the worker (baseline)", ...(await commits(t, "D", link, []))]); + await t.close(context); + await worker.close(context); + } + print( + "Commits through a relay (1,000 streamed tokens into the 1,000-item notes list view)", + ["Topology", "call to frame", "bytes per commit: to the tab / from the worker"], + rows, + ); +} + +/** + * Hold to live through a relay: a first hold of each of 20 sessions (the route's `resolve` starts an in-process worker, + * which loads and activates the view: one hop more than a later hold), then 100 holds of a resolved session, each + * released; and the same 100 holds directly on a worker. + */ +async function holds(): Promise { + const rows: string[][] = []; + const { host } = await server("memory", "notes"); + const { tab: t } = await memoryTab(host); + let start = performance.now(); + for (let i = 0; i < 20; i++) await t.get(View, { session: `H${i}` }, context); + rows.push([ + "first hold, `resolve` of an in-process worker included (mean of 20)", + ms((performance.now() - start) / 20), + ]); + start = performance.now(); + for (let i = 0; i < 100; i++) (await t.get(View, { session: "H0" }, context)).release(); + rows.push([ + "hold of a resolved session through the relay (four hops, mean of 100)", + ms((performance.now() - start) / 100), + ]); + await t.close(context); + await host.close(context); + const { worker, tab: d } = await direct("notes"); + await d.get(View, { session: "D" }, context); + start = performance.now(); + for (let i = 0; i < 100; i++) (await d.get(View, { session: "D" }, context)).release(); + rows.push(["no relay: hold on the worker (two hops, mean of 100, baseline)", ms((performance.now() - start) / 100)]); + await d.close(context); + await worker.close(context); + print("Hold to live, the 1,000-item notes list view (133.7 KiB snapshot)", ["Case", "time"], rows); +} + +/** + * A tab's end of a link that delivers what arrives at `rate` bytes per virtual second, whole messages in order, and + * nothing while stalled; virtual time advances by `step`. Idle time banks no budget. + */ +class Throttled implements Link { + stalled = false; + delivered = 0; + readonly #link: Link; + readonly #rate: number; + readonly #queue: Uint8Array[] = []; + #budget = 0; + #listener: (message: Uint8Array) => void = () => {}; + + constructor(link: Link, rate: number) { + this.#link = link; + this.#rate = rate; + link.onMessage((message) => this.#queue.push(message)); + } + + send(message: Uint8Array): void { + this.#link.send(message); + } + + onMessage(listener: (message: Uint8Array) => void): void { + this.#listener = listener; + } + + close(): void { + this.#link.close(); + } + + onClose(listener: () => void): void { + this.#link.onClose(listener); + } + + /** `milliseconds` of virtual time pass. */ + step(milliseconds: number): void { + if (this.stalled) return; + const budget = (this.#rate * milliseconds) / 1000; + this.#budget += budget; + while (this.#queue.length > 0 && this.#queue[0]!.byteLength <= this.#budget) { + const message = this.#queue.shift()!; + this.#budget -= message.byteLength; + this.delivered += message.byteLength; + this.#listener(message); + } + if (this.#queue.length === 0) this.#budget = Math.min(this.#budget, budget); + } +} + +/** The heap in use once queued work ran and garbage was collected, repeatedly. */ +async function settledHeap(): Promise { + for (let i = 0; i < 5; i++) { + await new Promise((resolve) => setTimeout(resolve, 0)); + gc(); + } + return process.memoryUsage().heapUsed; +} + +/** Lets every host handle what the last step delivered (memory links deliver in microtasks). */ +const settle = () => new Promise((resolve) => setImmediate(resolve)); + +const STEP = 10; + +/** Tokens per virtual second: a fast model's streamed reply. */ +const TOKENS = 100; + +/** + * A pause of the throttled tab: a stall (it receives nothing and so acknowledges nothing), or the same time without one + * (the control: what the streaming alone costs). + */ +type Pause = { readonly kind: "stall" | "control"; readonly seconds: number }; + +/** + * A tab throttled to 64 KiB/s holding the 500 KB document view of a worker that streams a reply at 100 tokens/s, in + * virtual time (10 ms steps): 2 s of streaming, the pause, 10 s more of streaming, then until the tab caught up. + * `relayed`: through a server (it forwards at once, the worker folds the tab origin's ops); `direct`: on the worker (it + * folds the ops in its pending queues). + */ +async function throttled(topology: "direct" | "relayed", pause: Pause): Promise { + const relayed = topology === "relayed"; + let throttle: Throttled | undefined; + const throttling = (link: Link) => { + throttle = new Throttled(link, 64 * 1024); + return throttle; + }; + let t: Host

; + let producer: () => { readonly host: Host

; readonly end: Timed }; + let close: () => Promise; + if (relayed) { + const { host, workers } = await server("memory", "document"); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" } }, context); + t = await tab(throttling(b)); + producer = () => ({ host: workers[0]!.host, end: workers[0]!.link }); + close = async () => { + await t.close(context); + await host.close(context); + }; + } else { + const made = await direct("document", throttling); + t = made.tab; + producer = () => ({ host: made.worker, end: made.end }); + close = async () => { + await t.close(context); + await made.worker.close(context); + }; + } + // The first snapshot crosses the throttle in virtual time too. + const getting = t.get(View, { session: relayed ? "T" : "D" }, context); + let bound = false; + void getting.then(() => { + bound = true; + }); + while (!bound) { + throttle!.step(STEP); + await settle(); + } + const view = await getting; + const { host: worker, end: providerEnd } = producer(); + const store = await worker.get(Store, {}, context); + const last = (value: { readonly entries: readonly Entry[] }) => value.entries.at(-1)!.text.length; + const caughtUp = () => last(view.doc.value) === last(store.doc.value); + let clock = 0; + const commitTimes = { outside: 0, outsideCount: 0, during: 0, duringCount: 0 }; + /** One virtual step: a token when streaming (its commit timed), delivery, the hosts' turn. */ + const step = async (streaming: boolean, phase: "outside" | "during") => { + clock += STEP; + if (streaming && clock % (1000 / TOKENS) === 0) { + const start = performance.now(); + await store.stream(" tok", context); + commitTimes[phase] += performance.now() - start; + commitTimes[phase === "outside" ? "outsideCount" : "duringCount"]++; + } + throttle!.step(STEP); + await settle(); + }; + for (let i = 0; i < 2000 / STEP; i++) await step(true, "outside"); + while (!caughtUp()) await step(false, "outside"); + const heap0 = await settledHeap(); + throttle!.stalled = pause.kind === "stall"; + for (let i = 0; i < (pause.seconds * 1000) / STEP; i++) await step(true, "during"); + const heap = (await settledHeap()) - heap0; + throttle!.stalled = false; + const stallEnd = clock; + const delivered0 = throttle!.delivered; + providerEnd.longest = 0; + let recovered: number | undefined; + for (let i = 0; recovered === undefined && i < 120_000 / STEP; i++) { + await step(clock - stallEnd < 10_000, "outside"); + if (caughtUp()) recovered = clock - stallEnd; + } + const row = [ + pause.kind === "stall" + ? `${topology}, ${pause.seconds} s stall` + : `${topology}, ${pause.seconds} s without a stall (control)`, + recovered === undefined ? "> 120 s" : s(recovered / 1000), + b(throttle!.delivered - delivered0), + kb(heap), + `${ms(commitTimes.outside / commitTimes.outsideCount)} / ${ms(commitTimes.during / commitTimes.duringCount)}`, + ms(providerEnd.longest), + ]; + store.release(); + view.release(); + await close(); + return row; +} + +async function slowTabs(): Promise { + const rows: string[][] = []; + for (const topology of ["direct", "relayed"] as const) { + for (const seconds of [1, 10, 60]) rows.push(await throttled(topology, { kind: "stall", seconds })); + rows.push(await throttled(topology, { kind: "control", seconds: 60 })); + } + print( + "A tab throttled to 64 KiB/s, a reply streamed at 100 tokens/s into the 500 KB view (virtual time)", + [ + "Case", + "caught up after the stall", + "bytes to the tab until then", + "heap growth during the stall", + "commit at the worker, outside / during the pause", + "longest message handling at the provider after the stall", + ], + rows, + ); +} + +/** 10,000 calls from tab A through the relay at once, and tab B's sequential calls meanwhile; B alone as the baseline. */ +async function calls(): Promise { + const { host } = await server("memory", "notes"); + const a = await memoryTab(host); + const bTab = await memoryTab(host); + const viewA = await a.tab.get(View, { session: "K" }, context); + const viewB = await bTab.tab.get(View, { session: "K" }, context); + const latencies = async (until: () => boolean, max: number) => { + const times: number[] = []; + while (!until() && times.length < max) { + const start = performance.now(); + await viewB.ping(times.length, context); + times.push(performance.now() - start); + } + return times.sort((x, y) => x - y); + }; + const percentile = (times: readonly number[], p: number) => + times[Math.min(times.length - 1, Math.floor(times.length * p))]!; + for (let i = 0; i < 1_000; i++) await viewB.ping(i, context); + const alone = await latencies(() => false, 1_000); + let done = false; + const start = performance.now(); + const burst = Promise.all(Array.from({ length: 10_000 }, (_, i) => viewA.ping(i, context))).then(() => { + done = true; + }); + const during = await latencies(() => done, 100_000); + await burst; + const total = performance.now() - start; + print( + "Calls through the relay (in-process worker, memory links, N = 256 per link)", + ["Case", "p50", "p95", "max", "calls"], + [ + ["tab B alone", ms(percentile(alone, 0.5)), ms(percentile(alone, 0.95)), ms(alone.at(-1)!), `${alone.length}`], + [ + "tab B while tab A has 10,000 calls in flight or queued", + ms(percentile(during, 0.5)), + ms(percentile(during, 0.95)), + ms(during.at(-1)!), + `${during.length}`, + ], + ["tab A: 10,000 calls at once, until the last result", ms(total), "-", "-", "10000"], + ], + ); + await a.tab.close(context); + await bTab.tab.close(context); + await host.close(context); +} + +// The forked worker: its link first (it queues what arrives before the start message), then the start message. +if (process.argv[2] === "worker") { + // A forked child's `process` is an IPC port. + const link = ipcLink(process as unknown as IpcPort); + const start = await new Promise((resolve) => { + const listener = (message: unknown) => { + if (message instanceof Uint8Array) return; + process.off("message", listener); + // The app's own message, sent by `resolve` above. + resolve(message as Start); + }; + process.on("message", listener); + }); + startWorker(start, link); + link.onClose(() => process.exit(0)); +} else { + await transports(); + await holds(); + await slowTabs(); + await calls(); + for (const child of children) child.kill(); +} diff --git a/packages/chord/test/next/bench/store.ts b/packages/chord/test/next/bench/store.ts new file mode 100644 index 000000000..ce2923de9 --- /dev/null +++ b/packages/chord/test/next/bench/store.ts @@ -0,0 +1,450 @@ +// Benchmarks of the store (slice 6a §14, decision "The commit-step benchmark"): the commit step against Durable's +// `SqliteStorage` on the same SQLite file settings, (a) one document delta per commit, (b) one table put per commit, +// (c) 1,000 whole transactions back to back (a host's `change` against a Durable Session's `commit`), with and without +// a watched document, and (d) transactions of one in-memory document only (no store); as bursts (median and p95 per +// commit over several runs) and the heap after GC; the outbox's extra cost apart. Then (e) the other costs: queue wait +// against step time, query pages, a 10k-row `updateWhere`'s step, long text per row, and migration chunks (the first +// and the late ones). +// Run by hand from packages/chord (not in CI): +// node --conditions=source --experimental-strip-types --expose-gc test/next/bench/store.ts +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { performance } from "node:perf_hooks"; +import { defineDoc } from "../../../../durable/src/documents.ts"; +import { createSession } from "../../../../durable/src/session/session.ts"; +import { SqliteStorage } from "../../../../durable/src/storage/sqlite/storage.ts"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import { changeDocument, createDocument } from "../../../src/next/store/engine.ts"; +import { lt } from "../../../src/next/store/expression.ts"; +import { query } from "../../../src/next/store/query.ts"; +import { TableSql, tableSqlName } from "../../../src/next/store/sql.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import { type ChordStore, openStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import { evolveTable } from "../../../src/next/store/table-evolve.ts"; +import { TableHandle } from "../../../src/next/store/table-handle.ts"; +import { prepareTables } from "../../../src/next/store/table-migration.ts"; +import { stagePut, stageTable } from "../../../src/next/store/table-writes.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const COMMITS = 1_000; +const RUNS = 7; + +/** One commit as a benchmark makes it, given its index. */ +type Commit = (index: number) => Promise; + +/** Per-commit times (ms) of `RUNS` bursts of `COMMITS` commits each, after one warm-up burst. */ +async function bursts(setup: () => Promise<{ readonly commit: Commit; readonly close: () => Promise }>) { + const times: number[] = []; + const perRun: number[] = []; + for (let run = -1; run < RUNS; run++) { + const { commit, close } = await setup(); + const started = performance.now(); + for (let index = 0; index < COMMITS; index++) { + const before = performance.now(); + await commit(index); + if (run >= 0) times.push(performance.now() - before); + } + if (run >= 0) perRun.push((performance.now() - started) / COMMITS); + await close(); + } + times.sort((a, b) => a - b); + perRun.sort((a, b) => a - b); + return { + median: times[Math.floor(times.length / 2)]!, + p95: times[Math.floor(times.length * 0.95)]!, + burst: perRun[Math.floor(perRun.length / 2)]!, + }; +} + +/** A fresh SQLite file in its own directory. */ +async function file(): Promise<{ readonly path: string; readonly remove: () => Promise }> { + const directory = await mkdtemp(join(tmpdir(), "chord-store-bench-")); + return { path: join(directory, "bench.sqlite"), remove: () => rm(directory, { recursive: true, force: true }) }; +} + +const VALUE = { title: "a note", body: "x".repeat(200), n: 0 }; + +/** Chord's store: one document created, then one delta per commit. */ +function chordDeltas(outbox: boolean) { + return async () => { + const { path, remove } = await file(); + const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { + outbox, + })) as ChordStore; + const address = { namespace: "chord", owner: "session:bench.store", kind: "note", key: "" }; + const base = { version: 1, fingerprint: "0123456789abcdef", value: VALUE }; + let id = 0; + const fence = { + admit: () => true, + adopt: (_seq: number, ids: readonly number[]) => { + id = ids[0] ?? id; + }, + }; + await store.commit([createDocument(address, "latest", base, JSON.stringify(VALUE))], fence, context); + return { + commit: (index: number) => { + const ops = [["s", ["n"], index + 1]] as const; + return store.commit( + [changeDocument(id, index + 1, { kind: "delta", version: 1, ops }, JSON.stringify(ops))], + fence, + context, + ); + }, + close: async () => { + await store.close(context); + await remove(); + }, + }; + }; +} + +/** Durable's `SqliteStorage`: one session document created, then one delta per commit. */ +async function durableDeltas() { + const { path, remove } = await file(); + const storage = await SqliteStorage.open(await openNodeSqliteDatabase(path)); + const id = await storage.mintId(); + const record = { id, kind: "note", scope: { kind: "session" } } as never; + await storage.commit( + [{ type: "document.create", record, content: { kind: "base", version: 1, value: VALUE } }], + () => {}, + context, + ); + return { + commit: (index: number) => + storage.commit( + [ + { type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["n"], index + 1]] } }, + ] as never, + () => {}, + context, + ), + close: async () => { + await storage.close(context); + await remove(); + }, + }; +} + +const EDIT_COLUMNS = { id: str(), path: str(), at: int(), body: str() }; +const Edits = defineTable("edits", { version: 1, columns: EDIT_COLUMNS, key: ["id"] }); +/** `Edits` with an index, for pages in a non-key order. */ +const IndexedEdits = defineTable("edits", { + version: 1, + columns: EDIT_COLUMNS, + key: ["id"], + indexes: { by_at: ["at"] }, +}); + +/** Chord's store: one table created, then one `put` per commit (the row checked and encoded in the commit). */ +function chordPuts(outbox: boolean) { + return async () => { + const { path, remove } = await file(); + const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { + outbox, + })) as ChordStore; + const table = tableSqlName("session", "bench.store", Edits.name); + const sql = new TableSql(table, Edits.columns, Edits.rows.physical, Edits.key); + const fence = { admit: () => true, adopt: () => {} }; + const { version, record: columns, key, fingerprint } = Edits; + await store.commit( + [stageTable({ kind: "table", table, version, state: "ready", columns, key, fingerprint })], + fence, + context, + ); + return { + commit: (index: number) => { + const row = Edits.rows.put( + { id: `e${index}`, path: `src/file${index % 50}.ts`, at: index, body: VALUE.body }, + version, + ); + return store.commit([stagePut(sql.put, table, row, version)], fence, context); + }, + close: async () => { + await store.close(context); + await remove(); + }, + }; + }; +} + +/** Durable's `SqliteStorage`: one conversation created, then one entry appended per commit. */ +async function durableEntries() { + const { path, remove } = await file(); + const storage = await SqliteStorage.open(await openNodeSqliteDatabase(path)); + await storage.commit([{ type: "conversation", value: { id: 1 } }] as never, () => {}, context); + return { + commit: async (index: number) => { + const id = await storage.mintId(); + const data = { path: `src/file${index % 50}.ts`, at: index, body: VALUE.body }; + return storage.commit( + [{ type: "entry", value: { id, conversationId: 1, kind: "edit", data } }] as never, + () => {}, + context, + ); + }, + close: async () => { + await storage.close(context); + await remove(); + }, + }; +} + +/** The heap after GC, in KiB. */ +const NoteSchema = obj({ title: str(), body: str(), n: int() }); + +const InMemoryNote = defineState("bench.note", { schema: NoteSchema, version: 1, initial: () => ({ ...VALUE }) }); + +const PersistedNote = defineState("bench.note", { + schema: NoteSchema, + version: 1, + initial: () => ({ ...VALUE }), + persist: true, + checkpointWhen: () => false, +}); + +const Counter = local("bench.counter", { version: 1, params: {}, members: { touch: method() } }); + +/** Whole Chord transactions on a host: one `change` of one document per commit, persisted or in memory, watched or not. */ +function chordTransactions(options: { + readonly persist: boolean; + readonly watched: boolean; + readonly outbox: boolean; +}) { + return async () => { + const { path, remove } = await file(); + const store = options.persist + ? await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { outbox: options.outbox }) + : undefined; + let note: SingletonHandle | undefined; + const host = createHost({ store, onError: () => {} }); + const facet = defineFacet((h) => + h.serve(Counter, (_, self) => { + note = self.state(options.persist ? PersistedNote : InMemoryNote); + return { touch: async () => null }; + }), + ); + await host.load([{ name: "@bench/store", facet }], context); + const reference = await host.get(Counter, {}, context); + await note!.change((draft) => { + draft.n = -1; + }, context); + const watch = options.watched ? await note!.watch(context) : undefined; + watch?.start(async () => {}); + return { + commit: (index: number) => + note!.change((draft) => { + draft.n = index + 1; + }, context), + close: async () => { + await watch?.stop(); + reference.release(); + await host.close(context); + await store?.close(context); + await remove(); + }, + }; + }; +} + +const DurableNote = defineDoc({ + kind: "bench.note", + version: 1, + scope: "session", + initial: () => ({ ...VALUE }), + checkpointWhen: () => false, +}); + +/** Whole Durable transactions on a Session over `SqliteStorage`: one document change per commit, watched or not. */ +function durableTransactions(watched: boolean) { + return async () => { + const { path, remove } = await file(); + const storage = await SqliteStorage.open(await openNodeSqliteDatabase(path)); + const session = createSession(storage); + await session.commit(async (tx) => { + (await tx.doc(DurableNote)).n = -1; + }, context); + const watch = watched ? await session.watchDoc(DurableNote, context) : undefined; + watch?.start(async () => {}); + return { + commit: (index: number) => + session.commit(async (tx) => { + (await tx.doc(DurableNote)).n = index + 1; + }, context), + close: async () => { + await watch?.stop(); + await session.close(context); + await remove(); + }, + }; + }; +} + +function heap(): number { + globalThis.gc?.(); + return Math.round(process.memoryUsage().heapUsed / 1024); +} + +const format = ({ median, p95, burst }: { median: number; p95: number; burst: number }) => + `median ${(median * 1000).toFixed(1)} µs, p95 ${(p95 * 1000).toFixed(1)} µs, burst ${(burst * 1000).toFixed(1)} µs/commit`; + +const before = heap(); +const durable = await bursts(durableDeltas); +const chord = await bursts(chordDeltas(false)); +const outbox = await bursts(chordDeltas(true)); +console.log(`(a, c) one document delta per commit, ${RUNS} bursts of ${COMMITS}:`); +console.log(` Durable SqliteStorage: ${format(durable)}`); +console.log(` Chord store: ${format(chord)}`); +console.log(` Chord store + outbox: ${format(outbox)}`); +const durablePuts = await bursts(durableEntries); +const chordTable = await bursts(chordPuts(false)); +const tableOutbox = await bursts(chordPuts(true)); +console.log(`(b) one table put (Durable: one entry append) per commit, ${RUNS} bursts of ${COMMITS}:`); +console.log(` Durable SqliteStorage: ${format(durablePuts)}`); +console.log(` Chord store: ${format(chordTable)}`); +console.log(` Chord store + outbox: ${format(tableOutbox)}`); +for (const watched of [false, true]) { + const durableTx = await bursts(durableTransactions(watched)); + const chordTx = await bursts(chordTransactions({ persist: true, watched, outbox: false })); + const chordTxOutbox = await bursts(chordTransactions({ persist: true, watched, outbox: true })); + console.log(`(c) whole transactions, ${watched ? "a watched document" : "no watch"}, ${RUNS} bursts of ${COMMITS}:`); + console.log(` Durable Session: ${format(durableTx)}`); + console.log(` Chord host: ${format(chordTx)}`); + console.log(` Chord host + outbox: ${format(chordTxOutbox)}`); +} +const memory = await bursts(chordTransactions({ persist: false, watched: false, outbox: false })); +const memoryWatched = await bursts(chordTransactions({ persist: false, watched: true, outbox: false })); +console.log(`(d) one in-memory document per transaction (no store), ${RUNS} bursts of ${COMMITS}:`); +console.log(` Chord host: ${format(memory)}`); +console.log(` Chord host, watched: ${format(memoryWatched)}`); +console.log(`heap after GC: ${heap() - before} KiB above the start`); + +/** A file store with `count` rows of `Edits` at version 1 (one commit), as a table handle. */ +async function filledEdits(count: number, body = VALUE.body, definition: typeof Edits = Edits) { + const { path, remove } = await file(); + const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context)) as ChordStore; + const table = tableSqlName("session", "bench.store", Edits.name); + const fence = { admit: () => true, adopt: () => {} }; + await prepareTables([{ definition, table }], store, fence, "bench", context); + const handle = new TableHandle(definition, table, store, { + label: "bench", + scope: {}, + assertLive: () => {}, + fail: () => {}, + }); + const staged: StagedWrite[] = []; + const changes = new TableChanges(handle, staged); + for (let i = 0; i < count; i++) + changes.put({ id: `e${String(i).padStart(6, "0")}`, path: `src/f${i % 50}.ts`, at: i, body }); + await store.commit(staged, fence, context); + return { + store, + table, + handle, + fence, + close: async () => { + await store.close(context); + await remove(); + }, + }; +} + +/** The median of `values`. */ +function median(values: number[]): number { + values.sort((a, b) => a - b); + return values[Math.floor(values.length / 2)]!; +} + +console.log("(e) other costs:"); +{ + // 100 commits submitted at once: the time each waits in the queue, apart from its own step. + const { store, table, fence, close } = await filledEdits(0); + const sql = new TableSql(table, Edits.columns, Edits.rows.physical, Edits.key); + const waits: number[] = []; + const steps: number[] = []; + await Promise.all( + Array.from({ length: 100 }, async (_, index) => { + const row = Edits.rows.put({ id: `q${index}`, path: "p", at: index, body: VALUE.body }, 1); + const put = stagePut(sql.put, table, row, 1); + const submitted = performance.now(); + let started = 0; + const timed: StagedWrite = { + order: put.order, + apply: (step) => { + started = performance.now(); + return put.apply(step); + }, + }; + await store.commit([timed], fence, context); + waits.push(started - submitted); + steps.push(performance.now() - started); + }), + ); + console.log( + ` queue: 100 commits at once, median wait ${(median(waits) * 1000).toFixed(0)} µs, step ${(median(steps) * 1000).toFixed(1)} µs`, + ); + await close(); +} +{ + const { handle, close } = await filledEdits(10_000, VALUE.body, IndexedEdits); + const q = query(IndexedEdits).orderBy((e) => e.at, "desc"); + let after: string | undefined; + const started = performance.now(); + for (let page = 0; page < 100; page++) { + const result = await handle.page(q, 100, context, after === undefined ? {} : { after }); + after = result.next; + } + console.log( + ` query pages: 100 rows in an index's order, ${((performance.now() - started) * 10).toFixed(0)} µs per page`, + ); + await close(); +} +{ + const { store, handle, fence, close } = await filledEdits(10_000); + const staged: StagedWrite[] = []; + new TableChanges(handle, staged).updateWhere((e: { at: never }) => lt(e.at, 10_000 as never), { path: "moved" }); + const started = performance.now(); + await store.commit(staged, fence, context); + console.log( + ` updateWhere over 10,000 rows: one step of ${(performance.now() - started).toFixed(1)} ms (every owner's commits wait)`, + ); + await close(); +} +for (const length of [10, 10_000]) { + const body = "ü".repeat(length); + const started = performance.now(); + const { close } = await filledEdits(2_000, body); + console.log( + ` text: 2,000 puts with ${length}-character cells, ${(((performance.now() - started) / 2_000) * 1000).toFixed(1)} µs per row`, + ); + await close(); +} +{ + const { store, table, fence, close } = await filledEdits(10_000); + const EditsV2 = evolveTable(Edits, (t) => + t.add("dir", str({ default: "" })).rows((row) => ({ dir: row.path.slice(0, 3) })), + ); + const chunks: number[] = []; + const commit = store.commit.bind(store); + store.commit = async (writes, commitFence, commitContext) => { + const started = performance.now(); + const seq = await commit(writes, commitFence, commitContext); + chunks.push(performance.now() - started); + return seq; + }; + const started = performance.now(); + await prepareTables([{ definition: EditsV2, table }], store, fence, "bench", context); + const total = performance.now() - started; + const rows = chunks.slice(1, -1); + console.log( + ` migration (rows) of 10,000 rows: ${total.toFixed(0)} ms, ${((total / 10_000) * 1000).toFixed(1)} µs per row; ` + + `chunk commits: first ${rows[0]!.toFixed(2)} ms, last ${rows.at(-1)!.toFixed(2)} ms, median ${median(rows).toFixed(2)} ms`, + ); + await close(); +} diff --git a/packages/chord/test/next/contracts/accepts.test.ts b/packages/chord/test/next/contracts/accepts.test.ts new file mode 100644 index 000000000..83cbfcca9 --- /dev/null +++ b/packages/chord/test/next/contracts/accepts.test.ts @@ -0,0 +1,247 @@ +import { describe, expect, it, vi } from "vitest"; +import { + type AcceptsTable, + arr, + evolve, + int, + local, + method, + nul, + obj, + remote, + str, +} from "../../../src/next/contracts/index.ts"; +import { checkAccepts } from "../../../src/next/contracts/internal.ts"; + +const Note = obj({ text: str() }); +const NoteV2 = obj({ text: str(), tag: str() }); + +const NotesV1 = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(Note, str()), clear: method() } }, + reader: { state: { notes: arr(Note) }, methods: { add: method(Note, str()) } }, + }, +}); + +/** Calls `checkAccepts` through an untyped door, as a JavaScript caller would. */ +const checkUntyped = checkAccepts as (contract: unknown, accepts: unknown) => unknown; + +const toV1 = (note: { text: string; tag: string }) => ({ text: note.text }); + +describe("state mappings", () => { + const v2 = evolve(NotesV1, (e) => e.state("editor", "notes", arr(NoteV2))); + + it("a changed member maps the current member to the old one", () => { + const accepts: AcceptsTable = { 1: { state: { editor: { notes: (current) => current.map(toV1) } } } }; + expect(checkAccepts(v2, accepts)).toEqual({ versions: [{ version: 1, fingerprint: NotesV1.fingerprint }] }); + // @ts-expect-error: the changed member needs a mapping + expect(() => checkAccepts(v2, { 1: { state: { editor: {} } } })).toThrow( + "notes accepts[1].state.editor.notes: missing mapping", + ); + // @ts-expect-error: "refuse" is only for removed members + expect(() => checkAccepts(v2, { 1: { state: { editor: { notes: "refuse" } } } })).toThrow( + "notes accepts[1].state.editor.notes: expected a function", + ); + // @ts-expect-error: the mapping returns the old shape, not the current one + checkAccepts(v2, { 1: { state: { editor: { notes: (current) => current.map((note) => note.tag) } } } }); + }); + + it("a removed member maps the variant's whole current state, or refuses", () => { + const v3 = evolve(v2, (e) => e.removeState("editor", "count")); + const accepts: AcceptsTable = { + 1: { state: { editor: { notes: (current) => current.map(toV1), count: (state) => state.notes.length } } }, + 2: { state: { editor: { count: "refuse" } } }, + }; + expect(checkAccepts(v3, accepts).versions.map(({ version }) => version)).toEqual([1, 2]); + expect(() => checkUntyped(v3, { 2: { state: { editor: { count: 1 } } } })).toThrow( + 'notes accepts[2].state.editor.count: expected a function or "refuse"', + ); + }); + + it("an absent section is allowed only when nothing is needed there", () => { + // @ts-expect-error: the state section is needed + expect(() => checkAccepts(v2, { 1: {} })).toThrow("notes accepts[1].state.editor: missing mapping"); + const v3 = evolve(v2, (e) => e.addState("reader", "count", int())); + expect(() => checkAccepts(v3, { 2: {} })).not.toThrow(); + expect(() => checkAccepts(v3, { 2: { state: {}, methods: {} } })).not.toThrow(); + }); + + it("unnecessary mappings are refused", () => { + expect(() => + // @ts-expect-error: reader.notes did not change + checkAccepts(v2, { 1: { state: { editor: { notes: (c) => c.map(toV1) }, reader: { notes: (c) => c } } } }), + ).toThrow("notes accepts[1].state.reader: unnecessary mapping"); + const v3 = evolve(v2, (e) => e.addState("reader", "count", int())); + // @ts-expect-error: added members need nothing + expect(() => checkAccepts(v3, { 2: { state: { reader: { count: () => 0 } } } })).toThrow( + "notes accepts[2].state.reader: unnecessary mapping", + ); + }); + + it("is checked by edit, not by schema: a no-op edit and a reverted change still need mappings", () => { + const noop = evolve(NotesV1, (e) => e.state("reader", "notes", arr(Note))); + // @ts-expect-error: the edit needs an (identity) mapping + expect(() => checkAccepts(noop, { 1: {} })).toThrow("notes accepts[1].state.reader: missing mapping"); + expect(() => checkAccepts(noop, { 1: { state: { reader: { notes: (current) => current } } } })).not.toThrow(); + const reverted = evolve(v2, (e) => e.state("editor", "notes", arr(Note))); + expect(() => checkAccepts(reverted, { 1: { state: { editor: { notes: (current) => current } } } })).not.toThrow(); + const readded = evolve(NotesV1, (e) => e.removeState("editor", "count").addState("editor", "count", int())); + expect(() => checkAccepts(readded, { 1: { state: { editor: { count: (current) => current } } } })).not.toThrow(); + }); +}); + +describe("method mappings", () => { + it("changed args map old to current; a changed result maps current to old; both need both", () => { + const args = evolve(NotesV1, (e) => e.args("add", NoteV2)); + expect(() => + checkAccepts(args, { 1: { methods: { add: { args: (old) => ({ ...old, tag: "" }) } } } }), + ).not.toThrow(); + expect(() => + // @ts-expect-error: only args changed + checkAccepts(args, { 1: { methods: { add: { args: (old) => ({ ...old, tag: "" }), result: (r) => r } } } }), + ).toThrow("notes accepts[1].methods.add.result: unnecessary mapping"); + // @ts-expect-error: args must produce the current shape + checkAccepts(args, { 1: { methods: { add: { args: (old) => old } } } }); + + const result = evolve(NotesV1, (e) => e.result("add", int())); + expect(() => + checkAccepts(result, { 1: { methods: { add: { result: (current) => String(current) } } } }), + ).not.toThrow(); + // @ts-expect-error: the result mapping is missing + expect(() => checkAccepts(result, { 1: { methods: { add: {} } } })).toThrow( + "notes accepts[1].methods.add.result: missing mapping", + ); + + const both = evolve(NotesV1, (e) => e.method("add", NoteV2, int())); + const accepts: AcceptsTable = { + 1: { methods: { add: { args: (old) => ({ ...old, tag: "" }), result: (current) => String(current) } } }, + }; + expect(() => checkAccepts(both, accepts)).not.toThrow(); + // @ts-expect-error: both sides changed + expect(() => checkAccepts(both, { 1: { methods: { add: { args: (old) => ({ ...old, tag: "" }) } } } })).toThrow( + "notes accepts[1].methods.add.result: missing mapping", + ); + }); + + it("a removed method takes a call replacement or refuses", () => { + const v2 = evolve(NotesV1, (e) => e.removeMethod("clear")); + const call = async (args: null) => args; + expect(() => checkAccepts(v2, { 1: { methods: { clear: { call } } } })).not.toThrow(); + expect(() => checkAccepts(v2, { 1: { methods: { clear: "refuse" } } })).not.toThrow(); + // @ts-expect-error: a removed method has no args mapping + expect(() => checkAccepts(v2, { 1: { methods: { clear: { args: (old) => old } } } })).toThrow( + 'notes accepts[1].methods.clear: expected { call } or "refuse"', + ); + // @ts-expect-error: the removed method needs a replacement or a refusal + expect(() => checkAccepts(v2, { 1: {} })).toThrow("notes accepts[1].methods.clear: missing mapping"); + }); + + it("a method removed from one variant that had it counts as removed", () => { + const v2 = evolve(NotesV1, (e) => e.removeMethod("add").addMethod("editor", "add", Note, str())); + // @ts-expect-error: reader lost `add` + expect(() => checkAccepts(v2, { 1: { methods: { add: { args: (old) => old, result: (r) => r } } } })).toThrow( + 'notes accepts[1].methods.add: expected { call } or "refuse"', + ); + expect(() => checkAccepts(v2, { 1: { methods: { add: "refuse" } } })).not.toThrow(); + }); + + it("call and refuse are also allowed for changed methods", () => { + const v2 = evolve(NotesV1, (e) => e.args("add", NoteV2)); + expect(() => checkAccepts(v2, { 1: { methods: { add: { call: async (old) => old.text } } } })).not.toThrow(); + expect(() => checkAccepts(v2, { 1: { methods: { add: "refuse" } } })).not.toThrow(); + expect(() => + // @ts-expect-error: call cannot be combined with args + checkAccepts(v2, { 1: { methods: { add: { call: async () => "", args: (old) => ({ ...old, tag: "" }) } } } }), + ).toThrow('notes accepts[1].methods.add: expected { call } or "refuse"'); + // @ts-expect-error: call returns the old result type + checkAccepts(v2, { 1: { methods: { add: { call: async () => 1 } } } }); + }); + + it("added methods and methods added to a new variant need nothing", () => { + const v2 = evolve(NotesV1, (e) => e.addMethod("reader", "clear", nul(), nul())); + expect(checkAccepts(v2, { 1: {} }).versions).toHaveLength(1); + // @ts-expect-error: unnecessary mapping + expect(() => checkAccepts(v2, { 1: { methods: { clear: "refuse" } } })).toThrow( + "notes accepts[1].methods.clear: unnecessary mapping", + ); + }); + + it("refuses wrong shapes", () => { + const v2 = evolve(NotesV1, (e) => e.args("add", NoteV2)); + expect(() => checkUntyped(v2, { 1: { methods: { add: { args: 1 } } } })).toThrow( + "notes accepts[1].methods.add.args: expected a function", + ); + expect(() => checkUntyped(v2, { 1: { methods: { add: { args: () => 0, extra: 1 } } } })).toThrow( + 'notes accepts[1].methods.add: unknown key "extra"', + ); + expect(() => checkUntyped(v2, { 1: { methods: { add: { args: () => 0 } } }, other: 1 })).toThrow( + "notes accepts[other]: not an older version", + ); + expect(() => checkUntyped(v2, { 1: { methods: { add: { args: () => 0 } }, extra: {} } })).toThrow( + 'notes accepts[1]: unknown key "extra"', + ); + expect(() => checkUntyped(v2, null)).toThrow("notes accepts: expected an object"); + }); +}); + +describe("versions", () => { + it("refuses unknown versions and the current version", () => { + const v2 = evolve(NotesV1, (e) => e); + // @ts-expect-error: version 3 does not exist + expect(() => checkAccepts(v2, { 3: {} })).toThrow("notes accepts[3]: not an older version"); + // @ts-expect-error: version 2 is current + expect(() => checkAccepts(v2, { 2: {} })).toThrow("notes accepts[2]: not an older version"); + expect(checkAccepts(v2, {})).toEqual({ versions: [] }); + }); + + it("maps directly over several versions: every edit after the accepted version counts", () => { + const v2 = evolve(NotesV1, (e) => e.args("add", NoteV2)); + const v3 = evolve(v2, (e) => e.result("add", int()).removeState("editor", "count")); + const accepts: AcceptsTable = { + 1: { + state: { editor: { count: "refuse" } }, + methods: { add: { args: (old) => ({ ...old, tag: "" }), result: (current) => String(current) } }, + }, + 2: { + state: { editor: { count: (state) => state.notes.length } }, + methods: { add: { result: (r) => `${r}` } }, + }, + }; + expect(checkAccepts(v3, accepts).versions).toEqual([ + { version: 1, fingerprint: NotesV1.fingerprint }, + { version: 2, fingerprint: v2.fingerprint }, + ]); + const fromV1 = { + state: { editor: { count: "refuse" } }, + methods: { add: { result: (r: number) => `${r}` } }, + } as const; + // @ts-expect-error: from v1, args changed too + expect(() => checkAccepts(v3, { 1: fromV1 })).toThrow("notes accepts[1].methods.add.args: missing mapping"); + }); + + it("never calls a mapping", () => { + const v2 = evolve(NotesV1, (e) => e.args("add", NoteV2).removeMethod("clear").state("editor", "count", str())); + const mappings = { args: vi.fn(), call: vi.fn(), count: vi.fn() }; + checkUntyped(v2, { + 1: { + state: { editor: { count: mappings.count } }, + methods: { add: { args: mappings.args }, clear: { call: mappings.call } }, + }, + }); + expect(Object.values(mappings).map((mapping) => mapping.mock.calls.length)).toEqual([0, 0, 0]); + }); + + it("local tables are flat: state without a variant level", () => { + const v1 = local("info", { version: 1, params: {}, members: { title: str(), save: method(str()) } }); + const v2 = evolve(v1, (e) => e.state("title", int()).args("save", int())); + const accepts: AcceptsTable = { + 1: { state: { title: (current) => String(current) }, methods: { save: { args: (old) => old.length } } }, + }; + expect(() => checkAccepts(v2, accepts)).not.toThrow(); + expect(() => + checkUntyped(v2, { 1: { state: { members: { title: String } }, methods: { save: "refuse" } } }), + ).toThrow("info accepts[1].state.members: unnecessary mapping"); + }); +}); diff --git a/packages/chord/test/next/contracts/contract.test.ts b/packages/chord/test/next/contracts/contract.test.ts new file mode 100644 index 000000000..c9a6f12ab --- /dev/null +++ b/packages/chord/test/next/contracts/contract.test.ts @@ -0,0 +1,115 @@ +import { describe, expect, expectTypeOf, it } from "vitest"; +import { BACKGROUND_CONTEXT, type Context } from "../../../src/context/index.ts"; +import { type Contract, contractData } from "../../../src/next/contracts/contract.ts"; +import { + arr, + bytes, + evolve, + type Implementation, + int, + local, + method, + nul, + obj, + type ParamsOf, + remote, + str, + union, +} from "../../../src/next/contracts/index.ts"; +import { checkAccepts, currentVariants, LOCAL, takesNoArgs } from "../../../src/next/contracts/internal.ts"; +import { bindProvider, checkParams } from "../../../src/next/data/internal.ts"; +import type { Watchable } from "../../../src/next/state/index.ts"; +import { expectCollected } from "../support.ts"; + +const Notes = remote("notes", { + version: 1, + params: { session: str(), page: int() }, + variants: { + editor: { state: { notes: arr(str()) }, methods: { add: method(obj({ text: str() }), str()), clear: method() } }, + reader: { state: { notes: arr(str()) } }, + }, +}); + +describe("contract handles", () => { + it("carry kind, name, version and fingerprint only", () => { + expect(Object.keys(Notes)).toEqual(["kind", "name", "version", "fingerprint"]); + }); + + it("are branded: spread and hand-built copies are refused", () => { + const copy = { ...Notes }; + const refusal = "notes: not a declared contract (spread and hand-built copies are refused)"; + expect(() => contractData(copy)).toThrow(refusal); + expect(() => evolve(copy, (e) => e)).toThrow(refusal); + expect(() => checkAccepts(copy, {})).toThrow(refusal); + expect(() => bindProvider(copy, {})).toThrow(refusal); + expect(() => checkParams(copy, { session: "s", page: 1 }, BACKGROUND_CONTEXT)).toThrow(refusal); + }); + + it("are collected when dropped, with their data", async () => { + const refs = (() => { + const contract = local("gc", { version: 1, params: {}, members: { s: str() } }); + return [new WeakRef(contract), new WeakRef(contractData(contract))]; + })(); + await expectCollected(...refs); + }); +}); + +/** The members of one variant of a remote implementation. */ +type Members = Extract, { readonly variant: V }>["members"]; + +describe("derived types", () => { + it("Implementation: one variant with its members, state watchables and methods taking args and a context", () => { + expectTypeOf["variant"]>().toEqualTypeOf<"editor" | "reader">(); + type Editor = Members; + expectTypeOf().toEqualTypeOf>(); + expectTypeOf().toEqualTypeOf<(args: { text: string }, context: Context) => Promise>(); + // A method without arguments omits them. + expectTypeOf().toEqualTypeOf<(context: Context) => Promise>(); + expectTypeOf>().toEqualTypeOf<"notes">(); + }); + + it("Implementation of a local contract is flat", () => { + const Info = local("info", { + version: 1, + params: {}, + members: { title: str(), blob: bytes(), save: method(str()) }, + }); + type Info = Implementation; + expectTypeOf().toEqualTypeOf<"title" | "blob" | "save">(); + expectTypeOf().toEqualTypeOf<(args: string, context: Context) => Promise>(); + const Counts = local("counts", { + version: 1, + params: {}, + members: { reset: method(), total: method(undefined, int()), maybe: method(union([nul(), str()])) }, + }); + type Counts = Implementation; + expectTypeOf().toEqualTypeOf<(context: Context) => Promise>(); + expectTypeOf().toEqualTypeOf<(context: Context) => Promise>(); + // Only a `null` argument schema is omitted; a union with null is an argument. + expectTypeOf().toEqualTypeOf<(args: string | null, context: Context) => Promise>(); + }); + + it("takesNoArgs: exactly the methods whose argument schema is null", () => { + const Counts = local("counts", { + version: 1, + params: {}, + members: { + reset: method(), + total: method(undefined, int()), + set: method(int()), + maybe: method(union([nul(), str()])), + }, + }); + const methods = [...currentVariants(Counts).get(LOCAL)!.methods].map(([name, data]) => [name, takesNoArgs(data)]); + expect(methods).toEqual([ + ["reset", true], + ["total", true], + ["set", false], + ["maybe", false], + ]); + }); + + it("ParamsOf: the params object", () => { + expectTypeOf>().toEqualTypeOf<{ readonly session: string; readonly page: number }>(); + }); +}); diff --git a/packages/chord/test/next/contracts/declare.test.ts b/packages/chord/test/next/contracts/declare.test.ts new file mode 100644 index 000000000..016fb2f90 --- /dev/null +++ b/packages/chord/test/next/contracts/declare.test.ts @@ -0,0 +1,228 @@ +import Type from "typebox"; +import { describe, expect, it } from "vitest"; +import { arr, evolve, int, keyed, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { currentVariants, LOCAL } from "../../../src/next/contracts/internal.ts"; + +const Note = obj({ text: str() }); + +/** Declares through an untyped door, as a JavaScript caller would. */ +const untyped = { + remote: remote as (name: unknown, declaration: unknown) => unknown, + local: local as (name: unknown, declaration: unknown) => unknown, +}; + +describe("declaration checks", () => { + it("accept a remote and a local declaration", () => { + const Notes = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: arr(Note) }, methods: { add: method(Note, str()), clear: method() } }, + reader: { state: { notes: arr(Note) } }, + }, + }); + expect(Notes).toMatchObject({ kind: "remote", name: "notes", version: 1 }); + expect(Notes.fingerprint).toMatch(/^[0-9a-f]{16}$/); + const Info = local("app.session-info", { + version: 3, + params: {}, + members: { init: obj({ title: str() }), clear: method() }, + }); + expect(Info).toMatchObject({ kind: "local", name: "app.session-info", version: 3 }); + }); + + it.each([ + ["empty name", "", { version: 1, params: {}, variants: { a: {} } }, 'A contract name "": expected lowercase'], + [ + "name not a string", + 1, + { version: 1, params: {}, variants: { a: {} } }, + "A contract name 1: expected lowercase", + ], + ["an uppercase name", "Notes", { version: 1, params: {}, variants: { a: {} } }, "expected lowercase"], + ["a name with a colon", "a:b", { version: 1, params: {}, variants: { a: {} } }, "expected lowercase"], + ["version zero", "n", { version: 0, params: {}, variants: { a: {} } }, "n.version: expected a positive integer"], + [ + "fractional version", + "n", + { version: 1.5, params: {}, variants: { a: {} } }, + "n.version: expected a positive integer", + ], + [ + "unsafe version", + "n", + { version: 2 ** 53, params: {}, variants: { a: {} } }, + "n.version: expected a positive integer", + ], + [ + "unknown declaration key", + "n", + { version: 1, params: {}, variants: { a: {} }, place: "x" }, + 'n: unknown key "place"', + ], + [ + "a placement that is no function", + "n", + { version: 1, params: {}, variants: { a: {} }, placement: "x" }, + "n.placement: expected a function of the params", + ], + ["no variants", "n", { version: 1, params: {}, variants: {} }, "n.variants: at least one variant is required"], + ["missing params", "n", { version: 1, variants: { a: {} } }, "n.params: expected an object"], + [ + "unknown variant key", + "n", + { version: 1, params: {}, variants: { a: { members: {} } } }, + 'n.a: unknown key "members"', + ], + [ + "a schema under methods", + "n", + { version: 1, params: {}, variants: { a: { methods: { m: str() } } } }, + "n.a.methods.m: expected method(...)", + ], + [ + "a method under state", + "n", + { version: 1, params: {}, variants: { a: { state: { s: method() } } } }, + "n.a.state.s: a method belongs in methods", + ], + [ + "bad state schema", + "n", + { version: 1, params: {}, variants: { a: { state: { s: { not: {} } } } } }, + 'n.a.state.s: unsupported keyword "not"', + ], + [ + "open arguments", + "n", + { version: 1, params: {}, variants: { a: { methods: { m: method(Type.Object({})) } } } }, + "n.a.methods.m.args: argument objects must be closed", + ], + [ + "bad result", + "n", + { version: 1, params: {}, variants: { a: { methods: { m: method(undefined, { not: {} }) } } } }, + 'n.a.methods.m.result: unsupported keyword "not"', + ], + ])("remote refuses %s", (_case, name, declaration, message) => { + expect(() => untyped.remote(name, declaration)).toThrow(message); + }); + + it.each([ + ["unknown key", { version: 1, params: {}, members: {}, variants: {} }, 'i: unknown key "variants"'], + ["a placement", { version: 1, params: {}, members: {}, placement: () => null }, 'i: unknown key "placement"'], + [ + "bad member schema", + { version: 1, params: {}, members: { s: { not: {} } } }, + 'i.members.s: unsupported keyword "not"', + ], + [ + "open arguments", + { version: 1, params: {}, members: { m: method(Type.Object({})) } }, + "i.members.m.args: argument objects must be closed", + ], + ])("local refuses %s", (_case, declaration, message) => { + expect(() => untyped.local("i", declaration)).toThrow(message); + }); + + it("does not check state for closed objects", () => { + expect(() => local("i", { version: 1, params: {}, members: { s: Type.Object({ a: str() }) } })).not.toThrow(); + }); +}); + +describe("D11: members across variants", () => { + it("refuses a name that is state in one variant and a method in another", () => { + expect(() => + remote("n", { + version: 1, + params: {}, + // @ts-expect-error: `x` is a method in variant b + variants: { a: { state: { x: str() } }, b: { methods: { x: method() } } }, + }), + ).toThrow('n.b.methods.x: "x" is a state member; its kind never changes'); + }); + + it("refuses one name as state and method within a variant", () => { + expect(() => + remote("n", { + version: 1, + params: {}, + // @ts-expect-error: `x` is both state and a method + variants: { a: { state: { x: str() }, methods: { x: method() } } }, + }), + ).toThrow('n.a.methods.x: "x" is a state member'); + }); + + it("refuses a method with different signatures in two variants", () => { + expect(() => + remote("n", { + version: 1, + params: {}, + // @ts-expect-error: `m` takes a string in a and an integer in b + variants: { a: { methods: { m: method(str()) } }, b: { methods: { m: method(int()) } } }, + }), + ).toThrow("n.b.methods.m: a method has the same signature in every variant"); + expect(() => + remote("n", { + version: 1, + params: {}, + variants: { + a: { methods: { m: method(str(), str()) } }, + b: { methods: { m: method(str({ max: 3 }), str()) } }, + }, + }), + ).toThrow("n.b.methods.m: a method has the same signature in every variant"); + }); + + it("allows state schemas to differ per variant (redaction)", () => { + expect(() => + remote("n", { + version: 1, + params: {}, + variants: { + editor: { state: { notes: arr(obj({ text: str(), author: str() })) }, methods: { m: method(str()) } }, + reader: { state: { notes: arr(obj({ text: str() })) }, methods: { m: method(str()) } }, + }, + }), + ).not.toThrow(); + }); +}); + +describe("reserved member names", () => { + it.each(["then", "release", "variant", "state", "retry"])("refuses %s in remote, local and evolve", (name) => { + expect(() => remote("n", { version: 1, params: {}, variants: { v: { state: { [name]: str() } } } })).toThrow( + `n.v.state.${name}: "${name}" is reserved: a reference has it`, + ); + expect(() => local("n", { version: 1, params: {}, members: { [name]: method() } })).toThrow( + `n.members.${name}: "${name}" is reserved: a reference has it`, + ); + const base = local("n", { version: 1, params: {}, members: {} }); + expect(() => evolve(base, (e) => e.addState(name, str()))).toThrow( + `n v2.members.${name}: "${name}" is reserved: a reference has it`, + ); + }); +}); + +describe("keyed members (slice 6a §12)", () => { + it("declare a keyed member in a local contract: its own kind and a node of its own in the fingerprint", () => { + const Keyed = local("app.store", { version: 1, params: {}, members: { notes: keyed(Note), add: method() } }); + const State = local("app.store", { version: 1, params: {}, members: { notes: Note, add: method() } }); + const variant = currentVariants(Keyed).get(LOCAL)!; + expect([[...variant.keyed.keys()], [...variant.state.keys()]]).toEqual([["notes"], []]); + expect(Keyed.fingerprint).not.toBe(State.fingerprint); + // Evolving the contract carries its keyed members over unchanged (their versions are slice 7's). + const next = evolve(Keyed, (e) => e.addMethod("clear", str(), str())); + expect(currentVariants(next).get(LOCAL)!.keyed.has("notes")).toBe(true); + }); + + it("refuse a keyed member in a remote contract, at the type level and at declaration", () => { + expect(() => + remote("app.notes", { + version: 1, + params: {}, + // @ts-expect-error keyed members are only in local contracts + variants: { editor: { state: { notes: keyed(Note) } } }, + }), + ).toThrow("app.notes.editor.state.notes: keyed members are only in local contracts"); + }); +}); diff --git a/packages/chord/test/next/contracts/evolve.test.ts b/packages/chord/test/next/contracts/evolve.test.ts new file mode 100644 index 000000000..abc2263e3 --- /dev/null +++ b/packages/chord/test/next/contracts/evolve.test.ts @@ -0,0 +1,217 @@ +import Type from "typebox"; +import { describe, expect, expectTypeOf, it } from "vitest"; +import type { Context } from "../../../src/context/index.ts"; +import { type Contract, contractData, type ShapeOf } from "../../../src/next/contracts/contract.ts"; +import type { CurrentStep } from "../../../src/next/contracts/history.ts"; +import { + arr, + bool, + evolve, + type Implementation, + int, + local, + method, + nul, + obj, + remote, + type Static, + str, +} from "../../../src/next/contracts/index.ts"; + +const Note = obj({ text: str() }); + +const NotesV1 = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(Note, str()), clear: method() } }, + reader: { state: { notes: arr(Note) }, methods: { add: method(Note, str()) } }, + }, +}); + +const InfoV1 = local("info", { version: 1, params: {}, members: { title: str(), clear: method() } }); + +/** The current members of a contract's variant, as Chord holds them. */ +function current(contract: Contract, variant: string) { + const data = contractData(contract); + const members = data.versions.at(-1)!.variants.get(variant)!; + return { state: [...members.state.keys()], methods: [...members.methods.keys()], data }; +} + +/** The current version of a contract's type-level history. */ +type VersionOf = CurrentStep["history"]>["version"]; + +/** The members of one variant of a remote implementation. */ +type Members = Extract, { readonly variant: V }>["members"]; + +describe("edits", () => { + it("state changes one state member of one variant", () => { + const v2 = evolve(NotesV1, (e) => e.state("editor", "count", str())); + expect(v2.version).toBe(2); + expectTypeOf["count"]["value"]>().toEqualTypeOf(); + expectTypeOf["notes"]["value"]>().toEqualTypeOf<{ text: string }[]>(); + // @ts-expect-error: `missing` is not a state member of editor + expect(() => evolve(NotesV1, (e) => e.state("editor", "missing", str()))).toThrow( + "notes v2.editor.state.missing: no such state member", + ); + }); + + it("addState adds a state member", () => { + const v2 = evolve(NotesV1, (e) => e.addState("reader", "count", int())); + expect(current(v2, "reader").state).toEqual(["notes", "count"]); + expectTypeOf["count"]["value"]>().toEqualTypeOf(); + expect(() => evolve(NotesV1, (e) => e.addState("reader", "notes", int()))).toThrow( + "notes v2.reader.state.notes: already declared", + ); + }); + + it("removeState removes a state member", () => { + const v2 = evolve(NotesV1, (e) => e.removeState("editor", "count")); + expect(current(v2, "editor").state).toEqual(["notes"]); + expectTypeOf>().toEqualTypeOf<"notes" | "add" | "clear">(); + }); + + it("args and result change one side of a method in every variant that has it", () => { + const v2 = evolve(NotesV1, (e) => e.args("add", obj({ text: str(), tag: str() })).result("add", int())); + const { data } = current(v2, "editor"); + const editor = data.versions[1]!.variants.get("editor")!.methods.get("add")!; + const reader = data.versions[1]!.variants.get("reader")!.methods.get("add")!; + expect(editor).toEqual(reader); + expect(data.versions[1]!.changes).toEqual([ + ["editor", "add", "args"], + ["reader", "add", "args"], + ["editor", "add", "result"], + ["reader", "add", "result"], + ]); + type Add = Members["add"]; + expectTypeOf[0]>().toEqualTypeOf<{ text: string; tag: string }>(); + expectTypeOf>>().toEqualTypeOf(); + }); + + it("method replaces both sides", () => { + const v2 = evolve(NotesV1, (e) => e.method("clear", obj({ all: bool() }), bool())); + type Clear = Members["clear"]; + expectTypeOf[0]>().toEqualTypeOf<{ all: boolean }>(); + expectTypeOf>>().toEqualTypeOf(); + // @ts-expect-error: no method `missing` + expect(() => evolve(NotesV1, (e) => e.method("missing", nul(), nul()))).toThrow('notes v2: no method "missing"'); + }); + + it("addMethod adds a method; to another variant only with the same signature", () => { + const v2 = evolve(NotesV1, (e) => e.addMethod("reader", "clear", nul(), nul())); + expect(current(v2, "reader").methods).toEqual(["add", "clear"]); + expect(() => evolve(NotesV1, (e) => e.addMethod("reader", "clear", str(), nul()))).toThrow( + "notes v2.reader.methods.clear: a method has the same signature in every variant", + ); + expect(() => evolve(NotesV1, (e) => e.addMethod("reader", "add", nul(), nul()))).toThrow( + "notes v2.reader.methods.add: already declared", + ); + }); + + it("removeMethod removes a method from every variant", () => { + const v2 = evolve(NotesV1, (e) => e.removeMethod("add")); + expect(current(v2, "editor").methods).toEqual(["clear"]); + expect(current(v2, "reader").methods).toEqual([]); + expectTypeOf>().toEqualTypeOf<"notes">(); + }); + + it("local edits take no variant", () => { + const v2 = evolve(InfoV1, (e) => + e + .state("title", int()) + .addState("body", str()) + .addMethod("save", obj({ force: bool() }), nul()) + .removeMethod("clear"), + ); + expect(current(v2, "members")).toMatchObject({ state: ["title", "body"], methods: ["save"] }); + expectTypeOf>().toEqualTypeOf<"title" | "body" | "save">(); + const v3 = evolve(v2, (e) => e.removeState("body").args("save", nul())); + // Arguments changed to `null`: the method omits them now. + expectTypeOf["save"]>>().toEqualTypeOf<[context: Context]>(); + }); + + it("checks every edit: closed arguments and the vocabulary", () => { + expect(() => evolve(NotesV1, (e) => e.args("add", Type.Object({ text: str() })))).toThrow( + "notes v2.editor.methods.add.args: argument objects must be closed", + ); + expect(() => evolve(InfoV1, (e) => e.addMethod("m", Type.Object({}), nul()))).toThrow( + "info v2.members.m.args: argument objects must be closed", + ); + expect(() => evolve(NotesV1, (e) => e.state("editor", "count", { not: {} } as never))).toThrow( + 'notes v2.editor.state.count: unsupported keyword "not"', + ); + expect(() => evolve(NotesV1, (e) => e.state("editor", "count", Type.Object({})))).not.toThrow(); + }); +}); + +describe("kinds and fixed parts", () => { + it("a member's kind never changes, also not after a removal and re-add", () => { + expect(() => + evolve(NotesV1, (e) => e.removeState("editor", "count").addMethod("editor", "count", nul(), nul())), + ).toThrow('notes v2.editor.methods.count: "count" is a state member; its kind never changes'); + const v2 = evolve(NotesV1, (e) => e.removeMethod("clear")); + const v3 = evolve(v2, (e) => e.state("editor", "count", str())); + expect(() => evolve(v3, (e) => e.addState("reader", "clear", str()))).toThrow( + 'notes v4.reader.state.clear: "clear" is a method member; its kind never changes', + ); + }); + + it("the variant set never changes", () => { + // @ts-expect-error: no variant `admin` + expect(() => evolve(NotesV1, (e) => e.addState("admin", "x", str()))).toThrow( + "notes v2.admin: unknown variant (the variant set never changes)", + ); + }); + + it("a new base is a full declaration with an explicit version and no older versions", () => { + const base = remote("notes", { version: 5, params: { session: str(), tenant: str() }, variants: { editor: {} } }); + expect(contractData(base).versions.map(({ version }) => version)).toEqual([5]); + expect(evolve(base, (e) => e).version).toBe(6); + }); + + it("evolve must return the editor it was given", () => { + const other = evolve(NotesV1, (e) => e); + expect(other.version).toBe(2); + let foreign: unknown; + evolve(InfoV1, (e) => { + foreign = e; + return e; + }); + expect(() => evolve(NotesV1, () => foreign as never)).toThrow( + "notes v2: evolve must return the editor it was given", + ); + }); + + it("leaves the previous contract untouched", () => { + evolve(NotesV1, (e) => e.removeState("editor", "count").removeMethod("add")); + expect(current(NotesV1, "editor")).toMatchObject({ state: ["notes", "count"], methods: ["add", "clear"] }); + }); +}); + +describe("long histories", () => { + it("count versions beyond 9 and 99, at runtime and in the types", () => { + let contract: Contract = InfoV1; + for (let step = 0; step < 120; step++) contract = evolve(contract, (e) => e); + expect(contract.version).toBe(121); + expect(contractData(contract).versions).toHaveLength(121); + + const v9 = local("i", { version: 9, params: {}, members: { s: str() } }); + const v10 = evolve(v9, (e) => e); + const v99 = local("i", { version: 99, params: {}, members: { s: str() } }); + const v100 = evolve(v99, (e) => e); + const v101 = evolve(v100, (e) => e.state("s", int())); + expectTypeOf>().toEqualTypeOf<"10">(); + expectTypeOf>().toEqualTypeOf<"100">(); + expectTypeOf>().toEqualTypeOf<"101">(); + }); + + it("the types track every step with its changes", () => { + const v2 = evolve(NotesV1, (e) => e.state("editor", "count", str()).addMethod("reader", "clear", nul(), nul())); + type History = ShapeOf["history"]; + expectTypeOf().toEqualTypeOf<2>(); + expectTypeOf().toEqualTypeOf< + ["editor", "count", "state"] | ["reader", "clear", "args" | "result"] + >(); + expectTypeOf>().toEqualTypeOf(); + }); +}); diff --git a/packages/chord/test/next/contracts/fingerprint.test.ts b/packages/chord/test/next/contracts/fingerprint.test.ts new file mode 100644 index 000000000..d28e360fb --- /dev/null +++ b/packages/chord/test/next/contracts/fingerprint.test.ts @@ -0,0 +1,80 @@ +import Type, { type TSchema } from "typebox"; +import { describe, expect, it } from "vitest"; +import { evolve, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { NORMAL_FORM_VERSION } from "../../../src/next/contracts/internal.ts"; +import type { Params } from "../../../src/next/contracts/params.ts"; + +/** Fingerprint of a one-variant remote contract with state `s` and params `params`. */ +function fingerprintOf(state: TSchema, params: Params = {}): string { + return remote("n", { version: 1, params, variants: { a: { state: { s: state } } } }).fingerprint; +} + +describe("fingerprints", () => { + it("pin the normal form: a fixed contract has a fixed hash", () => { + expect(NORMAL_FORM_VERSION).toBe(2); + const Notes = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: obj({ text: str() }) }, methods: { add: method(obj({ text: str() }), str()) } }, + reader: { state: { notes: obj({ text: str() }) } }, + }, + }); + expect(Notes.fingerprint).toBe("b3eda86070a7e319"); + }); + + it("are stable across key order and annotations", () => { + const plain = { + type: "object", + properties: { a: { type: "string" }, b: { type: "integer" } }, + required: ["a", "b"], + }; + const reordered = { + required: ["b", "a"], + title: "T", + properties: { + b: { type: "integer", description: "d" }, + a: { $comment: "c", type: "string", examples: ["x"] }, + }, + type: "object", + deprecated: true, + }; + expect(fingerprintOf(reordered)).toBe(fingerprintOf(plain)); + }); + + it.each([ + ["defaults", Type.String(), Type.String({ default: "x" })], + ["bounds", Type.String(), Type.String({ maxLength: 3 })], + ["formats", Type.String(), Type.String({ format: "email" })], + ["union order", Type.Union([Type.String(), Type.Integer()]), Type.Union([Type.Integer(), Type.String()])], + ["additionalProperties", Type.Object({}), Type.Object({}, { additionalProperties: false })], + ["readOnly", Type.String(), Type.String({ readOnly: true })], + ])("are sensitive to %s", (_case, a, b) => { + expect(fingerprintOf(a)).not.toBe(fingerprintOf(b)); + }); + + it("are sensitive to params order", () => { + expect(fingerprintOf(str(), { a: str(), b: int() })).not.toBe(fingerprintOf(str(), { b: int(), a: str() })); + }); + + it("give the same v1 from a separate declaration, and evolve keeps v1's", () => { + const declare = () => remote("n", { version: 1, params: {}, variants: { a: { state: { s: str() } } } }); + const v1 = declare(); + const v2 = evolve(v1, (e) => e.state("a", "s", int())); + expect(declare().fingerprint).toBe(v1.fingerprint); + expect(v2.fingerprint).not.toBe(v1.fingerprint); + expect(evolve(declare(), (e) => e.state("a", "s", int())).fingerprint).toBe(v2.fingerprint); + }); + + it("differ between local and remote", () => { + const asRemote = remote("n", { version: 1, params: {}, variants: { members: { state: { s: str() } } } }); + const asLocal = local("n", { version: 1, params: {}, members: { s: str() } }); + expect(asLocal.fingerprint).not.toBe(asRemote.fingerprint); + }); + + it("do not depend on the name or version", () => { + const a = remote("a", { version: 1, params: {}, variants: { v: {} } }); + const b = remote("b", { version: 7, params: {}, variants: { v: {} } }); + expect(a.fingerprint).toBe(b.fingerprint); + }); +}); diff --git a/packages/chord/test/next/contracts/history.test.ts b/packages/chord/test/next/contracts/history.test.ts new file mode 100644 index 000000000..9d7ab7ce2 --- /dev/null +++ b/packages/chord/test/next/contracts/history.test.ts @@ -0,0 +1,37 @@ +import { describe, expectTypeOf, it } from "vitest"; +import type { + ChangesAfter, + CurrentStep, + Increment, + OlderVersions, + StepAt, +} from "../../../src/next/contracts/history.ts"; + +type Empty = { readonly a: { readonly state: Record; readonly methods: Record } }; +type History = [ + { version: "9"; variants: Empty; changes: never }, + { version: "10"; variants: Empty; changes: ["a", "x", "state"] }, + { version: "11"; variants: Empty; changes: ["a", "m", "args"] }, +]; + +describe("type-level history", () => { + it("increments decimal versions", () => { + expectTypeOf>().toEqualTypeOf<"2">(); + expectTypeOf>().toEqualTypeOf<"10">(); + expectTypeOf>().toEqualTypeOf<"20">(); + expectTypeOf>().toEqualTypeOf<"100">(); + expectTypeOf>().toEqualTypeOf<"1100">(); + }); + + it("finds the current step, older versions and steps by version", () => { + expectTypeOf["version"]>().toEqualTypeOf<"11">(); + expectTypeOf>().toEqualTypeOf<"9" | "10">(); + expectTypeOf["changes"]>().toEqualTypeOf<["a", "x", "state"]>(); + }); + + it("collects every change after a version", () => { + expectTypeOf>().toEqualTypeOf<["a", "x", "state"] | ["a", "m", "args"]>(); + expectTypeOf>().toEqualTypeOf<["a", "m", "args"]>(); + expectTypeOf>().toEqualTypeOf(); + }); +}); diff --git a/packages/chord/test/next/contracts/host-kind.test.ts b/packages/chord/test/next/contracts/host-kind.test.ts new file mode 100644 index 000000000..3414ffd97 --- /dev/null +++ b/packages/chord/test/next/contracts/host-kind.test.ts @@ -0,0 +1,80 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { evolve, hostKind, int, oneOf, opt, remote, str } from "../../../src/next/contracts/index.ts"; +import { contractData, kindParams } from "../../../src/next/contracts/internal.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { checkAddress } from "../../../src/next/data/internal.ts"; +import { expectCollected } from "../support.ts"; + +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); +const Shard = hostKind("shard", { region: oneOf("eu", "us"), n: int({ min: 0 }) }); + +describe("host kinds", () => { + it("make addresses: the kind, the params and a key built like an identity key", () => { + expect(Server()).toEqual({ kind: "server", params: {}, key: "server" }); + expect(Session({ id: "S1" })).toEqual({ kind: "session", params: { id: "S1" }, key: "session|S1" }); + expect(Shard({ region: "eu", n: 3 }).key).toBe("shard|eu|3"); + // Escaped like identity keys: two different param lists never make one key. + expect(Session({ id: "a|b" }).key).not.toBe(Shard({ region: "a" as "eu", n: "b" as unknown as number }).key); + expect(Session({ id: "a|b" }).key).toBe("session|a\\|b"); + }); + + it("follow the params policy of contracts at declaration", () => { + // @ts-expect-error: params fields are required + expect(() => hostKind("k", { id: opt(str()) })).toThrow("k.params.id: a param is"); + expect(() => hostKind("")).toThrow(`A host kind's name "": expected lowercase [a-z0-9@/._~-]`); + expect(() => hostKind("Session")).toThrow("expected lowercase"); + expect(() => hostKind("a:b")).toThrow("expected lowercase"); + expect(() => hostKind("client")).toThrow('"client" is reserved for hosts without an address'); + expect(hostKind("@pi/session.v2_~-").kind).toBe("@pi/session.v2_~-"); + }); + + it("check an address's params where it is routed, with the kind's compiled check: invalid/placement", () => { + const params = kindParams(Session); + expect(checkAddress("session", params, { id: "S1" }, context).key).toBe("session|S1"); + for (const bad of [{ id: 1 }, {}, { id: "S1", extra: 1 }, null]) { + let thrown: unknown; + try { + checkAddress("session", params, bad, context); + } catch (error) { + thrown = error; + } + expect(isChordError(thrown, "invalid") && thrown.reason).toBe("placement"); + } + }); + + it("keep their params data only while the kind is referenced", async () => { + const ref = (() => { + const Temp = hostKind("temp", { id: str() }); + kindParams(Temp); + return new WeakRef(Temp); + })(); + await expectCollected(ref); + }); +}); + +describe("placement", () => { + const variants = { v: { state: { n: int() } } }; + + it("is not part of the fingerprint and is carried by evolve", () => { + const placement = ({ session }: { session: string }) => Session({ id: session }); + const placed = remote("app.placed", { version: 1, params: { session: str() }, placement, variants }); + const unplaced = remote("app.placed", { version: 1, params: { session: str() }, variants }); + expect(placed.fingerprint).toBe(unplaced.fingerprint); + expect(contractData(placed).placement).toBe(placement); + expect(contractData(unplaced).placement).toBeUndefined(); + const next = evolve(placed, (v) => v.addState("v", "m", int())); + expect(contractData(next).placement).toBe(placement); + }); + + it("is typed by the contract's params", () => { + remote("app.typed", { + version: 1, + params: { session: str() }, + // @ts-expect-error: the params have no `id` + placement: ({ id }) => Session({ id }), + variants, + }); + }); +}); diff --git a/packages/chord/test/next/contracts/params.test.ts b/packages/chord/test/next/contracts/params.test.ts new file mode 100644 index 000000000..159ff72ee --- /dev/null +++ b/packages/chord/test/next/contracts/params.test.ts @@ -0,0 +1,41 @@ +import Type from "typebox"; +import { describe, expect, it } from "vitest"; +import { arr, bool, bytes, int, num, obj, oneOf, opt, remote, str } from "../../../src/next/contracts/index.ts"; + +const variants = { a: { state: { s: str() } } }; + +describe("params policy", () => { + it("accepts required str, int, bool and oneOf of strings, with constraints", () => { + expect(() => + remote("n", { + version: 1, + params: { + a: str({ min: 1, max: 9, pattern: "^x" }), + b: int({ min: 0 }), + c: bool(), + d: oneOf("chat", "agent"), + }, + variants, + }), + ).not.toThrow(); + }); + + it("refuses optional fields, in the types and at declaration", () => { + // @ts-expect-error: params fields are required + expect(() => remote("n", { version: 1, params: { a: opt(str()) }, variants })).toThrow( + "n.params.a: a param is a required str(), int(), bool() or oneOf(...) of strings", + ); + }); + + it.each([ + ["a number", num()], + ["nested objects", obj({ a: str() })], + ["arrays", arr(str())], + ["bytes", bytes()], + ["numeric literals", oneOf(1, 2)], + ["defaults", Type.String({ default: "x" })], + ])("refuses %s, in the types and at declaration", (_case, schema) => { + // @ts-expect-error: not a params field schema + expect(() => remote("n", { version: 1, params: { p: schema }, variants })).toThrow("n.params.p: a param is"); + }); +}); diff --git a/packages/chord/test/next/contracts/schema.test.ts b/packages/chord/test/next/contracts/schema.test.ts new file mode 100644 index 000000000..f509d0262 --- /dev/null +++ b/packages/chord/test/next/contracts/schema.test.ts @@ -0,0 +1,99 @@ +import { Compile } from "typebox/compile"; +import { describe, expect, expectTypeOf, it } from "vitest"; +import { + arr, + bool, + bytes, + int, + lit, + map, + method, + nul, + num, + obj, + oneOf, + opt, + type Static, + str, + union, + unknown, +} from "../../../src/next/contracts/index.ts"; +import { Method } from "../../../src/next/contracts/internal.ts"; +import type { Equal } from "../support.ts"; + +describe("schema helpers", () => { + it("produce ordinary TypeBox schemas with bounds under their JSON Schema names", () => { + expect(str({ min: 1, max: 3, pattern: "^a" })).toEqual({ + type: "string", + minLength: 1, + maxLength: 3, + pattern: "^a", + }); + expect(num({ min: 0 })).toEqual({ type: "number", minimum: 0 }); + expect(int({ max: 9 })).toEqual({ type: "integer", maximum: 9 }); + expect(arr(bool(), { min: 1, max: 2 })).toEqual({ + type: "array", + items: { type: "boolean" }, + minItems: 1, + maxItems: 2, + }); + expect(nul()).toEqual({ type: "null" }); + expect(lit(3)).toEqual({ type: "number", const: 3 }); + expect(oneOf("a", "b")).toEqual({ anyOf: [lit("a"), lit("b")] }); + expect(union([str(), nul()])).toEqual({ anyOf: [str(), nul()] }); + expect(map(int())).toEqual({ type: "object", patternProperties: { "^.*$": int() } }); + expect(unknown()).toEqual({}); + }); + + it("leave absent bounds out instead of writing undefined", () => { + expect(Object.keys(str())).toEqual(["type"]); + expect(Object.keys(bytes())).toEqual(["type"]); + }); + + it("obj() is always closed and opt() makes a field optional", () => { + const schema = obj({ a: str(), b: opt(int()) }); + expect(schema).toEqual({ + type: "object", + required: ["a"], + properties: { a: str(), b: int() }, + additionalProperties: false, + }); + expect(Compile(schema).Check({ a: "x" })).toBe(true); + expect(Compile(schema).Check({ a: "x", c: 1 })).toBe(false); + }); + + it("Static and TypeBox's compiler work unchanged", () => { + const schema = obj({ kind: oneOf("chat", "agent"), n: opt(int()), tags: arr(str()), data: bytes() }); + type Expected = { kind: "chat" | "agent"; n?: number; tags: string[]; data: Uint8Array }; + expectTypeOf, Expected>>().toEqualTypeOf(); + expect(Compile(schema).Check({ kind: "chat", tags: [], data: new Uint8Array(1) })).toBe(true); + }); +}); + +describe("bytes", () => { + it("is the data marker with the check attached non-enumerably", () => { + const schema = bytes({ min: 1, max: 2 }); + expect(JSON.parse(JSON.stringify(schema))).toEqual({ type: "bytes", minLength: 1, maxLength: 2 }); + expect(Object.keys(schema)).toEqual(["type", "minLength", "maxLength"]); + expect(Object.getOwnPropertyDescriptor(schema, "~refine")?.enumerable).toBe(false); + }); + + it("checks instanceof Uint8Array and the length bounds", () => { + const validator = Compile(bytes({ min: 1, max: 2 })); + expect(validator.Check(new Uint8Array(1))).toBe(true); + expect(validator.Check(new Uint8Array(2))).toBe(true); + expect(validator.Check(new Uint8Array(0))).toBe(false); + expect(validator.Check(new Uint8Array(3))).toBe(false); + expect(validator.Check([1])).toBe(false); + expect(validator.Check("ab")).toBe(false); + }); +}); + +describe("method", () => { + it("an omitted side is null", () => { + expect(method()).toEqual(new Method(nul(), nul())); + expect(method(str())).toEqual(new Method(str(), nul())); + expect(method(undefined, int())).toEqual(new Method(nul(), int())); + expectTypeOf(method(undefined, int())).toEqualTypeOf, ReturnType>>(); + }); +}); diff --git a/packages/chord/test/next/contracts/snapshot.test.ts b/packages/chord/test/next/contracts/snapshot.test.ts new file mode 100644 index 000000000..d90e5a284 --- /dev/null +++ b/packages/chord/test/next/contracts/snapshot.test.ts @@ -0,0 +1,190 @@ +import Type from "typebox"; +import { Compile } from "typebox/compile"; +import { describe, expect, it } from "vitest"; +import { arr, bytes, int, map, obj, opt, str } from "../../../src/next/contracts/schema.ts"; +import { snapshotArguments, snapshotSchema } from "../../../src/next/contracts/snapshot.ts"; + +describe("vocabulary", () => { + it("accepts primitives, arrays, tuples, objects, records, unions, intersections, $defs/$ref, constraints", () => { + const recursive = Type.Cyclic( + { Node: Type.Object({ value: Type.Number(), next: Type.Union([Type.Null(), Type.Ref("Node")]) }) }, + "Node", + ); + const schemas = [ + Type.Tuple([Type.String(), Type.Integer()]), + Type.Intersect([Type.Object({ a: Type.String() }), Type.Object({ b: Type.String() })]), + Type.Enum(["a", "b"]), + recursive, + str({ min: 1, pattern: "^x" }), + Type.String({ format: "email", default: "a@b.c", readOnly: true, title: "t", description: "d" }), + Type.Number({ multipleOf: 2, exclusiveMinimum: 0 }), + Type.Array(Type.String(), { uniqueItems: true }), + Type.Object({}, { minProperties: 1, maxProperties: 2, $comment: "c", deprecated: true, examples: [{}] }), + map(int()), + bytes({ max: 4 }), + ]; + for (const schema of schemas) expect(() => snapshotSchema(schema, "s")).not.toThrow(); + }); + + it.each([ + ["conditional", { if: { type: "string" }, else: { minLength: 1 } }, 'unsupported keyword "else"'], + ["dependent", { type: "object", dependentSchemas: { a: {} } }, 'unsupported keyword "dependentSchemas"'], + [ + "dependentRequired", + { type: "object", dependentRequired: { a: ["b"] } }, + 'unsupported keyword "dependentRequired"', + ], + ["not", { not: { type: "string" } }, 'unsupported keyword "not"'], + ["contains", { type: "array", contains: { type: "string" } }, 'unsupported keyword "contains"'], + ["propertyNames", { type: "object", propertyNames: { pattern: "^a" } }, 'unsupported keyword "propertyNames"'], + ["unevaluatedProperties", { unevaluatedProperties: false }, 'unsupported keyword "unevaluatedProperties"'], + ["unevaluatedItems", { unevaluatedItems: false }, 'unsupported keyword "unevaluatedItems"'], + ["contentSchema", { type: "string", contentSchema: {} }, 'unsupported keyword "contentSchema"'], + ["definitions", { definitions: {} }, 'unsupported keyword "definitions"'], + ["oneOf", { oneOf: [{ type: "string" }] }, 'unsupported keyword "oneOf"'], + ["unknown keyword", { type: "string", flavour: "x" }, 'unsupported keyword "flavour"'], + ["non-JSON type", Type.BigInt(), 'unsupported type "bigint"'], + ["undefined", Type.Undefined(), 'unsupported type "undefined"'], + ["function", Type.Function([], Type.Null()), "s: unsupported"], + ["Unsafe", Type.Unsafe({ type: "string" }), "unsafe types are not supported"], + ["refinement", Type.Refine(Type.String(), () => true), "refinements"], + [ + "codec", + Type.Codec(Type.String()) + .Decode((v) => v) + .Encode((v) => v), + "codecs", + ], + ["non-JSON data", { type: "number", default: Number.NaN }, "s.default: expected JSON data"], + ["undefined data", { type: "number", maximum: undefined }, "s.maximum: expected JSON data"], + ["foreign prototype", new (class Schema {})(), "s: expected a schema"], + ["array as schema", [], "s: expected a schema"], + ])("refuses %s", (_name, schema, message) => { + expect(() => snapshotSchema(schema, "s")).toThrow(message); + }); + + it("refuses symbol keys, accessors and cycles, naming the path", () => { + expect(() => snapshotSchema({ type: "string", [Symbol("x")]: 1 }, "s")).toThrow( + "s: symbol keys are not supported", + ); + const accessor = Object.defineProperty({ type: "object" }, "properties", { get: () => ({}), enumerable: true }); + expect(() => snapshotSchema(accessor, "s")).toThrow("s.properties: accessors are not supported"); + const cyclic: { type: string; items?: object } = { type: "array" }; + cyclic.items = cyclic; + expect(() => snapshotSchema(cyclic, "s")).toThrow("s.items: cyclic schema"); + expect(() => snapshotSchema({ type: "object", properties: { a: { type: 1 } } }, "s")).toThrow( + "s.properties.a.type: unsupported type 1", + ); + }); + + it("allows a schema object shared by two places (not a cycle)", () => { + const shared = str(); + expect(() => snapshotSchema(obj({ a: shared, b: shared }), "s")).not.toThrow(); + }); +}); + +describe("closed arguments", () => { + it("requires every object, nested ones included, to be closed", () => { + expect(() => snapshotArguments(obj({ a: obj({ b: str() }) }), "m.args")).not.toThrow(); + expect(() => snapshotArguments(obj({ a: Type.Object({ b: str() }) }), "m.args")).toThrow( + "m.args.properties.a: argument objects must be closed (additionalProperties: false)", + ); + expect(() => snapshotArguments(Type.Object({}), "m.args")).toThrow("m.args: argument objects must be closed"); + expect(() => snapshotArguments(obj({ list: arr(Type.Object({})) }), "m.args")).toThrow( + "m.args.properties.list.items: argument objects must be closed", + ); + expect(() => snapshotArguments({ type: "object" }, "m.args")).toThrow("argument objects must be closed"); + }); + + it("accepts a raw closed Type.Object and records", () => { + expect(() => snapshotArguments(Type.Object({ a: str() }, { additionalProperties: false }), "m")).not.toThrow(); + expect(() => snapshotArguments(obj({ tags: map(str()) }), "m")).not.toThrow(); + }); + + it("checks objects inside $defs, unions and tuples", () => { + const open = Type.Object({ a: str() }); + expect(() => snapshotArguments(Type.Union([open, str()]), "m")).toThrow("m.anyOf[0]: argument objects"); + expect(() => snapshotArguments(Type.Tuple([open]), "m")).toThrow("m.items[0]: argument objects"); + expect(() => snapshotArguments({ $defs: { A: open }, $ref: "A" }, "m")).toThrow("m.$defs.A: argument objects"); + }); + + it("does not apply to state or results", () => { + expect(() => snapshotSchema(Type.Object({ a: str() }), "state")).not.toThrow(); + }); +}); + +describe("snapshots", () => { + it("are copies, independent of later changes to the app's schema objects", () => { + const app = obj({ text: str(), tags: arr(str()) }); + const { schema } = snapshotSchema(app, "s"); + (app.properties.text as { minLength?: number }).minLength = 5; + (app.required as string[]).push("x"); + expect(schema).not.toBe(app); + expect(Compile(schema).Check({ text: "", tags: [] })).toBe(true); + }); + + it("copy data values (defaults, consts)", () => { + const fallback = { a: [1] }; + const { schema } = snapshotSchema(Type.Object({}, { default: fallback, additionalProperties: false }), "s"); + fallback.a.push(2); + const copied = (schema as { default: { a: number[] } }).default; + expect(copied).toEqual({ a: [1] }); + }); + + it("rebuild the bytes check from the marker; no app callback is copied", () => { + const app = bytes({ max: 2 }); + const appCheck = Object.getOwnPropertyDescriptor(app, "~refine")!.value; + const { schema } = snapshotSchema(app, "s"); + expect(Object.getOwnPropertyDescriptor(schema, "~refine")!.value).not.toBe(appCheck); + expect(Compile(schema).Check(new Uint8Array(2))).toBe(true); + expect(Compile(schema).Check(new Uint8Array(3))).toBe(false); + // A hand-written marker works too: the check comes from the marker, not from the app. + const marker = snapshotSchema({ type: "bytes", minLength: 1 }, "s").schema; + expect(Compile(marker).Check(new Uint8Array(0))).toBe(false); + expect(Compile(marker).Check(new Uint8Array(1))).toBe(true); + }); + + it("refuse a bytes marker with other keywords or bad bounds", () => { + expect(() => snapshotSchema({ type: "bytes", items: {} }, "s")).toThrow('s: unsupported keyword "items"'); + expect(() => snapshotSchema({ type: "bytes", maxLength: -1 }, "s")).toThrow( + "s: bytes bounds must be non-negative integers", + ); + }); + + it("normal form: sorted keys, sorted required and enum, annotations dropped on schema nodes only", () => { + const { normal } = snapshotSchema( + { + title: "T", + type: "object", + required: ["b", "a"], + properties: { b: { enum: ["y", "x"] }, a: { type: "string", default: { title: "kept", z: 1, a: 2 } } }, + additionalProperties: false, + }, + "s", + ); + expect(JSON.stringify(normal)).toBe( + JSON.stringify({ + additionalProperties: false, + properties: { a: { default: { a: 2, title: "kept", z: 1 }, type: "string" }, b: { enum: ["x", "y"] } }, + required: ["a", "b"], + type: "object", + }), + ); + }); + + it("normal form keeps union and tuple order", () => { + const ab = snapshotSchema(Type.Union([str(), int()]), "s").normal; + const ba = snapshotSchema(Type.Union([int(), str()]), "s").normal; + expect(ab).not.toEqual(ba); + expect(snapshotSchema(opt(str()), "s").normal).toEqual({ type: "string" }); + }); +}); + +describe("bytes", () => { + it("mark a snapshot with a bytes leaf anywhere inside", () => { + expect(snapshotSchema(str(), "s").bytes).toBe(false); + expect(snapshotSchema(obj({ a: arr(obj({ b: bytes() })) }), "s").bytes).toBe(true); + expect(snapshotSchema(Type.Union([str(), bytes()]), "s").bytes).toBe(true); + expect(snapshotSchema(Type.Object({ a: str() }, { default: { type: "bytes" } }), "s").bytes).toBe(false); + }); +}); diff --git a/packages/chord/test/next/data/args.test.ts b/packages/chord/test/next/data/args.test.ts new file mode 100644 index 000000000..90dc8cae6 --- /dev/null +++ b/packages/chord/test/next/data/args.test.ts @@ -0,0 +1,223 @@ +import { describe, expect, expectTypeOf, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT, type Context } from "../../../src/context/index.ts"; +import { + type AcceptsTable, + arr, + bytes, + evolve, + int, + local, + method, + nul, + obj, + remote, + str, +} from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { bindProvider, type CheckedArgs, checkArgs, type Value } from "../../../src/next/data/internal.ts"; +import { counted, expectCollected } from "../support.ts"; + +const Note = obj({ text: str() }); +const NoteV2 = obj({ text: str(), tag: str() }); + +const NotesV1 = remote("notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { + state: { notes: arr(Note) }, + methods: { add: method(Note, str()), clear: method(), attach: method(obj({ data: bytes({ max: 4 }) })) }, + }, + reader: { state: { notes: arr(Note) } }, + }, +}); +const NotesV2 = evolve(NotesV1, (e) => e.args("add", NoteV2).result("add", int())); +const NotesV3 = evolve(NotesV2, (e) => e.removeMethod("attach")); + +const toV2 = vi.fn((old: { text: string }) => ({ ...old, tag: "v1" })); +const resultToOld = (current: number) => String(current); +const attach = async () => null; +const accepts: AcceptsTable = { + 1: { methods: { add: { args: toV2, result: resultToOld }, attach: { call: attach } } }, + 2: { methods: { attach: "refuse" } }, +}; +const provider = bindProvider(NotesV3, accepts); + +/** The error a refused check throws, as JSON. */ +function refusal(run: () => unknown): unknown { + try { + run(); + } catch (error) { + if (isChordError(error)) return error.toJSON(); + throw error; + } + throw new Error("expected a refusal"); +} + +describe("bindProvider", () => { + it("checks the accepts table once and lists the accepted versions", () => { + expect(provider.contract).toBe(NotesV3); + expect(provider.accepted).toEqual([ + { version: 1, fingerprint: NotesV1.fingerprint }, + { version: 2, fingerprint: NotesV2.fingerprint }, + ]); + expect(() => bindProvider(NotesV3, { 1: {} } as never)).toThrow("notes accepts[1].methods.add: missing mapping"); + }); + + it("compiles nothing and calls no mapping", async () => { + const counts = await counted(() => bindProvider(NotesV3, accepts)); + expect(counts).toEqual({ walks: 0, validations: 0, compilations: 0 }); + expect(toV2).not.toHaveBeenCalled(); + }); +}); + +describe("checkArgs", () => { + it("returns current-version arguments as passed: no copy, never mapped", () => { + const raw = { text: "t", tag: "x" }; + const checked = checkArgs(provider, "add", raw, 3, BACKGROUND_CONTEXT); + expect(checked).toEqual({ kind: "method", args: raw, result: undefined }); + expect(checked.args).toBe(raw); + expect(toV2).not.toHaveBeenCalled(); + expectTypeOf(checked).toEqualTypeOf>(); + if (checked.kind === "method") expectTypeOf(checked.args).toEqualTypeOf<{ text: string; tag: string }>(); + }); + + it("checks an old version against the old schema, then maps directly to the current shape", () => { + const checked = checkArgs(provider, "add", { text: "t" }, 1, BACKGROUND_CONTEXT); + expect(checked).toEqual({ kind: "method", args: { text: "t", tag: "v1" }, result: resultToOld }); + expect(refusal(() => checkArgs(provider, "add", { text: "t", tag: "x" }, 1, BACKGROUND_CONTEXT))).toMatchObject({ + code: "invalid", + reason: "args", + details: { path: "/tag" }, + }); + // Version 2 already has the current args: checked with that schema, not mapped. + expect(checkArgs(provider, "add", { text: "t", tag: "x" }, 2, BACKGROUND_CONTEXT)).toEqual({ + kind: "method", + args: { text: "t", tag: "x" }, + result: undefined, + }); + }); + + it("returns the old arguments with the call replacement; never invokes it", () => { + const raw = { data: new Uint8Array(2) }; + expect(checkArgs(provider, "attach", raw, 1, BACKGROUND_CONTEXT)).toEqual({ + kind: "call", + args: raw, + call: attach, + }); + }); + + it("refuses unknown fields, nested ones too, and checks bytes by instanceof and length", () => { + expect( + refusal(() => checkArgs(provider, "add", { text: "t", tag: "x", extra: 1 }, 3, BACKGROUND_CONTEXT)), + ).toMatchObject({ + details: { path: "/extra" }, + }); + expect(refusal(() => checkArgs(provider, "attach", { data: [1, 2] }, 1, BACKGROUND_CONTEXT))).toMatchObject({ + details: { path: "/data" }, + }); + expect( + refusal(() => checkArgs(provider, "attach", { data: new Uint8Array(5) }, 1, BACKGROUND_CONTEXT)), + ).toMatchObject({ + details: { path: "/data" }, + }); + const Nested = local("nested", { version: 1, params: {}, members: { m: method(obj({ a: obj({ b: str() }) })) } }); + const nested = bindProvider(Nested, {}); + expect(refusal(() => checkArgs(nested, "m", { a: { b: "x", c: 1 } }, 1, BACKGROUND_CONTEXT))).toMatchObject({ + details: { path: "/a/c" }, + }); + }); + + it("takes null for no-argument methods", () => { + expect(checkArgs(provider, "clear", null, 3, BACKGROUND_CONTEXT)).toEqual({ + kind: "method", + args: null, + result: undefined, + }); + expect(refusal(() => checkArgs(provider, "clear", {}, 3, BACKGROUND_CONTEXT))).toMatchObject({ + code: "invalid", + reason: "args", + details: { path: "/" }, + }); + }); + + it("checks local callers too", () => { + const Info = local("info", { version: 1, params: {}, members: { save: method(obj({ title: str() }), nul()) } }); + const local1 = bindProvider(Info, {}); + expect(checkArgs(local1, "save", { title: "t" }, 1, BACKGROUND_CONTEXT).args).toEqual({ title: "t" }); + expect(refusal(() => checkArgs(local1, "save", { title: 1 }, 1, BACKGROUND_CONTEXT))).toMatchObject({ + message: "args/title: must be string", + }); + }); + + it("refuses an unknown method, an unaccepted version and a refused method", () => { + expect(refusal(() => checkArgs(provider, "missing", null, 3, BACKGROUND_CONTEXT))).toEqual({ + code: "invalid", + reason: "member", + message: 'notes v3 has no method "missing"', + }); + expect(refusal(() => checkArgs(provider, "attach", null, 3, BACKGROUND_CONTEXT))).toMatchObject({ + code: "invalid", + reason: "member", + }); + expect(refusal(() => checkArgs(provider, "add", { text: "t" }, 4, BACKGROUND_CONTEXT))).toEqual({ + code: "contract_mismatch", + message: "notes v4 is not accepted", + }); + expect( + refusal(() => checkArgs(bindProvider(NotesV3, {}), "add", { text: "t" }, 1, BACKGROUND_CONTEXT)), + ).toMatchObject({ + code: "contract_mismatch", + }); + expect(refusal(() => checkArgs(provider, "attach", { data: new Uint8Array() }, 2, BACKGROUND_CONTEXT))).toEqual({ + code: "contract_mismatch", + message: 'notes v2 refuses "attach"', + }); + }); + + it("compiles lazily, once per accepted version and method; equal schema objects share a validator", async () => { + const v1 = local("lazy", { version: 1, params: {}, members: { m: method(obj({ a: str() })), clear: method() } }); + const v2 = evolve(v1, (e) => e.args("m", obj({ a: str(), b: str() }))); + const fresh = bindProvider(v2, { 1: { methods: { m: { args: (old) => ({ ...old, b: "" }) } } } }); + const calls = [ + (context: Context) => checkArgs(fresh, "clear", null, 2, context), + (context: Context) => checkArgs(fresh, "clear", null, 2, context), + // Version 1 shares the unchanged `clear` schema object with version 2. + (context: Context) => checkArgs(fresh, "clear", null, 1, context), + (context: Context) => checkArgs(fresh, "m", { a: "x" }, 1, context), + (context: Context) => checkArgs(fresh, "m", { a: "x" }, 1, context), + (context: Context) => checkArgs(fresh, "m", { a: "x", b: "y" }, 2, context), + ]; + const compilations = []; + for (const call of calls) compilations.push((await counted(call)).compilations); + expect(compilations).toEqual([1, 0, 0, 1, 0, 1]); + }); + + it("counts one validation per call and no walk", async () => { + const counts = await counted((context) => checkArgs(provider, "add", { text: "t" }, 1, context)); + expect(counts).toEqual({ walks: 0, validations: 1, compilations: 0 }); + }); + + it("types a call replacement's arguments as values", () => { + expectTypeOf, { kind: "call" }>["call"]>().toEqualTypeOf< + (args: Value, context: Context) => Promise + >(); + }); +}); + +describe("memory", () => { + it("a dropped binding releases its dispatch table, mappings, schemas and validators after the last call", async () => { + const refs = await (async () => { + const contract = evolve( + local("gc", { version: 1, params: {}, members: { m: method(obj({ a: str() })) } }), + (e) => e.args("m", obj({ a: str(), b: str() })), + ); + const mapping = (old: { a: string }) => ({ ...old, b: "" }); + const bound = bindProvider(contract, { 1: { methods: { m: { args: mapping } } } }); + const pending = Promise.resolve(checkArgs(bound, "m", { a: "x" }, 1, BACKGROUND_CONTEXT)); + await pending; + return [new WeakRef(contract), new WeakRef(bound), new WeakRef(mapping)]; + })(); + await expectCollected(...refs); + }); +}); diff --git a/packages/chord/test/next/data/errors.test.ts b/packages/chord/test/next/data/errors.test.ts new file mode 100644 index 000000000..65dee086a --- /dev/null +++ b/packages/chord/test/next/data/errors.test.ts @@ -0,0 +1,135 @@ +import { describe, expect, expectTypeOf, it } from "vitest"; +import { ChordError, type ChordErrorReason, isChordError, PluginError } from "../../../src/next/data/index.ts"; +import { + asChordError, + CHORD_ERROR_REASONS, + chordErrorFromJson, + invalid, + PROTOCOL_VERSION, +} from "../../../src/next/data/internal.ts"; + +describe("ChordError", () => { + it("is a tagged union: a wrong pairing is a type error", () => { + new ChordError({ code: "invalid", reason: "args", message: "m" }); + new ChordError({ code: "unavailable", message: "m" }); + new ChordError({ code: "unavailable", reason: "locked", message: "m" }); + // @ts-expect-error: invalid requires a reason + new ChordError({ code: "invalid", message: "m" }); + // @ts-expect-error: gone carries no reason + new ChordError({ code: "gone", reason: "args", message: "m" }); + // @ts-expect-error: part belongs to too_large + new ChordError({ code: "unavailable", reason: "part", message: "m" }); + }); + + it("narrowing on code narrows reason", () => { + const error = new ChordError({ code: "too_large", reason: "part", message: "m" }) as ChordError; + if (error.code === "invalid") expectTypeOf(error.reason).toEqualTypeOf>(); + if (error.code === "unavailable") + expectTypeOf(error.reason).toEqualTypeOf<"locked" | "park_timeout" | "busy" | "storage_failed" | undefined>(); + if (error.code === "gone") expectTypeOf(error.reason).toEqualTypeOf(); + expect(error).toBeInstanceOf(Error); + expect(error.name).toBe("ChordError"); + }); + + it("round-trips through JSON without stack or cause", () => { + const details = { path: "/text" }; + const error = new ChordError( + { code: "invalid", reason: "args", message: "args/text: bad", details }, + { cause: new Error("x") }, + ); + const json = JSON.parse(JSON.stringify(error)); + expect(json).toEqual({ code: "invalid", reason: "args", message: "args/text: bad", details }); + const back = chordErrorFromJson(json); + expect(back).toBeInstanceOf(ChordError); + expect(back.toJSON()).toEqual(json); + expect(new ChordError({ code: "gone", message: "m" }).toJSON()).toEqual({ code: "gone", message: "m" }); + }); + + it("keeps details by reference", () => { + const details = { large: [1, 2, 3] }; + expect(chordErrorFromJson({ code: "gone", message: "m", details }).details).toBe(details); + }); + + it.each([ + ["not an object", "x", "not an object"], + ["null", null, "not an object"], + ["an array", [], "not an object"], + ["unknown key", { code: "gone", message: "m", stack: "s" }, 'unknown key "stack"'], + ["unknown code", { code: "teapot", message: "m" }, "unknown code"], + ["inherited name as code", { code: "toString", message: "m" }, "unknown code"], + ["code not a string", { code: 1, message: "m" }, "unknown code"], + ["invalid without reason", { code: "invalid", message: "m" }, "reason not allowed for invalid"], + ["reason not allowed", { code: "gone", reason: "args", message: "m" }, "reason not allowed for gone"], + [ + "reason of another code", + { code: "unavailable", reason: "part", message: "m" }, + "reason not allowed for unavailable", + ], + ["message not a string", { code: "gone", message: 1 }, "message is not a string"], + ])("chordErrorFromJson refuses %s", (_case, raw, problem) => { + expect(() => chordErrorFromJson(raw)).toThrow(`Invalid ChordError JSON: ${problem}`); + }); +}); + +describe("protocol table", () => { + it("is pinned together with PROTOCOL_VERSION", () => { + const pinned = { + version: 1, + reasons: { + invalid: ["args", "params", "member", "placement", "reentrant", "read_after_bulk", "cycle"], + contract_mismatch: ["updating"], + forbidden: ["revoked"], + gone: [], + not_ready: [], + released: ["ended", "closed", "settled", "version", "discarded"], + too_large: ["part"], + unavailable: ["locked", "park_timeout", "busy", "storage_failed"], + aborted: [], + }, + }; + const changed = + PROTOCOL_VERSION !== pinned.version || JSON.stringify(CHORD_ERROR_REASONS) !== JSON.stringify(pinned.reasons); + expect(changed, "the error table changed: bump PROTOCOL_VERSION and update this pin").toBe(false); + }); +}); + +describe("helpers", () => { + it("isChordError checks the class and optionally the code", () => { + const error = invalid("member", "no method"); + expect(isChordError(error)).toBe(true); + expect(isChordError(error, "invalid")).toBe(true); + expect(isChordError(error, "gone")).toBe(false); + expect(isChordError(new Error("x"))).toBe(false); + expect(isChordError({ code: "invalid", reason: "member", message: "m" })).toBe(false); + }); + + it("invalid builds an invalid error with its reason and details", () => { + const error = invalid("params", "params/a: missing", { path: "/a" }); + expect(error.toJSON()).toEqual({ + code: "invalid", + reason: "params", + message: "params/a: missing", + details: { path: "/a" }, + }); + expectTypeOf(error.code).toEqualTypeOf<"invalid">(); + }); + + it("asChordError keeps a ChordError and wraps anything else as unavailable", () => { + const chord = invalid("cycle", "c"); + expect(asChordError(chord, "open")).toBe(chord); + const foreign = new TypeError("boom"); + const wrapped = asChordError(foreign, "open notes"); + expect(wrapped.toJSON()).toEqual({ code: "unavailable", message: "open notes: boom" }); + expect(wrapped.cause).toBe(foreign); + expect(asChordError("text", "open").message).toBe("open: text"); + }); +}); + +describe("PluginError", () => { + it("carries a plugin's code, message and JSON details", () => { + const error = new PluginError("quota", "too many", { limit: 3 }); + expect(error).toBeInstanceOf(Error); + expect(error).toMatchObject({ name: "PluginError", code: "quota", message: "too many", details: { limit: 3 } }); + expect(new PluginError("c", "m").details).toBeUndefined(); + }); +}); diff --git a/packages/chord/test/next/data/failure.test.ts b/packages/chord/test/next/data/failure.test.ts new file mode 100644 index 000000000..a5ddbacf2 --- /dev/null +++ b/packages/chord/test/next/data/failure.test.ts @@ -0,0 +1,27 @@ +import { describe, expect, it } from "vitest"; +import { broken } from "../../../src/next/data/index.ts"; +import { invalid, openFailure } from "../../../src/next/data/internal.ts"; + +describe("failure causes", () => { + it("a value thrown by an open is transient unless marked broken", () => { + const error = new Error("network down"); + const transient = openFailure(error, "open notes"); + expect(transient.kind).toBe("transient"); + expect(transient.error.toJSON()).toEqual({ code: "unavailable", message: "open notes: network down" }); + expect(transient.error.cause).toBe(error); + }); + + it("broken() marks an open's failure as broken; its message is the cause's", () => { + const marked = broken(new Error("bad config")); + expect(marked.message).toBe("bad config"); + const cause = openFailure(marked, "open notes"); + expect(cause.kind).toBe("broken"); + expect(cause.error.message).toBe("open notes: bad config"); + }); + + it("the kind comes from the stage, never from the error code", () => { + const chord = invalid("placement", "wrong host"); + expect(openFailure(chord, "open")).toEqual({ kind: "transient", error: chord }); + expect(openFailure(broken(chord), "open")).toEqual({ kind: "broken", error: chord }); + }); +}); diff --git a/packages/chord/test/next/data/params.test.ts b/packages/chord/test/next/data/params.test.ts new file mode 100644 index 000000000..55b9a8ebc --- /dev/null +++ b/packages/chord/test/next/data/params.test.ts @@ -0,0 +1,130 @@ +import { describe, expect, expectTypeOf, it } from "vitest"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { contractData } from "../../../src/next/contracts/contract.ts"; +import { bool, int, local, oneOf, remote, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { checkParams, identityKey } from "../../../src/next/data/internal.ts"; +import { validatorFor } from "../../../src/next/data/validator.ts"; +import { counted, expectCollected } from "../support.ts"; + +const Session = remote("session", { + version: 1, + params: { id: str({ min: 1 }), mode: oneOf("chat", "agent"), page: int({ min: 0 }), open: bool() }, + variants: { owner: {} }, +}); + +const valid = { id: "S1", mode: "chat", page: 2, open: true } as const; + +/** The `invalid/params` error of a refused check. */ +function refusal(raw: unknown): unknown { + try { + checkParams(Session, raw, BACKGROUND_CONTEXT); + } catch (error) { + if (isChordError(error, "invalid")) + return { reason: error.reason, message: error.message, details: error.details }; + throw error; + } + throw new Error("expected a refusal"); +} + +describe("checkParams", () => { + it("returns the caller's own object and the key of the field values in declared order", () => { + const { params, key } = checkParams(Session, valid, BACKGROUND_CONTEXT); + expect(params).toBe(valid); + expect(key).toBe("S1|chat|2|true"); + expectTypeOf(params).toEqualTypeOf<{ + readonly id: string; + readonly mode: "chat" | "agent"; + readonly page: number; + readonly open: boolean; + }>(); + }); + + it("gives the same key whatever the property order", () => { + expect(checkParams(Session, { open: true, page: 2, mode: "chat", id: "S1" }, BACKGROUND_CONTEXT).key).toBe( + "S1|chat|2|true", + ); + }); + + it("refuses missing, extra and wrong fields at their paths", () => { + const { id: _, ...missing } = valid; + expect(refusal(missing)).toEqual({ + reason: "params", + message: expect.stringMatching(/^params\/id: /), + details: { path: "/id" }, + }); + expect(refusal({ ...valid, extra: 1 })).toMatchObject({ + message: expect.stringMatching(/^params\/extra: /), + details: { path: "/extra" }, + }); + expect(refusal({ ...valid, page: "2" })).toMatchObject({ + message: "params/page: must be integer", + details: { path: "/page" }, + }); + expect(refusal("S1")).toMatchObject({ message: "params: must be object", details: { path: "/" } }); + }); + + it("enforces literal membership and constraints", () => { + expect(refusal({ ...valid, mode: "other" })).toMatchObject({ details: { path: "/mode" } }); + expect(refusal({ ...valid, id: "" })).toMatchObject({ details: { path: "/id" } }); + expect(refusal({ ...valid, page: -1 })).toMatchObject({ details: { path: "/page" } }); + expect(refusal({ ...valid, page: 1.5 })).toMatchObject({ details: { path: "/page" } }); + }); + + it("escapes the separator: no two params join to one key", () => { + const Pair = local("pair", { version: 1, params: { a: str(), b: str() }, members: {} }); + const keys = [ + { a: "x|y", b: "z" }, + { a: "x", b: "y|z" }, + { a: "x\\", b: "|z" }, + { a: "x\\|", b: "z" }, + ].map((raw) => checkParams(Pair, raw, BACKGROUND_CONTEXT).key); + expect(new Set(keys).size).toBe(keys.length); + expect(keys[0]).toBe("x\\|y|z"); + }); + + it("counts one compilation per contract, then one validation per get; a failure adds one walk", async () => { + const Fresh = local("fresh", { version: 1, params: { id: str() }, members: {} }); + expect(await counted((context) => checkParams(Fresh, { id: "a" }, context))).toEqual({ + walks: 0, + validations: 1, + compilations: 1, + }); + expect(await counted((context) => checkParams(Fresh, { id: "b" }, context))).toEqual({ + walks: 0, + validations: 1, + compilations: 0, + }); + const failed = await counted((context) => { + expect(() => checkParams(Fresh, { id: 1 }, context)).toThrow(); + }); + expect(failed).toEqual({ walks: 1, validations: 1, compilations: 0 }); + }); + + it("does not retain a dropped contract's params validator", async () => { + const refs = (() => { + const contract = local("gc", { version: 1, params: { id: str() }, members: {} }); + checkParams(contract, { id: "a" }, BACKGROUND_CONTEXT); + const validator = validatorFor(contractData(contract).params, BACKGROUND_CONTEXT); + return [new WeakRef(contract), new WeakRef(validator)]; + })(); + await expectCollected(...refs); + }); +}); + +describe("identityKey", () => { + it("joins name, key and principal key, each escaped", () => { + expect(identityKey(Session, "S1|chat|2|true", '{"kind":"user"}')).toBe( + 'session|S1\\|chat\\|2\\|true|{"kind":"user"}', + ); + }); + + it("local objects take no principal key, by type", () => { + const Info = local("info", { version: 1, params: {}, members: {} }); + expect(identityKey(Info, "")).toBe("info|"); + // @ts-expect-error: local has no principal + identityKey(Info, "", "p"); + // @ts-expect-error: remote needs a principal key + identityKey(Session, "k"); + }); +}); diff --git a/packages/chord/test/next/data/principal.test.ts b/packages/chord/test/next/data/principal.test.ts new file mode 100644 index 000000000..adca8418f --- /dev/null +++ b/packages/chord/test/next/data/principal.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { canonicalJson, principalKey } from "../../../src/next/data/internal.ts"; +import { counted, expectCollected } from "../support.ts"; + +describe("canonicalJson", () => { + it("sorts object keys and writes -0 as 0", () => { + expect(canonicalJson({ b: 1, a: [{ d: -0, c: null }] }, BACKGROUND_CONTEXT)).toBe( + '{"a":[{"c":null,"d":0}],"b":1}', + ); + expect(canonicalJson("x", BACKGROUND_CONTEXT)).toBe('"x"'); + }); + + it("counts one walk", async () => { + expect(await counted((context) => canonicalJson({ a: [1, 2] }, context))).toEqual({ + walks: 1, + validations: 0, + compilations: 0, + }); + }); +}); + +describe("principalKey", () => { + it("defaults to the canonical JSON, whatever the property order", () => { + expect(principalKey({ kind: "user", id: "u1" }, BACKGROUND_CONTEXT)).toBe('{"id":"u1","kind":"user"}'); + expect(principalKey({ id: "u1", kind: "user" }, BACKGROUND_CONTEXT)).toBe( + principalKey({ kind: "user", id: "u1" }, BACKGROUND_CONTEXT), + ); + }); + + it("uses a custom key function when given", () => { + const byId = (principal: { kind: string; id: string }) => `${principal.kind}:${principal.id}`; + expect(principalKey({ kind: "user", id: "u1" }, BACKGROUND_CONTEXT, byId)).toBe("user:u1"); + }); + + it("caches per principal object: no second walk", async () => { + const principal = { kind: "user", id: "u1" }; + const first = await counted((context) => principalKey(principal, context)); + const second = await counted((context) => principalKey(principal, context)); + expect([first.walks, second.walks]).toEqual([1, 0]); + }); + + it("caches custom keys per function", () => { + const principal = { kind: "user", id: "u1" }; + let calls = 0; + const custom = () => `key${++calls}`; + expect([ + principalKey(principal, BACKGROUND_CONTEXT, custom), + principalKey(principal, BACKGROUND_CONTEXT, custom), + ]).toEqual(["key1", "key1"]); + expect(principalKey(principal, BACKGROUND_CONTEXT, () => "other")).toBe("other"); + expect(principalKey(principal, BACKGROUND_CONTEXT)).toBe('{"id":"u1","kind":"user"}'); + }); + + it("never retains a dropped custom key function while its principal lives", async () => { + const principal = { kind: "user", id: "u1" }; + const ref = (() => { + const custom = () => "k"; + principalKey(principal, BACKGROUND_CONTEXT, custom); + return new WeakRef(custom); + })(); + await expectCollected(ref); + expect(principalKey(principal, BACKGROUND_CONTEXT)).toBeTypeOf("string"); + }); + + it("does not retain dropped principals", async () => { + const ref = (() => { + const principal = { kind: "user", id: "gone" }; + principalKey(principal, BACKGROUND_CONTEXT); + principalKey(principal, BACKGROUND_CONTEXT, () => "k"); + return new WeakRef(principal); + })(); + await expectCollected(ref); + }); +}); diff --git a/packages/chord/test/next/data/validator.test.ts b/packages/chord/test/next/data/validator.test.ts new file mode 100644 index 000000000..b78b55c58 --- /dev/null +++ b/packages/chord/test/next/data/validator.test.ts @@ -0,0 +1,106 @@ +import { GetExternal, GetFunctions } from "typebox/schema"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, bytes, int, map, obj, str } from "../../../src/next/contracts/schema.ts"; +import { snapshotArguments } from "../../../src/next/contracts/snapshot.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { check, validatorFor } from "../../../src/next/data/validator.ts"; +import { counted, expectCollected } from "../support.ts"; + +const args = snapshotArguments( + obj({ text: str(), items: arr(obj({ n: int() })), tags: map(str()), blob: bytes({ max: 2 }) }), + "m.args", +); + +const valid = { text: "t", items: [{ n: 1 }], tags: {}, blob: new Uint8Array(1) }; + +/** The `{ message, details }` of the `invalid/args` error a failed check throws. */ +function failure(value: unknown): unknown { + try { + check(validatorFor(args, BACKGROUND_CONTEXT), value, "args", BACKGROUND_CONTEXT); + } catch (error) { + if (isChordError(error, "invalid")) + return { reason: error.reason, message: error.message, details: error.details }; + throw error; + } + return "passed"; +} + +describe("failed checks", () => { + it("pass a valid value", () => { + expect(failure(valid)).toBe("passed"); + }); + + it("report TypeBox's first error with its JSON Pointer path", () => { + expect(failure({ ...valid, text: 1 })).toEqual({ + reason: "args", + message: "args/text: must be string", + details: { path: "/text" }, + }); + expect(failure({ ...valid, items: [{ n: "x" }] })).toMatchObject({ details: { path: "/items/0/n" } }); + expect(failure(null)).toEqual({ reason: "args", message: "args: must be object", details: { path: "/" } }); + }); + + it("report a missing field at its own path (TypeBox reports it at the parent)", () => { + const { text: _, ...missing } = valid; + expect(failure(missing)).toMatchObject({ + message: "args/text: missing", + details: { path: "/text" }, + }); + expect(failure({ ...valid, items: [{}] })).toMatchObject({ details: { path: "/items/0/n" } }); + }); + + it("report an unknown field at its own path, nested too", () => { + expect(failure({ ...valid, extra: 1 })).toEqual({ + reason: "args", + message: "args/extra: unknown field", + details: { path: "/extra" }, + }); + expect(failure({ ...valid, items: [{ n: 1, m: 2 }] })).toMatchObject({ details: { path: "/items/0/m" } }); + }); + + it("check bytes by instanceof and length", () => { + expect(failure({ ...valid, blob: [1] })).toMatchObject({ + message: "args/blob: expected bytes of the declared length", + }); + expect(failure({ ...valid, blob: new Uint8Array(3) })).toMatchObject({ details: { path: "/blob" } }); + }); + + it("leave keys containing `/` or `~` unescaped (TypeBox's instance path, ambiguous)", () => { + expect(failure({ ...valid, tags: { "a/b": 1 } })).toMatchObject({ details: { path: "/tags/a/b" } }); + }); +}); + +describe("validators", () => { + it("compile once per schema owner (counted), then validate (counted); a failure adds one walk", async () => { + const owner = snapshotArguments(obj({ a: str() }), "m"); + const first = await counted((context) => check(validatorFor(owner, context), { a: "x" }, "args", context)); + const second = await counted((context) => check(validatorFor(owner, context), { a: "y" }, "args", context)); + const failed = await counted((context) => { + expect(() => check(validatorFor(owner, context), {}, "args", context)).toThrow(); + }); + expect([first, second, failed]).toEqual([ + { walks: 0, validations: 1, compilations: 1 }, + { walks: 0, validations: 1, compilations: 0 }, + { walks: 1, validations: 1, compilations: 0 }, + ]); + }); + + it("leave nothing in TypeBox's build accumulators", () => { + validatorFor(snapshotArguments(obj({ blob: bytes(), n: int() }), "m"), BACKGROUND_CONTEXT); + expect(GetExternal().variables).toEqual([]); + expect(GetFunctions()).toEqual([]); + }); + + it("are collected with their schema owner; TypeBox retains no build state", async () => { + const refs = (() => { + const owner = snapshotArguments(obj({ blob: bytes() }), "m"); + const validator = validatorFor(owner, BACKGROUND_CONTEXT); + const blob = (owner.schema as { properties: { blob: object } }).properties.blob; + // TypeBox's external variables would hold the bytes check if the build state were retained. + const [check] = Object.getOwnPropertyDescriptor(blob, "~refine")!.value as object[]; + return [new WeakRef(owner), new WeakRef(validator), new WeakRef(check!)]; + })(); + await expectCollected(...refs); + }); +}); diff --git a/packages/chord/test/next/documents/address-source.test.ts b/packages/chord/test/next/documents/address-source.test.ts new file mode 100644 index 000000000..ea621c072 --- /dev/null +++ b/packages/chord/test/next/documents/address-source.test.ts @@ -0,0 +1,114 @@ +// Tests of address sources (slice 6a §11; decision "One retirement mechanism"): the frame a watched document address +// gets for a change, a write over another base, a retirement, a recreation after it in the same batch, and a +// replacement outside a commit. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import type { Op, Prepared } from "../../../src/delta/index.ts"; +import { AddressSource } from "../../../src/next/documents/address-source.ts"; +import type { Documents } from "../../../src/next/documents/documents.ts"; +import type { JsonObject } from "../../../src/next/documents/index.ts"; +import { Batch } from "../../../src/next/state/internal.ts"; +import { changesOf, delivered, expectCollected } from "../support.ts"; + +/** An address whose value the test sets. */ +class Address extends AddressSource { + // No object's documents: the test reads frames only. + readonly owner = {} as Documents; + current: JsonObject = { n: 0 }; + + get value(): JsonObject { + return this.current; + } +} + +/** A prepared write of `value` over `base` with `ops`. */ +function prepared(base: JsonObject, value: JsonObject, ops: readonly Op[]): Prepared { + return { base, value, ops, baseRevision: 0, abort: () => {} }; +} + +/** A watched address on its own batch, and the changes its watch got. */ +async function watched() { + const batch = new Batch(() => {}, undefined); + const address = new Address(batch); + const changes = changesOf(await address.watch(context)); + return { batch, address, changes }; +} + +describe("address frames", () => { + it("a write over the read value carries its ops; over another base, a root replacement", async () => { + const { batch, address, changes } = await watched(); + const first = address.current; + batch.commit(() => { + address.current = { n: 1 }; + address.written(first, prepared(first, address.current, [["s", ["n"], 1]])); + }, context); + await delivered(); + const second = address.current; + batch.commit(() => { + address.current = { n: 2 }; + address.written(second, prepared({ n: 7 }, address.current, [["s", ["n"], 2]])); + }, context); + await delivered(); + expect(changes).toEqual([ + { kind: "value", value: { n: 1 }, ops: [["s", ["n"], 1]], retired: false }, + { kind: "value", value: { n: 2 }, ops: [["r", { n: 2 }]], retired: false }, + ]); + }); + + it("a retirement reads initial() with the flag; a recreation in the same batch is one root replacement, flagged", async () => { + const { batch, address, changes } = await watched(); + batch.commit(() => { + address.current = { n: 0 }; + address.retired(address.current); + }, context); + await delivered(); + batch.commit(() => { + const initial = address.current; + address.retired(initial); + address.current = { n: 5 }; + address.written(initial, prepared(initial, address.current, [["s", ["n"], 5]])); + }, context); + await delivered(); + const recreated = address.current; + batch.commit(() => { + address.current = { n: 6 }; + address.written(recreated, prepared(recreated, address.current, [["s", ["n"], 6]])); + }, context); + await delivered(); + expect(changes).toEqual([ + { kind: "value", value: { n: 0 }, ops: [["r", { n: 0 }]], retired: true }, + { kind: "value", value: { n: 5 }, ops: [["r", { n: 5 }]], retired: true }, + // The flag was reset by the frame that carried it. + { kind: "value", value: { n: 6 }, ops: [["s", ["n"], 6]], retired: false }, + ]); + }); + + it("a replacement outside a commit (a load, a migration) is a root replacement", async () => { + const { batch, address, changes } = await watched(); + batch.commit(() => { + address.current = { n: 3 }; + address.replaced(); + }, context); + await delivered(); + expect(changes).toEqual([{ kind: "value", value: { n: 3 }, ops: [["r", { n: 3 }]], retired: false }]); + }); +}); + +describe("unwatched addresses", () => { + it("keep no ops of a write: no frame takes them", async () => { + const batch = new Batch(() => {}, undefined); + const address = new Address(batch); + let ops: Op[] | undefined = [["s", ["n"], 1]]; + const collected = new WeakRef(ops); + const base = address.current; + batch.commit(() => { + address.current = { n: 1 }; + address.written(base, prepared(base, address.current, ops!)); + }, context); + ops = undefined; + await expectCollected(collected); + // The address itself is still here. + expect(address.value).toEqual({ n: 1 }); + }); +}); diff --git a/packages/chord/test/next/documents/definition.test.ts b/packages/chord/test/next/documents/definition.test.ts new file mode 100644 index 000000000..9efbb1224 --- /dev/null +++ b/packages/chord/test/next/documents/definition.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, expectTypeOf, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type StateDefinition } from "../../../src/next/documents/index.ts"; +import { metered } from "../support.ts"; + +describe("defineState", () => { + it("types the value and whether it is keyed", () => { + const Notes = defineState("notes", { + schema: obj({ list: arr(str()) }), + version: 1, + initial: () => ({ list: [] }), + }); + const Drafts = defineState("drafts", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: 0 }), + keyed: true, + }); + expectTypeOf(Notes).toEqualTypeOf>(); + expectTypeOf(Drafts).toEqualTypeOf>(); + expect([Notes.keyed, Drafts.keyed]).toEqual([false, true]); + }); + + it('takes a restricted name only: lowercase [a-z0-9@/._~-] (decision "Restricted names")', () => { + const options = { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }; + for (const name of ["@app/notes.v_1~x-y", "n"]) expect(defineState(name, options).name).toBe(name); + for (const name of ["", "Notes", "app:notes", "app notes", "a\0b", "a\uD800", "é"]) { + expect(() => defineState(name, options)).toThrow( + `A state name ${JSON.stringify(name)}: expected lowercase [a-z0-9@/._~-]`, + ); + } + // Plugin code, not type-checked. + expect(() => defineState(42 as never, options)).toThrow("A state name 42"); + }); +}); + +describe("initial() (R8)", () => { + it("is checked against the schema once, the first time it is used", async () => { + const telemetry = await metered(); + const initial = vi.fn(() => ({ n: 0 })); + const Doc = defineState("doc", { schema: obj({ n: int() }), version: 1, initial }); + expect(Doc.initial(telemetry.context)).toEqual({ n: 0 }); + expect(Doc.initial(telemetry.context)).toEqual({ n: 0 }); + expect(initial).toHaveBeenCalledTimes(2); + expect(telemetry.counts()).toEqual({ "chord.data.compilations": 1, "chord.data.validations": 1 }); + }); + + it("refuses a value that does not match, every time until one does", () => { + let value: unknown = { n: "x" }; + const Doc = defineState("doc", { schema: obj({ n: int() }), version: 1, initial: () => value as { n: number } }); + expect(() => Doc.initial(context)).toThrow("doc: initial() does not match the schema"); + expect(() => Doc.initial(context)).toThrow("doc: initial() does not match the schema"); + value = { n: 1 }; + expect(Doc.initial(context)).toEqual({ n: 1 }); + }); +}); + +describe("fingerprints", () => { + it("are computed once from the schema's structure: descriptions ignored, a structural change seen", () => { + const make = (schema: Parameters[1]["schema"]) => + defineState("doc", { schema, version: 1, initial: () => ({ n: 0 }) as never }).fingerprint; + const plain = make(obj({ n: int() })); + expect(plain).toMatch(/^[0-9a-f]{16}$/); + expect(make(obj({ n: { ...int(), description: "a count" } }))).toBe(plain); + expect(make(obj({ n: str() }))).not.toBe(plain); + }); +}); + +describe("migrate (R8)", () => { + it("maps an older value and checks the result against the schema each time it produces one", async () => { + const telemetry = await metered(); + const Doc = defineState("doc", { + schema: obj({ n: int() }), + version: 2, + initial: () => ({ n: 0 }), + migrate: (old, from) => ({ n: (old.count as number) * from }), + }); + expect(Doc.migrate({ count: 3 }, 1, telemetry.context)).toEqual({ n: 3 }); + expect(() => Doc.migrate({ count: "x" }, 1, telemetry.context)).toThrow( + "doc: migrate() from version 1 does not match the schema", + ); + expect(telemetry.counts()["chord.data.validations"]).toBe(2); + }); + + it("refuses an older value without a migrate", () => { + const Doc = defineState("doc", { schema: obj({ n: int() }), version: 2, initial: () => ({ n: 0 }) }); + expect(() => Doc.migrate({ n: 1 }, 1, context)).toThrow("doc: no migrate from version 1 to 2"); + }); +}); diff --git a/packages/chord/test/next/documents/documents.test.ts b/packages/chord/test/next/documents/documents.test.ts new file mode 100644 index 000000000..acf9252da --- /dev/null +++ b/packages/chord/test/next/documents/documents.test.ts @@ -0,0 +1,111 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type KeyedHandle, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { delivered, framesOf, loaded, rejection } from "../support.ts"; + +const List = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); +const Drafts = defineState("drafts", { + schema: obj({ body: str() }), + version: 1, + initial: () => ({ body: "" }), + keyed: true, +}); +const Owner = local("app.owner", { + version: 1, + params: { id: int() }, + members: { list: obj({ items: arr(str()) }), touch: method() }, +}); + +/** Serves `Owner` and hands out its handles. */ +async function owner() { + const handles: { list?: SingletonHandle<{ items: string[] }>; drafts?: KeyedHandle<{ body: string }> } = {}; + const { host } = await loaded((h) => + h.serve(Owner, (_, self) => { + handles.list = self.state(List); + handles.drafts = self.state(Drafts); + return { list: handles.list, touch: async () => null }; + }), + ); + const reference = await host.get(Owner, { id: 1 }, context); + return { host, reference, list: handles.list!, drafts: handles.drafts! }; +} + +/** The keys a keyed handle lists. */ +async function keysOf(handle: KeyedHandle<{ body: string }>): Promise { + const keys: string[] = []; + for await (const key of handle.keys(context)) keys.push(key); + return keys; +} + +describe("singleton documents", () => { + it("read initial() until written and again after a retire, whose frame replaces the root", async () => { + const { list } = await owner(); + const initial = list.value; + expect(initial).toEqual({ items: [] }); + const frames = framesOf(await list.watch(context)); + await list.change((draft) => draft.items.push("a"), context); + expect(list.value).toEqual({ items: ["a"] }); + await list.retire(context); + expect(list.value).toBe(initial); + await list.retire(context); + await delivered(); + expect(frames).toEqual([ + [{ items: ["a"] }, [["p", ["items"], 0, 0, ["a"]]]], + [initial, [["r", initial]]], + ]); + }); + + it("throw not_ready when read in the factory", async () => { + const { host } = await loaded((h) => + h.serve(Owner, (_, self) => { + const list = self.state(List); + void list.value; + return { list, touch: async () => null }; + }), + ); + await expect(host.get(Owner, { id: 1 }, context)).rejects.toThrow("list: read before activation"); + }); +}); + +describe("keyed documents", () => { + it("are absent until written; get, keys and retire by key", async () => { + const { drafts } = await owner(); + expect(await drafts.get("a", context)).toBeUndefined(); + await drafts.change( + "a", + (draft) => { + draft.body = "x"; + }, + context, + ); + await drafts.change( + "b", + (draft) => { + draft.body = "y"; + }, + context, + ); + await drafts.change("c", () => {}, context); + expect(await drafts.get("a", context)).toEqual({ body: "x" }); + expect(await keysOf(drafts)).toEqual(["a", "b"]); + await drafts.retire("a", context); + await drafts.retire("missing", context); + expect(await drafts.get("a", context)).toBeUndefined(); + expect(await keysOf(drafts)).toEqual(["b"]); + }); +}); + +describe("when the owner ends", () => { + it("documents are dropped: every use throws released and watches end", async () => { + const { reference, list, drafts } = await owner(); + const watch = await list.watch(context); + reference.release(); + expect(await watch.closed).toEqual({ kind: "ended" }); + const released = { code: "released", reason: "ended", message: "app.owner(1): ended" }; + expect(() => list.value).toThrow("app.owner(1): ended"); + expect(await rejection(drafts.get("a", context))).toEqual(released); + expect(await rejection(keysOf(drafts))).toEqual(released); + expect(await rejection(list.change(() => {}, context))).toEqual(released); + }); +}); diff --git a/packages/chord/test/next/documents/fence.test.ts b/packages/chord/test/next/documents/fence.test.ts new file mode 100644 index 000000000..3bb649e38 --- /dev/null +++ b/packages/chord/test/next/documents/fence.test.ts @@ -0,0 +1,14 @@ +// Seal tokens (slice 6a §3, decision "The seal fence"): open until sealed, sealed for good. + +import { describe, expect, it } from "vitest"; +import { SealToken } from "../../../src/next/documents/internal.ts"; + +describe("seal tokens", () => { + it("start open and stay sealed once sealed", () => { + const token = new SealToken(); + expect(token.sealed).toBe(false); + token.seal(); + token.seal(); + expect(token.sealed).toBe(true); + }); +}); diff --git a/packages/chord/test/next/documents/handles.test.ts b/packages/chord/test/next/documents/handles.test.ts new file mode 100644 index 000000000..a574cb51a --- /dev/null +++ b/packages/chord/test/next/documents/handles.test.ts @@ -0,0 +1,75 @@ +// Document handles (slice 6a §4): keyed handles' reads, watches and `at(key)`, and sealed handles refusing every use. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type KeyedHandle, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { instancesOf, rejection, restartable } from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +const Body = defineState("body", { schema: obj({ text: str() }), version: 1, initial: () => ({ text: "" }) }); +const Notes = defineState("notes", { + schema: obj({ text: str() }), + version: 1, + initial: () => ({ text: "" }), + keyed: true, +}); + +/** The keys a keyed handle lists. */ +async function keysOf(handle: KeyedHandle<{ text: string }>): Promise { + const keys: string[] = []; + for await (const key of handle.keys(context)) keys.push(key); + return keys; +} + +describe("keyed handles", () => { + it("list keys in code point order, not UTF-16 order", async () => { + let notes: KeyedHandle<{ text: string }> | undefined; + await restartable(objects, (_, self) => { + notes = self.state(Notes); + }); + // U+FF5E sorts before U+1F600 by code point; as UTF-16 its unit 0xFF5E sorts after the surrogate 0xD83D. + for (const key of ["\u{1F600}", "\uFF5E", "b", "a"]) + await notes!.change( + key, + (d) => { + d.text = key; + }, + context, + ); + expect(await keysOf(notes!)).toEqual(["a", "b", "\uFF5E", "\u{1F600}"]); + expect(await notes!.get("\uFF5E", context)).toEqual({ text: "\uFF5E" }); + }); +}); + +describe("sealed handles", () => { + it("still read synchronously and from memory; changes and retirements are refused released", async () => { + let body: SingletonHandle<{ text: string }> | undefined; + let notes: KeyedHandle<{ text: string }> | undefined; + const { fail } = await restartable(objects, (_, self) => { + body = self.state(Body); + notes = self.state(Notes); + }); + await body!.change((d) => { + d.text = "x"; + }, context); + await notes!.change( + "k", + (d) => { + d.text = "y"; + }, + context, + ); + fail(); + expect(body!.value).toEqual({ text: "x" }); + expect(await notes!.get("k", context)).toEqual({ text: "y" }); + expect(await keysOf(notes!)).toEqual(["k"]); + const released = { code: "released", reason: "ended", message: "app.restartable(1): ended" }; + expect(await rejection(body!.change(() => {}, context))).toEqual(released); + expect(await rejection(body!.retire(context))).toEqual(released); + expect(await rejection(notes!.change("k", () => {}, context))).toEqual(released); + expect(await rejection(notes!.retire("k", context))).toEqual(released); + }); +}); diff --git a/packages/chord/test/next/documents/keyed-slot.test.ts b/packages/chord/test/next/documents/keyed-slot.test.ts new file mode 100644 index 000000000..674ff1e85 --- /dev/null +++ b/packages/chord/test/next/documents/keyed-slot.test.ts @@ -0,0 +1,704 @@ +// Keyed slots (slice 6a §4, §10, §11): keyed reads with a context, per-key watches and `at(key)` sources with their +// pins, persisted keyed documents loaded on demand on the object's line (one load per key, shared), and their +// eviction. + +import { afterEach, describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { keyed, local, obj, str } from "../../../src/next/contracts/index.ts"; +import { ChordError } from "../../../src/next/data/index.ts"; +import { defineState, type KeyedHandle, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { RETENTION_MS } from "../../../src/next/documents/residency.ts"; +import type { StoreSelf } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import type { Watchable, WatchHandle } from "../../../src/watch.ts"; +import { + changesOf, + delivered, + expectCollected, + failLive, + fakeTimers, + flushed, + holdNextCommit, + instancesOf, + loaded, + memoryStore, + rejection, + restartable, + storeRestartable, +} from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +type Note = { title: string }; + +const Memo = defineState("notes.memo", { + schema: obj({ title: str() }), + version: 1, + initial: () => ({ title: "" }), + keyed: true, +}); + +const Note = defineState("notes.note", { + schema: obj({ title: str() }), + version: 1, + initial: () => ({ title: "" }), + keyed: true, + persist: true, + checkpointWhen: () => false, +}); + +const stores: ChordStore[] = []; + +afterEach(async () => { + for (const store of stores.splice(0)) await store.close(context); +}); + +/** A fresh `:memory:` store, closed after the test. */ +async function freshStore(): Promise { + const store = await memoryStore({ outbox: true }); + stores.push(store); + return store; +} + +/** + * `StoreRestartable` over `store` declaring `Note` (and what `declare` adds) per attempt; `notes()` is the latest + * attempt's handle. + */ +async function persisted( + store: ChordStore, + declare: (attempt: number, self: StoreSelf, notes: KeyedHandle) => void = () => {}, +): Promise> & { readonly notes: () => KeyedHandle }> { + let notes: KeyedHandle | undefined; + const restartable = await storeRestartable( + objects, + (attempt, self) => { + notes = self.state(Note); + declare(attempt, self, notes); + }, + { store }, + ); + return { ...restartable, notes: () => notes! }; +} + +/** Writes `title` at `key` in `store`'s `Note` documents through a host of its own. */ +async function seed(store: ChordStore, entries: Record): Promise { + const { host, notes } = await persisted(store); + for (const [key, title] of Object.entries(entries)) { + await notes().change( + key, + (draft) => { + draft.title = title; + }, + context, + ); + } + await host.close(context); +} + +describe("keyed reads in memory", () => { + it("get and keys take a context; keys come in code point order", async () => { + let memos: KeyedHandle | undefined; + const { host } = await restartable(objects, (_, self) => { + memos = self.state(Memo); + }); + for (const key of ["b", "\u{1F600}", "a", "\uFFFF"]) + await memos!.change( + key, + (d) => { + d.title = key; + }, + context, + ); + expect(await memos!.get("a", context)).toEqual({ title: "a" }); + expect(await memos!.get("z", context)).toBeUndefined(); + const keys: string[] = []; + for await (const key of memos!.keys(context)) keys.push(key); + expect(keys).toEqual(["a", "b", "\uFFFF", "\u{1F600}"]); + await host.close(context); + }); +}); + +describe("migration on access", () => { + it("a newer version migrates each in-memory key when it is read; a key whose migrate fails fails the owner then", async () => { + const migrated: string[] = []; + const MemoV2 = defineState("notes.memo", { + schema: obj({ title: str() }), + version: 2, + initial: () => ({ title: "" }), + keyed: true, + migrate: (old) => { + const title = old.title as string; + if (title === "bad") throw new Error("cannot migrate"); + migrated.push(title); + return { title: `${title}!` }; + }, + }); + const memos: KeyedHandle[] = []; + const { attempts, restart, errors, reference } = await restartable(objects, (attempt, self) => { + memos.push(self.state(attempt === 1 ? Memo : MemoV2)); + }); + for (const key of ["a", "b", "c"]) { + await memos[0]!.change( + key, + (draft) => { + draft.title = key === "c" ? "bad" : key; + }, + context, + ); + } + await restart(); + // The activation migrated nothing: no key was read. + expect(migrated).toEqual([]); + expect(await memos[1]!.get("a", context)).toEqual({ title: "a!" }); + expect(migrated).toEqual(["a"]); + await attempts[1]!.tx(async (tx) => { + expect(await tx.draft(memos[1]!, "b")).toEqual({ title: "b!" }); + }, context); + expect(migrated).toEqual(["a", "b"]); + expect(await rejection(memos[1]!.get("c", context))).toMatchObject({ + message: expect.stringContaining("cannot migrate"), + }); + expect(errors.map(({ kind }) => kind)).toEqual(["object_failed", "object_failed"]); + expect(reference.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + }); +}); + +describe("keys and the text rules", () => { + /** Every key of `handle`, in order. */ + async function keysOf(handle: KeyedHandle): Promise { + const keys: string[] = []; + for await (const key of handle.keys(context)) keys.push(key); + return keys; + } + + const title = (value: string) => (draft: Note) => { + draft.title = value; + }; + + it("a lone surrogate in a key is U+FFFD at every entry point, in memory and in the store; NUL is refused", async () => { + const store = await freshStore(); + const { host, notes, attempts } = await persisted(store); + await notes().change("x\uD800", title("one"), context); + expect(await keysOf(notes())).toEqual(["x\uFFFD"]); + // One key: the change through either spelling reaches the same document and record. + await notes().change("x\uFFFD", title("two"), context); + await attempts[0]!.tx(async (tx) => { + (await tx.draft(notes(), "x\uD800")).title += "!"; + }, context); + expect(await notes().get("x\uD800", context)).toEqual({ title: "two!" }); + expect((await notes().watch("x\uD800", context))?.value).toEqual({ title: "two!" }); + for (const use of [ + () => notes().change("a\0b", title("nul"), context), + () => notes().get("a\0b", context), + () => notes().watch("a\0b", context), + () => notes().retire("a\0b", context), + ]) { + expect(await rejection(use())).toMatchObject({ code: "invalid", reason: "args" }); + } + await notes().retire("x\uD800", context); + expect(await keysOf(notes())).toEqual([]); + await host.close(context); + // The store has the key as memory had it. + const again = await persisted(store); + expect(await again.notes().get("x\uFFFD", context)).toBeUndefined(); + await again.host.close(context); + }); + + it("in memory too: one key per spelling, NUL refused", async () => { + let memos: KeyedHandle | undefined; + const { host } = await restartable(objects, (_, self) => { + memos = self.state(Memo); + }); + await memos!.change("x\uDC00", title("one"), context); + expect(await memos!.get("x\uFFFD", context)).toEqual({ title: "one" }); + expect(await keysOf(memos!)).toEqual(["x\uFFFD"]); + expect(await rejection(memos!.change("\0", title("nul"), context))).toMatchObject({ code: "invalid" }); + await host.close(context); + }); + + it("the empty key is a key: listed first, also on a later page", async () => { + const store = await freshStore(); + const entries: Record = { "": "empty" }; + for (let i = 0; i < 120; i++) entries[`k${String(i).padStart(3, "0")}`] = "x"; + await seed(store, entries); + const { host, notes } = await persisted(store); + const keys = await keysOf(notes()); + expect(keys.length).toBe(121); + expect(keys[0]).toBe(""); + expect(await notes().get("", context)).toEqual({ title: "empty" }); + await host.close(context); + }); +}); + +describe("keyed watches", () => { + it("watch(key) is undefined when absent; it follows the address through a retirement and a recreation", async () => { + let memos: KeyedHandle | undefined; + const { host, attempts } = await restartable(objects, (_, self) => { + memos = self.state(Memo); + }); + expect(await memos!.watch("a", context)).toBeUndefined(); + await memos!.change( + "a", + (d) => { + d.title = "x"; + }, + context, + ); + const watch = (await memos!.watch("a", context))!; + expect(watch.value).toEqual({ title: "x" }); + const changes = changesOf(watch); + await memos!.change( + "a", + (d) => { + d.title = "y"; + }, + context, + ); + await delivered(); + await memos!.retire("a", context); + await delivered(); + await memos!.change( + "a", + (d) => { + d.title = "z"; + }, + context, + ); + await delivered(); + await attempts[0]!.tx(async (tx) => { + await tx.retire(memos!, "a"); + (await tx.draft(memos!, "a")).title = "w"; + }, context); + await delivered(); + expect(changes).toEqual([ + { kind: "value", value: { title: "y" }, ops: [["s", ["title"], "y"]], retired: false }, + { kind: "value", value: { title: "" }, ops: [["r", { title: "" }]], retired: true }, + { kind: "value", value: { title: "z" }, ops: [["r", { title: "z" }]], retired: false }, + { kind: "value", value: { title: "w" }, ops: [["r", { title: "w" }]], retired: true }, + ]); + await host.close(context); + expect(await watch.closed).toEqual({ kind: "ended" }); + }); + + it("a stopped watch unpins its key: its source is collected", async () => { + let memos: KeyedHandle | undefined; + const { host } = await restartable(objects, (_, self) => { + memos = self.state(Memo); + }); + await memos!.change( + "a", + (d) => { + d.title = "x"; + }, + context, + ); + let watch: WatchHandle | undefined = (await memos!.watch("a", context))!; + // The watch's source: the slot's per-key source while pinned. + const slot = (memos as unknown as { slot: { source(key: string): object } }).slot; + const source = new WeakRef(slot.source("a")); + await watch.stop(); + watch = undefined; + await flushed(); + await expectCollected(source); + await host.close(context); + }); +}); + +describe("at(key)", () => { + it("is initial() while absent, a state member, derive and on source; retirement arrives with the flag", async () => { + let memos: KeyedHandle | undefined; + let pinned: Watchable | undefined; + let upper: Watchable | undefined; + const seen: { title: string; retired: boolean }[] = []; + const { host } = await restartable(objects, (_, self) => { + memos = self.state(Memo); + pinned = memos.at("a"); + upper = self.derive([pinned], ([note]) => note.title.toUpperCase()); + self.on(pinned, async ({ value, retired }) => void seen.push({ title: value.title, retired })); + }); + expect(pinned!.value).toEqual({ title: "" }); + const changes = changesOf(await upper!.watch(context)); + await memos!.change( + "a", + (d) => { + d.title = "x"; + }, + context, + ); + await delivered(); + expect(pinned!.value).toEqual({ title: "x" }); + await memos!.retire("a", context); + await delivered(); + await flushed(); + expect(changes).toEqual([ + { kind: "value", value: "X", ops: [["r", "X"]], retired: false }, + { kind: "value", value: "", ops: [["r", ""]], retired: true }, + ]); + expect(seen).toEqual([ + { title: "", retired: false }, + { title: "x", retired: false }, + { title: "", retired: true }, + ]); + await host.close(context); + }); + + it("only in the factory", async () => { + let memos: KeyedHandle | undefined; + const { host } = await restartable(objects, (_, self) => { + memos = self.state(Memo); + }); + expect(() => memos!.at("a")).toThrow("valid only in the factory"); + await host.close(context); + }); + + it("a persisted key is loaded before the object goes live and stays pinned until the next factory returns", async () => { + const store = await freshStore(); + await seed(store, { a: "x", b: "y" }); + let pinned: Watchable | undefined; + const { host, restart, notes } = await persisted(store, (attempt, _, notes) => { + pinned = notes.at(attempt === 1 ? "a" : "b"); + }); + expect(pinned!.value).toEqual({ title: "x" }); + const slot = (notes() as unknown as { slot: { source(key: string): object | undefined } }).slot; + const first = new WeakRef(slot.source("a")!); + await restart(); + expect(pinned!.value).toEqual({ title: "y" }); + expect(slot.source("a")).toBeUndefined(); + await expectCollected(first); + await host.close(context); + }); +}); + +describe("cold keyed loads on the line", () => { + it("load a cold key once; a cached get touches no SQLite; an absent key is not cached", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const { host, notes } = await persisted(store); + const read = vi.spyOn(store, "read"); + expect(await notes().get("a", context)).toEqual({ title: "x" }); + expect(read).toHaveBeenCalledTimes(1); + expect(await notes().get("a", context)).toEqual({ title: "x" }); + expect(read).toHaveBeenCalledTimes(1); + expect(await notes().get("z", context)).toBeUndefined(); + expect(await notes().get("z", context)).toBeUndefined(); + expect(read).toHaveBeenCalledTimes(3); + await host.close(context); + }); + + it("two readers share one load; a cancelled waiter rejects alone", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const { host, notes, attempts } = await persisted(store); + const read = vi.spyOn(store, "read"); + // Hold the line so that the load waits behind it. + const { promise: holding, resolve: release } = Promise.withResolvers(); + const tx = attempts[0]!.tx(() => holding, context); + const cancel = new AbortController(); + const first = notes().get("a", { ...context, abortSignal: cancel.signal }); + const second = notes().get("a", context); + cancel.abort(new Error("gone")); + await expect(first).rejects.toThrow("gone"); + release(); + await tx; + expect(await second).toEqual({ title: "x" }); + expect(read).toHaveBeenCalledTimes(1); + await host.close(context); + }); + + it("a cold get racing a draft of the same key: the load installs only while the key is absent", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const { host, notes, attempts } = await persisted(store); + let get: Promise | undefined; + await attempts[0]!.tx(async (tx) => { + get = notes().get("a", context); + const draft = await tx.draft(notes(), "a"); + expect(draft.title).toBe("x"); + draft.title = "y"; + }, context); + expect(await get).toEqual({ title: "y" }); + expect(await notes().get("a", context)).toEqual({ title: "y" }); + await host.close(context); + }); + + it("keys come from the store in code point order, a page per line job", async () => { + const store = await freshStore(); + const entries: Record = {}; + for (let i = 0; i < 150; i++) entries[`k${String(i).padStart(3, "0")}`] = "x"; + entries["\u{1F600}"] = "x"; + entries["\uFFFF"] = "x"; + await seed(store, entries); + const { host, notes } = await persisted(store); + const keys: string[] = []; + for await (const key of notes().keys(context)) keys.push(key); + expect(keys).toEqual([...Object.keys(entries).filter((key) => key.startsWith("k")), "\uFFFF", "\u{1F600}"]); + await host.close(context); + }); + + it("tx.retire of a cold key finds it without loading it; an absent key retires nothing", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const { host, notes } = await persisted(store); + const seq = store.lastSeq; + await notes().retire("z", context); + expect(store.lastSeq).toBe(seq); + await notes().retire("a", context); + expect(store.lastSeq).toBe(seq + 1); + expect(await notes().get("a", context)).toBeUndefined(); + await host.close(context); + }); + + it("a sealed handle reads cached documents; a cold read is refused released", async () => { + const store = await freshStore(); + await seed(store, { a: "x", b: "y" }); + const { host, notes, restart } = await persisted(store); + const old = notes(); + await old.get("a", context); + await restart(); + expect(await old.get("a", context)).toEqual({ title: "x" }); + expect(await rejection(old.get("b", context))).toMatchObject({ code: "released" }); + await host.close(context); + }); + + it("a sealed handle joining a cold load in flight is refused released (rule 4)", async () => { + const store = await freshStore(); + await seed(store, { b: "y" }); + const { host, notes, restart } = await persisted(store); + const gate = holdNextRead(store); + const old = notes(); + const first = old.get("b", context); + await gate.reached; + const restarting = restart(); + await delivered(); + expect(await rejection(old.get("b", context))).toMatchObject({ code: "released" }); + gate.open(); + expect(await first).toEqual({ title: "y" }); + await restarting; + await host.close(context); + }); + + it("cold reads queued on the line before the object's end are refused released when they run after it", async () => { + const store = await freshStore(); + await seed(store, { a: "x", b: "y", c: "z" }); + const { host, notes } = await persisted(store); + const gate = holdNextRead(store); + const first = notes().get("a", context); + await gate.reached; + const get = rejection(notes().get("b", context)); + const watch = rejection(notes().watch("c", context)); + const closing = host.close(context); + gate.open(); + expect(await get).toMatchObject({ code: "released" }); + expect(await watch).toMatchObject({ code: "released" }); + // The load holding the line before the end finishes. + expect(await first).toEqual({ title: "x" }); + await closing; + }); + + it("cold reads queued behind an admitted step are refused released when their attempt failed live meanwhile", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const P = defineState("notes.p", { + schema: obj({ title: str() }), + version: 1, + initial: () => ({ title: "" }), + persist: true, + checkpointWhen: () => false, + }); + let p: SingletonHandle | undefined; + const { host, notes, attempts } = await persisted(store, (_, self) => { + p = self.state(P); + }); + const held = holdNextCommit(store); + const change = p!.change((draft) => { + draft.title = "p"; + }, context); + await held.entered; + const get = rejection(notes().get("a", context)); + const watch = rejection(notes().watch("a", context)); + const keys = rejection( + (async () => { + for await (const _ of notes().keys(context)); + })(), + ); + // The attempt is sealed; the object stays open. + failLive(objects, attempts[0]!, new ChordError({ code: "unavailable", message: "boom" })); + held.release(); + await change; + expect(await get).toMatchObject({ code: "released" }); + expect(await watch).toMatchObject({ code: "released" }); + expect(await keys).toMatchObject({ code: "released" }); + await host.close(context); + }); + + it("a still-valid waiter sharing that load gets the document: only the sealed handle's read is refused", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const Notes = local("app.store", { version: 1, params: {}, members: { notes: keyed(obj({ title: str() })) } }); + let self: StoreSelf | undefined; + let notes: KeyedHandle | undefined; + const { host } = await loaded( + (h) => + h.serve(Notes, (_, s) => { + self = s; + notes = s.state(Note); + return { notes }; + }), + { store }, + ); + const ref = await host.get(Notes, {}, context); + const held = holdNextCommit(store); + const change = notes!.change( + "b", + (draft) => { + draft.title = "y"; + }, + context, + ); + await held.entered; + const sealed = rejection(notes!.get("a", context)); + // A consumer reads as the provider's own code may (the object's own token), joining the same load. + const shared = ref.notes.get("a", context); + await flushed(); + failLive(objects, self!, new ChordError({ code: "unavailable", message: "boom" })); + held.release(); + await change; + expect(await sealed).toMatchObject({ code: "released" }); + expect(await shared).toEqual({ title: "x" }); + await host.close(context); + }); + + it("a load cut by the seal never installs: a successor's retirement stands", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const { host, notes, attempts, restart } = await persisted(store); + const gate = holdNextRead(store); + const old = notes(); + const cut = attempts[0]!.tx(async (tx) => { + (await tx.draft(old, "a")).title = "z"; + }, context); + await gate.reached; + // The seal cuts the transaction and releases the line; the successor retires the key. + await restart(); + await notes().retire("a", context); + gate.open(); + expect(await rejection(cut)).toMatchObject({ code: "released" }); + expect(await notes().get("a", context)).toBeUndefined(); + await host.close(context); + }); + + it("a plugin migrate throwing a ChordError fails its owner like any other error", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const V2 = defineState("notes.note", { + schema: obj({ title: str() }), + version: 2, + initial: () => ({ title: "" }), + migrate: () => { + throw new ChordError({ code: "unavailable", message: "plugin says no" }); + }, + keyed: true, + persist: true, + checkpointWhen: () => false, + }); + let notes: KeyedHandle | undefined; + const { host, errors, reference } = await storeRestartable( + objects, + (_, self) => { + notes = self.state(V2); + }, + { store }, + ); + expect(await rejection(notes!.get("a", context))).toMatchObject({ message: "plugin says no" }); + expect(errors.map(({ kind }) => kind)).toEqual(["object_failed"]); + expect(reference.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + await host.close(context); + }); + + it("a key whose migrate fails fails its owner broken; the get rejects with the cause", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const V2 = defineState("notes.note", { + schema: obj({ title: str() }), + version: 2, + initial: () => ({ title: "" }), + migrate: () => { + throw new Error("cannot migrate"); + }, + keyed: true, + persist: true, + checkpointWhen: () => false, + }); + let notes: KeyedHandle | undefined; + const { host, errors, reference } = await storeRestartable( + objects, + (_, self) => { + notes = self.state(V2); + }, + { store }, + ); + expect(await rejection(notes!.get("a", context))).toMatchObject({ + code: "unavailable", + message: expect.stringContaining("cannot migrate"), + }); + expect(errors.map(({ kind }) => kind)).toEqual(["object_failed"]); + expect(reference.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + await host.close(context); + }); +}); + +describe("eviction", () => { + fakeTimers(); + + it("an unpinned key idle for the retention leaves memory; a pinned one stays", async () => { + const store = await freshStore(); + await seed(store, { a: "x", b: "y" }); + const { host, notes } = await persisted(store); + const read = vi.spyOn(store, "read"); + await notes().get("a", context); + const watch = (await notes().watch("b", context))!; + expect(read).toHaveBeenCalledTimes(2); + vi.advanceTimersByTime(RETENTION_MS + 1_000); + await flushed(); + await notes().get("a", context); + await notes().get("b", context); + expect(read).toHaveBeenCalledTimes(3); + await watch.stop(); + await flushed(); + vi.advanceTimersByTime(RETENTION_MS + 1_000); + await flushed(); + await notes().get("b", context); + expect(read).toHaveBeenCalledTimes(4); + await host.close(context); + }); + + it("an evicted document is collected", async () => { + const store = await freshStore(); + await seed(store, { a: "x" }); + const { host, notes } = await persisted(store); + let value: Note | undefined = await notes().get("a", context); + const collected = new WeakRef(value!); + value = undefined; + vi.advanceTimersByTime(RETENTION_MS + 1_000); + await flushed(); + await expectCollected(collected); + expect(await notes().get("a", context)).toEqual({ title: "x" }); + await host.close(context); + }); +}); + +/** Holds `store`'s next read after the backend returned its result, until `open()`. */ +function holdNextRead(store: ChordStore): { readonly reached: Promise; open(): void } { + const reached = Promise.withResolvers(); + const gate = Promise.withResolvers(); + const read = store.read.bind(store); + store.read = async (fn, ctx) => { + store.read = read; + const result = await read(fn, ctx); + reached.resolve(); + await gate.promise; + return result; + }; + return { reached: reached.promise, open: () => gate.resolve() }; +} diff --git a/packages/chord/test/next/documents/line.test.ts b/packages/chord/test/next/documents/line.test.ts new file mode 100644 index 000000000..b473cc378 --- /dev/null +++ b/packages/chord/test/next/documents/line.test.ts @@ -0,0 +1,45 @@ +// The line (§9): holders run one at a time in the order they asked; a release given twice changes nothing. + +import { describe, expect, it } from "vitest"; +import { Line } from "../../../src/next/documents/line.ts"; +import { flushed } from "../support.ts"; + +describe("the line", () => { + it("hands the line to each holder in the order they asked, once the previous one released", async () => { + const line = new Line(); + const order: string[] = []; + const first = await line.acquire(); + const second = line.acquire().then((release) => { + order.push("second"); + return release; + }); + const third = line.acquire().then((release) => { + order.push("third"); + return release; + }); + await flushed(); + expect(order).toEqual([]); + first(); + (await second)(); + await third; + expect(order).toEqual(["second", "third"]); + }); + + it("ignores a second release of the same holder", async () => { + const line = new Line(); + const first = await line.acquire(); + const second = line.acquire(); + let thirdHeld = false; + const third = line.acquire().then(() => { + thirdHeld = true; + }); + first(); + first(); + await second; + await flushed(); + expect(thirdHeld).toBe(false); + (await second)(); + await third; + expect(thirdHeld).toBe(true); + }); +}); diff --git a/packages/chord/test/next/documents/persistence.test.ts b/packages/chord/test/next/documents/persistence.test.ts new file mode 100644 index 000000000..6b97b83a2 --- /dev/null +++ b/packages/chord/test/next/documents/persistence.test.ts @@ -0,0 +1,292 @@ +// Persisted documents' store side (slice 6a §8, §10): loads with the version and fingerprint checks and `migrate`, +// finds, keys, and the writes one commit stages (creation, base or delta by `checkpointWhen`, retirement). + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { track } from "../../../src/delta/index.ts"; +import { int, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type JsonObject } from "../../../src/next/documents/index.ts"; +import { Persistence, type Stored, storedAfter } from "../../../src/next/documents/persistence.ts"; +import type { AnyDefinition } from "../../../src/next/documents/slot.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { memoryStore, metered } from "../support.ts"; + +const context = BACKGROUND_CONTEXT; + +const Note = defineState("notes.note", { + schema: obj({ title: str(), n: int() }), + version: 1, + initial: () => ({ title: "", n: 0 }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 2, +}); + +let store: ChordStore; +let notes: Persistence; + +beforeEach(async () => { + store = await memoryStore({ outbox: true }); + notes = new Persistence(store, "session:app.store", "notes.note"); +}); + +afterEach(async () => { + await store.close(context); +}); + +/** Stages the change of `before` to `after` at `key` and commits it; resolves with what is stored afterwards. */ +async function change( + key: string, + before: JsonObject | undefined, + after: (draft: JsonObject) => void, + stored: Stored | undefined, + definition: AnyDefinition = Note, +): Promise { + const tracker = track(before ?? definition.initial(context)); + const draft = tracker.beginChange(); + after(draft.state); + const prepared = draft.prepare(); + const writes: StagedWrite[] = []; + const result = notes.stage(key, definition, stored, { prepared, retires: false }, writes, 0); + let ids: readonly number[] = []; + await store.commit( + writes, + { + admit: () => true, + adopt: (_seq, created) => { + ids = created; + }, + }, + context, + ); + tracker.adopt(prepared); + return storedAfter(result, ids); +} + +/** The document at `key` read from the store and decoded by `definition`, `undefined` when absent. */ +async function load(key: string, definition: AnyDefinition, at: typeof context) { + const document = await notes.read(key, at); + return document === undefined ? undefined : notes.decode(document, definition, at); +} + +/** The kinds of the outbox's writes, per commit. */ +async function outboxKinds(): Promise { + const commits = await store.outbox!.read(0, 100, context); + return commits.map(({ writes }) => + writes.map((write) => (write.kind === "document.change" ? `change:${write.content.kind}` : write.kind)), + ); +} + +describe("loads", () => { + it("load an absent document as undefined and a stored one with its record", async () => { + expect(await load("a", Note, context)).toBeUndefined(); + const stored = await change( + "a", + undefined, + (d) => { + d.title = "x"; + }, + undefined, + ); + const { counts, context: counting } = await metered(); + expect(await load("a", Note, counting)).toEqual({ + value: { title: "x", n: 0 }, + stored: { id: stored!.id, revision: 1, version: 1, deltasSinceBase: 0 }, + }); + expect(counts()["chord.store.loads"]).toBe(1); + }); + + it("migrate an older stored version and check the result once", async () => { + const V1 = defineState("notes.note", { + schema: obj({ title: str() }), + version: 1, + initial: () => ({ title: "" }), + persist: true, + checkpointWhen: () => false, + }); + await change( + "a", + undefined, + (d) => { + d.title = "x"; + }, + undefined, + V1, + ); + const V2 = defineState("notes.note", { + schema: obj({ title: str(), n: int() }), + version: 2, + initial: () => ({ title: "", n: 0 }), + migrate: (old) => ({ title: String(old.title), n: 7 }), + persist: true, + checkpointWhen: () => false, + }); + const { counts, context: counting } = await metered(); + expect(await load("a", V2, counting)).toMatchObject({ + value: { title: "x", n: 7 }, + stored: { version: 1 }, + }); + expect(counts()["chord.data.validations"]).toBe(1); + const Bad = defineState("notes.note", { + schema: obj({ title: str(), n: int() }), + version: 2, + initial: () => ({ title: "", n: 0 }), + // A migrate returning a wrong shape: the load is refused. + migrate: (old) => ({ title: String(old.title) }) as { title: string; n: number }, + persist: true, + checkpointWhen: () => false, + }); + await expect(load("a", Bad, context)).rejects.toThrow("migrate() from version 1 does not match"); + }); + + it("refuse a stored version newer than the code's and the same version with another fingerprint (D9)", async () => { + await change( + "a", + undefined, + (d) => { + d.title = "x"; + }, + undefined, + ); + const Old = defineState("notes.note", { + schema: obj({ title: str(), n: int() }), + version: 0, + initial: () => ({ title: "", n: 0 }), + persist: true, + checkpointWhen: () => false, + }); + await expect(load("a", Old, context)).rejects.toThrow("stored at version 1, newer than the code's 0"); + const Edited = defineState("notes.note", { + schema: obj({ title: str(), n: int(), extra: str() }), + version: 1, + initial: () => ({ title: "", n: 0, extra: "" }), + persist: true, + checkpointWhen: () => false, + }); + await expect(load("a", Edited, context)).rejects.toThrow("stored with another schema at version 1"); + }); + + it("find a live record without loading it, and list keys in code point order by pages", async () => { + expect(await notes.find("a", context)).toBeUndefined(); + for (const key of ["b", "\u{1F600}", "a", "\uFFFF"]) + await change( + key, + undefined, + (d) => { + d.n = 1; + }, + undefined, + ); + expect(await notes.find("a", context)).toMatchObject({ revision: 1, version: 1 }); + expect(await notes.keys(undefined, 10, context)).toEqual(["a", "b", "\uFFFF", "\u{1F600}"]); + expect(await notes.keys("b", 1, context)).toEqual(["\uFFFF"]); + }); +}); + +describe("staged writes", () => { + it("store a creation as a base, changes as deltas until checkpointWhen asks for a base", async () => { + let stored = await change( + "a", + undefined, + (d) => { + d.n = 1; + }, + undefined, + ); + expect(stored).toMatchObject({ revision: 1, deltasSinceBase: 0 }); + let value = (await load("a", Note, context))!.value; + for (const n of [2, 3, 4]) { + stored = await change( + "a", + value, + (d) => { + d.n = n; + }, + stored, + ); + value = (await load("a", Note, context))!.value; + } + expect(stored).toMatchObject({ revision: 4, deltasSinceBase: 0 }); + expect(value).toEqual({ title: "", n: 4 }); + expect(await outboxKinds()).toEqual([["document.create"], ["change:delta"], ["change:delta"], ["change:base"]]); + expect((await load("a", Note, context))!.stored.deltasSinceBase).toBe(0); + }); + + it("store a base when the stored version is older than the definition's (the migrated base written back)", async () => { + const stored = await change( + "a", + undefined, + (d) => { + d.n = 1; + }, + undefined, + ); + const V2 = defineState("notes.note", { + schema: obj({ title: str(), n: int() }), + version: 2, + initial: () => ({ title: "", n: 0 }), + migrate: (old) => old as { title: string; n: number }, + persist: true, + checkpointWhen: () => false, + }); + const after = await change( + "a", + { title: "", n: 1 }, + (d) => { + d.n = 2; + }, + stored, + V2, + ); + expect(after).toMatchObject({ version: 2, revision: 2, deltasSinceBase: 0 }); + expect((await outboxKinds()).at(-1)).toEqual(["change:base"]); + expect(await load("a", V2, context)).toMatchObject({ value: { n: 2 }, stored: { version: 2 } }); + }); + + it("stage a retirement before a recreation, and number creations by their order in the commit", async () => { + const stored = await change( + "a", + undefined, + (d) => { + d.n = 1; + }, + undefined, + ); + const writes: StagedWrite[] = []; + const fresh = (n: number) => { + const tracker = track(Note.initial(context)); + const draft = tracker.beginChange(); + draft.state.n = n; + return draft.prepare(); + }; + const first = notes.stage("b", Note, undefined, { prepared: fresh(2), retires: false }, writes, 0); + const again = notes.stage("a", Note, stored, { prepared: fresh(3), retires: true }, writes, 1); + const gone = notes.stage("c", Note, undefined, { prepared: undefined, retires: true }, writes, 2); + expect([first, again, gone]).toEqual([ + { kind: "create", version: 1, index: 0 }, + { kind: "create", version: 1, index: 1 }, + { kind: "retire" }, + ]); + let ids: readonly number[] = []; + await store.commit( + writes, + { + admit: () => true, + adopt: (_seq, created) => { + ids = created; + }, + }, + context, + ); + expect(ids).toEqual([stored!.id + 1, stored!.id + 2]); + expect(storedAfter(again, ids)).toEqual({ id: stored!.id + 2, revision: 1, version: 1, deltasSinceBase: 0 }); + expect((await outboxKinds()).at(-1)).toEqual(["document.retire", "document.create", "document.create"]); + }); + + it("serialize at staging: a value JSON cannot encode fails there, never in the step (rule 8)", () => { + // Drafts refuse such values when placed; a prepared change made by hand shows where the text is made. + const value = { title: "", n: 1n } as unknown as JsonObject; + const prepared = { base: value, value, ops: [], baseRevision: 0, abort: () => {} }; + expect(() => notes.stage("a", Note, undefined, { prepared, retires: false }, [], 0)).toThrow(TypeError); + }); +}); diff --git a/packages/chord/test/next/documents/residency.test.ts b/packages/chord/test/next/documents/residency.test.ts new file mode 100644 index 000000000..1451cee02 --- /dev/null +++ b/packages/chord/test/next/documents/residency.test.ts @@ -0,0 +1,77 @@ +// Residency (slice 6a §10, rule 23): Durable's idle structure, one timer per object, the granule rule, eviction on +// the line oldest first. + +import { describe, expect, it, vi } from "vitest"; +import { track } from "../../../src/delta/index.ts"; +import { RETENTION_MS, Residency, Resident } from "../../../src/next/documents/residency.ts"; +import { fakeTimers, flushed } from "../support.ts"; + +/** A residency whose sweeps run at once (no line), and the keys it evicted, in order. */ +function idle() { + const evicted: string[] = []; + const slot = { evict: (resident: Resident) => void evicted.push(resident.key) }; + + const residency = new Residency((sweep) => { + sweep(); + }); + const resident = (key: string) => new Resident(slot, key, track({}), undefined, 1); + return { residency, resident, evicted }; +} + +describe("the idle structure", () => { + fakeTimers(); + + it("evict idle documents oldest first once their retention passed, with one timer", async () => { + const { residency, resident, evicted } = idle(); + const a = resident("a"); + const b = resident("b"); + residency.touch(a); + vi.advanceTimersByTime(5_000); + residency.touch(b); + expect(vi.getTimerCount()).toBe(1); + vi.advanceTimersByTime(RETENTION_MS - 5_000); + await flushed(); + expect(evicted).toEqual(["a"]); + expect(vi.getTimerCount()).toBe(1); + vi.advanceTimersByTime(5_000); + expect(evicted).toEqual(["a", "b"]); + expect(vi.getTimerCount()).toBe(0); + vi.advanceTimersByTime(RETENTION_MS); + expect(evicted).toEqual(["a", "b"]); + }); + + it("move a touched document to the back at most once per granule", () => { + const { residency, resident, evicted } = idle(); + const a = resident("a"); + const b = resident("b"); + residency.touch(a); + residency.touch(b); + vi.advanceTimersByTime(500); + residency.touch(a); + vi.advanceTimersByTime(RETENTION_MS - 500); + // Within its granule the touch did not move `a`: it leaves up to a granule early. + expect(evicted).toEqual(["a", "b"]); + residency.touch(a); + residency.touch(b); + vi.advanceTimersByTime(1_000); + residency.touch(a); + vi.advanceTimersByTime(RETENTION_MS - 1_000); + expect(evicted).toEqual(["a", "b", "b"]); + vi.advanceTimersByTime(1_000); + expect(evicted).toEqual(["a", "b", "b", "a"]); + }); + + it("forget pinned documents; after end nothing is evicted and no timer is left", () => { + const { residency, resident, evicted } = idle(); + const a = resident("a"); + residency.touch(a); + residency.forget(a); + vi.advanceTimersByTime(RETENTION_MS); + expect(evicted).toEqual([]); + residency.touch(resident("b")); + residency.end(); + expect(vi.getTimerCount()).toBe(0); + vi.advanceTimersByTime(RETENTION_MS); + expect(evicted).toEqual([]); + }); +}); diff --git a/packages/chord/test/next/documents/slot.test.ts b/packages/chord/test/next/documents/slot.test.ts new file mode 100644 index 000000000..7141fc6ab --- /dev/null +++ b/packages/chord/test/next/documents/slot.test.ts @@ -0,0 +1,375 @@ +// Singleton slots (slice 6a §4, §10, §11): retirement, slots kept across attempts, persisted singletons loaded once. + +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, obj, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { + defineState, + type JsonObject, + type KeyedHandle, + type SingletonHandle, +} from "../../../src/next/documents/index.ts"; +import type { HostError, StoreSelf } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { + changesOf, + delivered, + instancesOf, + loaded, + memoryStore, + rejection, + restartable, + StoreRestartable, + storeRestartable, + valuesOf, +} from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +const ListV1 = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); + +describe("retirement", () => { + it("reaches watches as initial() with retired: true", async () => { + let list: SingletonHandle<{ items: string[] }> | undefined; + await restartable(objects, (_, self) => { + list = self.state(ListV1); + }); + const initial = list!.value; + const changes = changesOf(await list!.watch(context)); + await list!.change((draft) => void draft.items.push("a"), context); + await list!.retire(context); + await delivered(); + expect(changes).toEqual([ + { kind: "value", value: { items: ["a"] }, ops: [["p", ["items"], 0, 0, ["a"]]], retired: false }, + { kind: "value", value: initial, ops: [["r", initial]], retired: true }, + ]); + }); + + it("a retire and a recreate of a present document in one transaction: one frame, retired, the new value", async () => { + let list: SingletonHandle<{ items: string[] }> | undefined; + const { attempts } = await restartable(objects, (_, self) => { + list = self.state(ListV1); + }); + await list!.change((draft) => void draft.items.push("a"), context); + const changes = changesOf(await list!.watch(context)); + await attempts[0]!.tx(async (tx) => { + await tx.retire(list!); + (await tx.draft(list!)).items.push("b"); + }, context); + await delivered(); + expect(changes).toEqual([ + { kind: "value", value: { items: ["b"] }, ops: [["r", { items: ["b"] }]], retired: true }, + ]); + }); +}); + +describe("across attempts", () => { + it("keep in-memory documents across a failure and retry; initial() is taken once", async () => { + let initials = 0; + const List = defineState("list", { + schema: obj({ items: arr(str()) }), + version: 1, + initial: () => { + initials++; + return { items: [] }; + }, + }); + const handles: SingletonHandle<{ items: string[] }>[] = []; + const { restart } = await restartable(objects, (_, self) => { + handles.push(self.state(List)); + }); + const changes = changesOf(await handles[0]!.watch(context)); + await handles[0]!.change((draft) => void draft.items.push("a"), context); + await restart(); + expect(handles[1]!.value).toEqual({ items: ["a"] }); + await handles[1]!.change((draft) => void draft.items.push("b"), context); + await delivered(); + // The watch taken through the first attempt's handle follows the slot across attempts. + expect(valuesOf(changes)).toEqual([{ items: ["a"] }, { items: ["a", "b"] }]); + expect(initials).toBe(1); + }); + + it("drop the in-memory documents a later factory no longer declares: their watches end", async () => { + let list: SingletonHandle<{ items: string[] }> | undefined; + const { restart } = await restartable(objects, (attempt, self) => { + if (attempt === 1) list = self.state(ListV1); + }); + await list!.change((draft) => void draft.items.push("a"), context); + const watch = await list!.watch(context); + await restart(); + expect(await watch.closed).toEqual({ kind: "ended" }); + }); + + it("keep them across a throwing factory: only a factory that returns drops what it does not declare", async () => { + const handles: SingletonHandle<{ items: string[] }>[] = []; + const { fail, retry, errors } = await restartable(objects, (attempt, self) => { + if (attempt === 2) throw new Error("factory threw"); + if (attempt !== 2) handles.push(self.state(ListV1)); + }); + await handles[0]!.change((draft) => void draft.items.push("a"), context); + fail(); + await rejection(retry()); + expect(failure(errors)).toBe("app.restartable(1): factory threw"); + await retry(); + expect(handles[1]!.value).toEqual({ items: ["a"] }); + }); + + it("migrate in memory when a later attempt declares a newer version; the old handles are refused", async () => { + let initials = 0; + const ListV2 = defineState("list", { + schema: obj({ entries: arr(obj({ text: str() })) }), + version: 2, + initial: () => { + initials++; + return { entries: [] }; + }, + migrate: (old) => ({ entries: (old.items as string[]).map((text) => ({ text })) }), + }); + const Notes1 = defineState("notes", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: 0 }), + keyed: true, + }); + const Notes2 = defineState("notes", { + schema: obj({ n: int(), m: int() }), + version: 2, + initial: () => ({ n: 0, m: 0 }), + keyed: true, + migrate: (old) => ({ n: old.n as number, m: 1 }), + }); + const lists: SingletonHandle[] = []; + const notes: KeyedHandle[] = []; + const inFactory: unknown[] = []; + const { restart } = await restartable(objects, (attempt, self) => { + lists.push(self.state(attempt === 1 ? ListV1 : ListV2) as unknown as SingletonHandle); + notes.push(self.state(attempt === 1 ? Notes1 : Notes2) as unknown as KeyedHandle); + // The new code's handle in its own factory: the values are still at version 1 until the activation. + if (attempt === 2) inFactory.push(rejectionOf(() => lists[1]!.value)); + }); + await lists[0]!.change((draft) => void (draft.items as string[]).push("a"), context); + await notes[0]!.change( + "k", + (draft) => { + draft.n = 5; + }, + context, + ); + const changes = changesOf(await lists[0]!.watch(context)); + await restart(); + expect(lists[1]!.value).toEqual({ entries: [{ text: "a" }] }); + expect(await notes[1]!.get("k", context)).toEqual({ n: 5, m: 1 }); + const stale = "list: a handle of version 1; the documents are at version 2"; + expect(() => lists[0]!.value).toThrow(stale); + expect(await rejection(notes[0]!.get("k", context))).toMatchObject({ code: "released" }); + await delivered(); + expect(changes).toEqual([ + { + kind: "value", + value: { entries: [{ text: "a" }] }, + ops: [["r", { entries: [{ text: "a" }] }]], + retired: false, + }, + ]); + expect(initials).toBe(1); + expect(inFactory).toEqual([expect.objectContaining({ code: "not_ready" })]); + }); + + it("fail the attempt broken for a flipped keyed, an older version or a schema changed without a bump", async () => { + const KeyedList = defineState("list", { + schema: obj({ items: arr(str()) }), + version: 1, + initial: () => ({ items: [] }), + keyed: true, + }); + const Changed = defineState("list", { + schema: obj({ items: arr(int()) }), + version: 1, + initial: () => ({ items: [] }), + }); + const Newer = defineState("list", { + schema: obj({ items: arr(str()) }), + version: 2, + initial: () => ({ items: [] }), + migrate: (old) => old as { items: string[] }, + }); + const cases = [ + [ListV1, KeyedList, "list: declared keyed, its documents are not"], + [ListV1, Changed, "list: the schema changed without a version bump (version 1)"], + [Newer, ListV1, "list: version 1 is older than its documents' (2)"], + ] as const; + for (const [first, later, message] of cases) { + const { fail, retry, errors } = await restartable(objects, (attempt, self) => { + self.state(attempt === 1 ? first : later); + }); + fail(); + await rejection(retry()); + expect(errors.map(({ kind }) => kind)).toEqual(["object_failed", "activation_failed"]); + expect(failure(errors)).toBe(`app.restartable(1): ${message}`); + } + }); +}); + +describe("persisted singletons", () => { + const Prefs = defineState("prefs", { + schema: obj({ theme: str(), n: int() }), + version: 1, + initial: () => ({ theme: "light", n: 0 }), + persist: true, + checkpointWhen: () => false, + }); + + /** A store host declaring `definition` per attempt; `prefs()` is the latest attempt's handle. */ + async function prefsIn(store: ChordStore, definition: typeof Prefs = Prefs) { + let prefs: SingletonHandle<{ theme: string; n: number }> | undefined; + const restartable = await storeRestartable( + objects, + (_, self) => { + prefs = self.state(definition); + }, + { store }, + ); + return { ...restartable, prefs: () => prefs! }; + } + + /** The kinds of the outbox's writes, per commit. */ + async function outboxKinds(store: ChordStore): Promise { + const commits = await store.outbox!.read(0, 100, context); + return commits.map(({ writes }) => + writes.map((write) => (write.kind === "document.change" ? `change:${write.content.kind}` : write.kind)), + ); + } + + it("load once per object at activation (not again on a retry), and survive the host", async () => { + const store = await memoryStore({ outbox: true }); + const first = await prefsIn(store); + expect(first.prefs().value).toEqual({ theme: "light", n: 0 }); + await first.prefs().change((draft) => { + draft.theme = "dark"; + }, context); + await first.prefs().change((draft) => { + draft.n = 1; + }, context); + await first.host.close(context); + const read = vi.spyOn(store, "read"); + const second = await prefsIn(store); + expect(second.prefs().value).toEqual({ theme: "dark", n: 1 }); + expect(read).toHaveBeenCalledTimes(1); + await second.restart(); + expect(second.prefs().value).toEqual({ theme: "dark", n: 1 }); + expect(read).toHaveBeenCalledTimes(1); + await second.prefs().retire(context); + expect(second.prefs().value).toEqual({ theme: "light", n: 0 }); + expect(await outboxKinds(store)).toEqual([["document.create"], ["change:delta"], ["document.retire"]]); + await second.host.close(context); + await store.close(context); + }); + + it("migrate a stored older version at load; the next change writes the migrated base back", async () => { + const store = await memoryStore({ outbox: true }); + const old = await prefsIn(store); + await old.prefs().change((draft) => { + draft.n = 1; + }, context); + await old.host.close(context); + const V2 = defineState("prefs", { + schema: obj({ theme: str(), n: int() }), + version: 2, + initial: () => ({ theme: "light", n: 0 }), + migrate: (value) => ({ theme: "migrated", n: Number(value.n) }), + persist: true, + checkpointWhen: () => false, + }); + const { host, prefs } = await prefsIn(store, V2 as unknown as typeof Prefs); + expect(prefs().value).toEqual({ theme: "migrated", n: 1 }); + await prefs().change((draft) => { + draft.n = 2; + }, context); + expect((await outboxKinds(store)).at(-1)).toEqual(["change:base"]); + await host.close(context); + await store.close(context); + }); + + it("fail the activation broken for a newer stored version or another fingerprint at the same version (D9)", async () => { + const store = await memoryStore(); + const V2 = defineState("prefs", { + schema: obj({ theme: str(), n: int() }), + version: 2, + initial: () => ({ theme: "light", n: 0 }), + persist: true, + checkpointWhen: () => false, + }); + const Edited = defineState("prefs", { + schema: obj({ theme: str() }), + version: 2, + initial: () => ({ theme: "light" }), + persist: true, + checkpointWhen: () => false, + }); + const writer = await prefsIn(store, V2 as unknown as typeof Prefs); + await writer.prefs().change((draft) => { + draft.n = 1; + }, context); + await writer.host.close(context); + for (const [definition, message] of [ + [Prefs, "prefs: stored at version 2, newer than the code's 1"], + [Edited, "prefs: stored with another schema at version 2 (a schema changed without a bump)"], + ] as const) { + const { host, errors } = await loaded( + (h) => + h.serve(StoreRestartable, (_, self) => { + self.state(definition); + return { touch: async () => null }; + }), + { store }, + ); + await rejection(host.get(StoreRestartable, {}, context)); + expect(failure(errors)).toBe(`app.store: ${message}`); + await host.close(context); + } + await store.close(context); + }); + + it("refuse persisting without a stable identity or a store, and a flipped persist", async () => { + const InMemory = defineState("prefs", { + schema: obj({ theme: str(), n: int() }), + version: 1, + initial: () => ({ theme: "light", n: 0 }), + }); + const keyed = await restartable(objects, (_, self) => { + // The types refuse it; a plugin may still try (P3). + (self as StoreSelf).state(Prefs); + }).catch((error: unknown) => error); + expect(String(keyed)).toContain("persist needs a stable identity"); + const storeless = await storeRestartable(objects, (_, self) => void self.state(Prefs)).catch((error) => error); + expect(String(storeless)).toContain("persist needs a store"); + const store = await memoryStore(); + const { fail, retry, errors } = await storeRestartable( + objects, + (attempt, self) => void self.state(attempt === 1 ? Prefs : InMemory), + { store }, + ); + fail(); + await rejection(retry()); + expect(failure(errors)).toBe("app.store: prefs: declared in memory, its documents are not"); + await store.close(context); + }); +}); + +/** The message of the last reported activation failure's error. */ +function failure(errors: readonly HostError[]): string | undefined { + const last = errors.findLast((error) => error.kind === "activation_failed"); + return last?.kind === "activation_failed" ? last.failure.error.message : undefined; +} + +/** What `read` throws, as ChordError JSON when it is one. */ +function rejectionOf(read: () => unknown): unknown { + try { + read(); + } catch (error) { + return isChordError(error) ? error.toJSON() : error; + } + return undefined; +} diff --git a/packages/chord/test/next/documents/transaction.test.ts b/packages/chord/test/next/documents/transaction.test.ts new file mode 100644 index 000000000..ff75ea375 --- /dev/null +++ b/packages/chord/test/next/documents/transaction.test.ts @@ -0,0 +1,731 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import type { Draft } from "../../../src/delta/index.ts"; +import { int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type KeyedHandle, type SingletonHandle, type Tx } from "../../../src/next/documents/index.ts"; +import type { Staged, Transaction } from "../../../src/next/documents/internal.ts"; +import type { Self } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { query } from "../../../src/next/store/query.ts"; +import type { StagedWrite as StoreStaged } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import type { TableOf } from "../../../src/next/store/table-handle.ts"; +import { + delivered, + framesOf, + instancesOf, + loaded, + memoryStore, + metered, + rejection, + restartable, + storeRestartable, +} from "../support.ts"; + +const instances = instancesOf(ChordObject, "attempt"); + +const A = defineState("a", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const B = defineState("b", { schema: obj({ s: str() }), version: 1, initial: () => ({ s: "" }) }); +const Owner = local("app.owner", { version: 1, params: { id: int() }, members: { sum: str() } }); + +type Handles = { + readonly self: Self; + readonly a: SingletonHandle<{ n: number }>; + readonly b: SingletonHandle<{ s: string }>; +}; + +/** Objects with two documents and a derive over them; returns each object's handles. */ +async function objects(telemetry?: Awaited>["telemetry"]) { + const handles = new Map(); + const { host } = await loaded( + (h) => { + h.serve(Owner, ({ id }, self) => { + const a = self.state(A); + const b = self.state(B); + handles.set(id, { self, a, b }); + return { sum: self.derive([a, b], ([x, y]) => `${x.n}${y.s}`) }; + }); + }, + { telemetry }, + ); + const reference = await host.get(Owner, { id: 1 }, context); + await host.get(Owner, { id: 2 }, context); + return { reference, one: handles.get(1)!, two: handles.get(2)! }; +} + +describe("transactions", () => { + it("run one at a time per object and return what their callback returns", async () => { + const { one } = await objects(); + const order: string[] = []; + const gate = Promise.withResolvers(); + const first = one.self.tx(async (tx) => { + order.push("first"); + await gate.promise; + (await tx.draft(one.a)).n = 1; + order.push("first done"); + return "one"; + }, context); + const second = one.self.tx(async (tx) => { + order.push(`second sees ${one.a.value.n}`); + (await tx.draft(one.a)).n++; + return "two"; + }, context); + await delivered(); + gate.resolve(); + expect(await Promise.all([first, second])).toEqual(["one", "two"]); + expect(order).toEqual(["first", "first done", "second sees 1"]); + expect(one.a.value).toEqual({ n: 2 }); + }); + + it("give the same draft for the same document", async () => { + const { one } = await objects(); + await one.self.tx(async (tx) => { + expect(await tx.draft(one.a)).toBe(await tx.draft(one.a)); + }, context); + }); + + it("settle a warm draft at once: a continuation the callback did not await runs before the commit (rule 3)", async () => { + const { one } = await objects(); + await one.self.tx((tx) => { + void tx.draft(one.a).then((draft) => { + draft.n = 7; + }); + }, context); + expect(one.a.value).toEqual({ n: 7 }); + }); + + it("reject a draft of a handle a transaction cannot use instead of throwing", async () => { + const { one, two } = await objects(); + await one.self.tx(async (tx) => { + const draft = tx.draft(two.a); + expect(draft).toBeInstanceOf(Promise); + await expect(draft).rejects.toThrow("drafts only its own object's documents"); + }, context); + }); + + it("commit nothing when nothing changed", async () => { + const telemetry = await metered(); + const { one, reference } = await objects(telemetry.telemetry); + const frames = framesOf(await reference.sum.watch(context)); + const value = one.a.value; + const computes = telemetry.counts()["chord.derive.computes"]; + await one.self.tx(async (tx) => { + void (await tx.draft(one.a)).n; + }, context); + await delivered(); + expect(one.a.value).toBe(value); + expect(frames).toEqual([]); + expect(telemetry.counts()["chord.derive.computes"]).toBe(computes); + }); + + it("discard everything on a throw and on an aborted context", async () => { + const { one } = await objects(); + const change = async (tx: Tx) => { + (await tx.draft(one.a)).n = 5; + }; + await expect( + one.self.tx(async (tx) => { + await change(tx); + throw new Error("no"); + }, context), + ).rejects.toThrow("no"); + const { context: cancelled, cancel } = withCancel(context); + await expect( + one.self.tx(async (tx) => { + await change(tx); + cancel(new Error("cancelled")); + }, cancelled), + ).rejects.toThrow("cancelled"); + expect(one.a.value).toEqual({ n: 0 }); + }); + + it("revoke their drafts when they settle", async () => { + const { one } = await objects(); + let kept: Draft<{ n: number }> | undefined; + await one.self.tx(async (tx) => { + kept = await tx.draft(one.a); + kept.n = 1; + }, context); + expect(() => kept!.n).toThrow(); + await expect(one.self.tx(async (tx) => tx, context).then((tx) => tx.draft(one.a))).rejects.toThrow( + "app.owner(1): the transaction has settled", + ); + }); + + it("refuse another object's documents", async () => { + const { one, two } = await objects(); + expect(await rejection(one.self.tx((tx) => tx.draft(two.a), context))).toEqual( + new Error("app.owner(1): a transaction drafts only its own object's documents"), + ); + }); + + it("commit the object's documents in one batch: one compute, one frame", async () => { + const telemetry = await metered(); + const { one, reference } = await objects(telemetry.telemetry); + const frames = framesOf(await reference.sum.watch(context)); + const computes = telemetry.counts()["chord.derive.computes"]!; + await one.self.tx(async (tx) => { + (await tx.draft(one.a)).n = 1; + (await tx.draft(one.b)).s = "b"; + }, context); + await delivered(); + expect(frames).toEqual([["1b", [["r", "1b"]]]]); + expect(telemetry.counts()["chord.derive.computes"]).toBe(computes + 1); + }); + + it("commit their other changes when they retire an absent document (R13)", async () => { + const { one } = await objects(); + await one.self.tx(async (tx) => { + await tx.retire(one.b); + (await tx.draft(one.a)).n = 7; + }, context); + expect([one.a.value, one.b.value]).toEqual([{ n: 7 }, { s: "" }]); + }); + + it("draft a retired document anew: watches get a root replacement", async () => { + const { one } = await objects(); + await one.a.change((draft) => { + draft.n = 3; + }, context); + const frames = framesOf(await one.a.watch(context)); + await one.self.tx(async (tx) => { + await tx.retire(one.a); + (await tx.draft(one.a)).n += 1; + }, context); + await delivered(); + expect(frames).toEqual([[{ n: 1 }, [["r", { n: 1 }]]]]); + }); +}); + +describe("the in-memory change's cost", () => { + it("settles an in-memory change within five microtask turns (a guard: an extra await per tx operation shows here)", async () => { + const { one } = await objects(); + for (let run = 0; run < 3; run++) { + let turns = 0; + let running = true; + const change = one.a.change((draft) => { + draft.n++; + }, context); + void change.then(() => { + running = false; + }); + while (running) { + turns++; + await null; + } + await change; + expect(turns).toBeLessThanOrEqual(5); + } + expect(one.a.value.n).toBe(3); + }); +}); + +describe("staged changes", () => { + /** A staged change recording what happens to it, and the document value its adoption sees. */ + function recorder(events: string[], a: SingletonHandle<{ n: number }>): Staged { + return { + adopt: () => void events.push(`adopt sees ${a.value.n}`), + committed: () => void events.push("committed"), + discarded: () => void events.push("discarded"), + }; + } + + it("adopt in the commit step with the documents, then hear of the commit once the line is free", async () => { + const { one } = await objects(); + const events: string[] = []; + const owner = {}; + await one.self.tx(async (tx) => { + (await tx.draft(one.a)).n = 7; + const transaction = tx as Transaction; + const staged = transaction.stage(owner, () => recorder(events, one.a)); + // One per owner and transaction. + expect(transaction.stage(owner, () => recorder([], one.a))).toBe(staged); + events.push("callback done"); + }, context); + expect(events).toEqual(["callback done", "adopt sees 7", "committed"]); + }); + + it("adopt without any document change, and hear of a discard instead when the transaction throws", async () => { + const { one } = await objects(); + const events: string[] = []; + await one.self.tx((tx) => void (tx as Transaction).stage({}, () => recorder(events, one.a)), context); + await rejection( + one.self.tx((tx) => { + (tx as Transaction).stage({}, () => recorder(events, one.a)); + throw new Error("no"); + }, context), + ); + expect(events).toEqual(["adopt sees 0", "committed", "discarded"]); + }); +}); + +describe("the seal fence", () => { + const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + + /** A restartable object with `N`; its handles by attempt. */ + async function sealable() { + const handles: SingletonHandle<{ n: number }>[] = []; + const object = await restartable(instances, (_, self) => { + handles.push(self.state(N)); + }); + return { ...object, handles }; + } + + it("discards the transaction holding the line at the seal: the line is free, its later uses are refused", async () => { + const { attempts, handles, restart } = await sealable(); + const gate = Promise.withResolvers(); + let held: Tx | undefined; + const first = attempts[0]!.tx(async (tx) => { + (await tx.draft(handles[0]!)).n = 1; + held = tx; + await gate.promise; + return "late"; + }, context); + await delivered(); + await restart(); + // The next attempt's transaction runs while the cut one's callback still waits. + await handles[1]!.change((draft) => { + draft.n = 2; + }, context); + gate.resolve(); + expect(await rejection(first)).toMatchObject({ code: "released" }); + expect(await rejection(held!.draft(handles[1]!))).toMatchObject({ + code: "released", + message: "app.restartable(1): the transaction was discarded at the seal", + }); + expect(handles[1]!.value).toEqual({ n: 2 }); + }); + + it("refuses a sealed attempt's transactions waiting for the line and every new one; its handles still read", async () => { + const { attempts, handles, fail } = await sealable(); + const gate = Promise.withResolvers(); + const first = attempts[0]!.tx(() => gate.promise, context); + const queued = handles[0]!.change((draft) => { + draft.n = 1; + }, context); + await delivered(); + fail(); + expect(await rejection(queued)).toMatchObject({ code: "released" }); + expect(await rejection(handles[0]!.change(() => {}, context))).toMatchObject({ code: "released" }); + expect(handles[0]!.value).toEqual({ n: 0 }); + gate.resolve(); + expect(await rejection(first)).toMatchObject({ code: "released" }); + }); +}); + +describe("tables in a transaction", () => { + const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const Edits = defineTable("edits", { version: 1, columns: { id: str(), at: int() }, key: ["id"] }); + type EditsTable = TableOf; + + /** A store object with `N` and `Edits`, on a store with an outbox; its handles by attempt. */ + async function stored(telemetry?: Awaited>["telemetry"]) { + const store: ChordStore = await memoryStore({ outbox: true }); + const handles: { n: SingletonHandle<{ n: number }>; edits: EditsTable }[] = []; + const object = await storeRestartable( + instances, + (_, self) => { + handles.push({ n: self.state(N), edits: self.table(Edits) }); + }, + { store, telemetry }, + ); + return { ...object, store, handles, self: object.attempts[0]!, ...handles[0]! }; + } + + it("commit rows and documents in one step: the documents adopt inside it, one seq", async () => { + const { self, n, edits, store, host } = await stored(); + const watch = await n.watch(context); + const seen: number[] = []; + watch.start(async () => { + seen.push(store.lastSeq); + }); + await delivered(); + const before = store.lastSeq; + await self.tx(async (tx) => { + tx.put(edits, { id: "e1", at: 1 }); + (await tx.draft(n)).n = 1; + }, context); + await delivered(); + expect(store.lastSeq).toBe(before + 1); + expect(seen.at(-1)).toBe(before + 1); + expect(n.value).toEqual({ n: 1 }); + expect(await edits.get({ id: "e1" }, context)).toEqual({ id: "e1", at: 1 }); + watch.stop(); + await host.close(context); + await store.close(context); + }); + + it("in-memory-only commits take no step and no seq (W12)", async () => { + const { telemetry, context: counting, counts } = await metered(); + const { self, n, store, host } = await stored(telemetry); + const commits = counts()["chord.store.commits"] ?? 0; + const seq = store.lastSeq; + await self.tx(async (tx) => { + (await tx.draft(n)).n = 5; + }, counting); + expect(store.lastSeq).toBe(seq); + expect(counts()["chord.store.commits"] ?? 0).toBe(commits); + await host.close(context); + await store.close(context); + }); + + it("read committed rows with tx.page and tx.rows, overlay own writes with tx.get", async () => { + const { self, edits, store, host } = await stored(); + await self.tx((tx) => tx.put(edits, { id: "e1", at: 1 }), context); + await self.tx(async (tx) => { + tx.put(edits, { id: "e2", at: 2 }); + expect((await tx.page(edits, query(Edits), 10)).items.map((row) => row.id)).toEqual(["e1"]); + const all: string[] = []; + for await (const row of tx.rows(edits, query(Edits))) all.push(row.id); + expect(all).toEqual(["e1"]); + expect(await tx.get(edits, { id: "e2" })).toEqual({ id: "e2", at: 2 }); + }, context); + expect(await edits.count(query(Edits), context)).toBe(2); + await host.close(context); + await store.close(context); + }); + + it("write nothing on a throw or an abort, and refuse a callback that settles with a pending read (rule 3)", async () => { + const { self, edits, store, host } = await stored(); + const seq = store.lastSeq; + await expect( + self.tx((tx) => { + tx.put(edits, { id: "e1", at: 1 }); + throw new Error("no"); + }, context), + ).rejects.toThrow("no"); + const { context: cancelled, cancel } = withCancel(context); + const aborted = self.tx(async (tx) => { + tx.put(edits, { id: "e1", at: 1 }); + cancel(); + }, cancelled); + await expect(aborted).rejects.toThrow(); + await expect( + self.tx((tx) => { + void tx.get(edits, { id: "e1" }); + tx.put(edits, { id: "e1", at: 1 }); + }, context), + ).rejects.toThrow("settled before its pending tx operations"); + expect(store.lastSeq).toBe(seq); + expect(await edits.count(query(Edits), context)).toBe(0); + await host.close(context); + await store.close(context); + }); + + it("refuse another object's table", async () => { + const { self, store, host } = await stored(); + const other = await stored(); + await expect(self.tx((tx) => tx.put(other.edits, { id: "e1", at: 1 }), context)).rejects.toThrow( + "writes only its own object's tables", + ); + await host.close(context); + await other.host.close(context); + await store.close(context); + await other.store.close(context); + }); + + it("a commit queued behind another owner's step is refused at the seal; an admitted one is adopted", async () => { + const { attempts, handles, store, fail, retry, host } = await stored(); + // Hold the commit queue with a step that waits. + const gate = Promise.withResolvers(); + const holding = store.commit( + [ + { + order: "write", + apply: async () => { + await gate.promise; + return { kind: "dropIndex", table: "x", name: "y" }; + }, + }, + ], + { admit: () => true, adopt: () => {} }, + context, + ); + const queued = attempts[0]!.tx(async (tx) => { + tx.put(handles[0]!.edits, { id: "e1", at: 1 }); + (await tx.draft(handles[0]!.n)).n = 1; + }, context); + await delivered(); + fail(); + gate.resolve(); + await holding.catch(() => {}); + expect(await rejection(queued)).toMatchObject({ code: "released" }); + await retry(); + expect(handles[1]!.n.value).toEqual({ n: 0 }); + expect(await handles[1]!.edits.get({ id: "e1" }, context)).toBeUndefined(); + // Admitted, then sealed while in the step: durable, adopted into the object's slots, `self.tx` resolves. + const commit = store.commit.bind(store); + store.commit = (writes, fence, ctx) => + commit(writes, { admit: () => sealedAfter(fence.admit()), adopt: fence.adopt }, ctx); + const sealedAfter = (admitted: boolean): boolean => { + if (admitted) fail(); + return admitted; + }; + const admitted = await attempts[1]!.tx(async (tx) => { + tx.put(handles[1]!.edits, { id: "e2", at: 2 }); + (await tx.draft(handles[1]!.n)).n = 2; + return "done"; + }, context); + store.commit = commit; + expect(admitted).toBe("done"); + await retry(); + expect(handles[2]!.n.value).toEqual({ n: 2 }); + expect(await handles[2]!.edits.get({ id: "e2" }, context)).toEqual({ id: "e2", at: 2 }); + await host.close(context); + await store.close(context); + }); +}); + +describe("persisted documents in a transaction", () => { + const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const P = defineState("p", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: 0 }), + persist: true, + checkpointWhen: () => false, + }); + const K = defineState("k", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: 0 }), + keyed: true, + persist: true, + checkpointWhen: () => false, + }); + + /** A store object with in-memory `N`, persisted `P` and keyed persisted `K`, on a store with an outbox. */ + async function stored(telemetry?: Awaited>["telemetry"], on?: ChordStore) { + const store: ChordStore = on ?? (await memoryStore({ outbox: true })); + const handles: { + n: SingletonHandle<{ n: number }>; + p: SingletonHandle<{ n: number }>; + k: KeyedHandle<{ n: number }>; + }[] = []; + const object = await storeRestartable( + instances, + (_, self) => { + handles.push({ n: self.state(N), p: self.state(P), k: self.state(K) }); + }, + { store, telemetry }, + ); + return { ...object, store, self: object.attempts[0]!, ...handles[0]! }; + } + + it("commit persisted and in-memory documents in one step; a delta commit takes three statements", async () => { + const { telemetry, context: counting, counts } = await metered(); + const { self, n, p, store, host } = await stored(telemetry); + await self.tx(async (tx) => { + (await tx.draft(n)).n = 1; + (await tx.draft(p)).n = 1; + }, counting); + expect(store.lastSeq).toBe(1); + const before = { ...counts() }; + await self.tx(async (tx) => { + (await tx.draft(p)).n = 2; + }, counting); + expect(counts()["chord.store.commits"]! - before["chord.store.commits"]!).toBe(1); + // The revision, the record's conditional UPDATE, lastSeq; plus the outbox commit. + expect(counts()["chord.store.statements"]! - before["chord.store.statements"]!).toBe(4); + expect([n.value, p.value]).toEqual([{ n: 1 }, { n: 2 }]); + await host.close(context); + await store.close(context); + }); + + it("assign the created documents' ids by their order in the commit; retire and recreate in one commit", async () => { + const { self, k, store, host } = await stored(); + await self.tx(async (tx) => { + (await tx.draft(k, "b")).n = 1; + (await tx.draft(k, "a")).n = 2; + }, context); + await self.tx(async (tx) => { + await tx.retire(k, "b"); + (await tx.draft(k, "b")).n = 3; + }, context); + const commits = await store.outbox!.read(0, 10, context); + const created = commits.flatMap(({ writes }) => + writes.flatMap((write) => (write.kind === "document.create" ? [[write.record.key, write.record.id]] : [])), + ); + expect(created).toEqual([ + ["b", 1], + ["a", 2], + ["b", 3], + ]); + expect(commits[1]!.writes.map(({ kind }) => kind)).toEqual(["document.retire", "document.create"]); + // What is stored of `b` is the new record's: the next change is a delta on it. + await k.change( + "b", + (draft) => { + draft.n = 4; + }, + context, + ); + expect((await store.outbox!.read(2, 10, context))[0]!.writes).toMatchObject([{ kind: "document.change", id: 3 }]); + await host.close(context); + await store.close(context); + }); + + it("draft a cold keyed document by loading it inside the transaction; refuse a callback with a pending load", async () => { + const first = await stored(); + const { store } = first; + await first.k.change( + "a", + (draft) => { + draft.n = 1; + }, + context, + ); + await first.host.close(context); + const { self, k, host } = await stored(undefined, store); + await self.tx(async (tx) => { + (await tx.draft(k, "a")).n += 1; + }, context); + expect(await k.get("a", context)).toEqual({ n: 2 }); + await expect( + self.tx(async (tx) => { + // Not awaited: the draft is still loading when the callback returns (it rejects later, settled). + tx.draft(k, "b").catch(() => {}); + }, context), + ).rejects.toThrow("the callback settled before its pending tx operations"); + await host.close(context); + await store.close(context); + }); + + it("settle a warm keyed draft at once: a continuation the callback did not await runs before the commit", async () => { + const { self, k, host, store } = await stored(); + await self.tx(async (tx) => { + (await tx.draft(k, "a")).n = 1; + }, context); + await self.tx((tx) => { + void tx.draft(k, "a").then((draft) => { + draft.n = 7; + }); + }, context); + expect(await k.get("a", context)).toEqual({ n: 7 }); + await host.close(context); + await store.close(context); + }); + + it("a commit cut at the seal while it waits in the store's queue rejects released at once, writing nothing", async () => { + const { p, store, host } = await stored(); + // A suspended export holds the commit queue. + const parts = store.export(context); + await parts.next(); + const change = rejection( + p.change((draft) => { + draft.n = 5; + }, context), + ); + await delivered(); + await host.close(context); + expect(await change).toMatchObject({ code: "released" }); + await parts.return!(); + expect(store.lastSeq).toBe(0); + const next = await stored(undefined, store); + expect(next.p.value).toEqual({ n: 0 }); + await next.p.change((draft) => { + draft.n = 6; + }, context); + expect(store.lastSeq).toBe(1); + await next.host.close(context); + await store.close(context); + }); + + it("a host closing while an admitted commit is in the SQL waits for its adoption; the store stays usable", async () => { + const { p, store, host } = await stored(); + // The next commit's step waits inside the backend transaction, after its admission. + const gate = Promise.withResolvers(); + const inStep = Promise.withResolvers(); + const commit = store.commit.bind(store); + store.commit = (writes, fence, ctx) => { + const held: StoreStaged = { + order: "write", + apply: async () => { + inStep.resolve(); + await gate.promise; + return { kind: "dropIndex", table: "x", name: "y" }; + }, + }; + return commit([...writes, held], fence, ctx); + }; + // Not from a method handler: the drain does not wait for it. + const change = p.change((draft) => { + draft.n = 5; + }, context); + await inStep.promise; + let closed = false; + const closing = host.close(context).then(() => { + closed = true; + }); + await delivered(); + expect(closed).toBe(false); + gate.resolve(); + await change; + await closing; + store.commit = commit; + expect(store.lastSeq).toBe(1); + const next = await stored(undefined, store); + expect(next.p.value).toEqual({ n: 5 }); + await next.p.change((draft) => { + draft.n = 6; + }, context); + expect(store.lastSeq).toBe(2); + await next.host.close(context); + await store.close(context); + }); + + it("a handler's commit held in the SQL past closeTimeout still adopts before the object ends; the store stays usable", async () => { + const Writer = local("app.writer", { version: 1, params: {}, members: { write: method() } }); + const store = await memoryStore({ outbox: true }); + const serve = (h: Parameters[0]>[0]) => + h.serve(Writer, (_, self) => { + const p = self.state(P); + return { + write: async (ctx) => { + await p.change((draft) => { + draft.n++; + }, ctx); + return null; + }, + }; + }); + const { host, errors } = await loaded(serve, { store, closeTimeout: 20 }); + const gate = Promise.withResolvers(); + const inStep = Promise.withResolvers(); + const commit = store.commit.bind(store); + store.commit = (writes, fence, ctx) => { + const held: StoreStaged = { + order: "write", + apply: async () => { + inStep.resolve(); + await gate.promise; + return { kind: "dropIndex", table: "x", name: "y" }; + }, + }; + return commit([...writes, held], fence, ctx); + }; + const call = rejection((await host.get(Writer, {}, context)).write(context)); + await inStep.promise; + let closed = false; + const closing = host.close(context).then(() => { + closed = true; + }); + await new Promise((resolve) => setTimeout(resolve, 60)); + // The drain gave up on the handler; the object's teardown still waits for the admitted step. + expect(errors.map(({ kind }) => kind)).toEqual(["drain_timeout"]); + expect(closed).toBe(false); + gate.resolve(); + await closing; + // The stuck call was released at the end; its commit, admitted before, is durable and was adopted. + expect(await call).toMatchObject({ code: "released" }); + store.commit = commit; + expect(store.lastSeq).toBe(1); + const next = await loaded(serve, { store }); + await (await next.host.get(Writer, {}, context)).write(context); + expect(store.lastSeq).toBe(2); + expect(next.errors).toEqual([]); + await next.host.close(context); + await store.close(context); + }); +}); diff --git a/packages/chord/test/next/examples/01-objects.ts b/packages/chord/test/next/examples/01-objects.ts new file mode 100644 index 000000000..57a691e58 --- /dev/null +++ b/packages/chord/test/next/examples/01-objects.ts @@ -0,0 +1,68 @@ +// Objects on one host: a contract, a facet serving it, references, calls, and how long objects live. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/01-objects.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +// Every call takes a context, used for cancellation; BACKGROUND_CONTEXT is never cancelled. +const context = BACKGROUND_CONTEXT; + +// A contract declares what an object type offers: state members (schemas) and methods (arguments, result). +// `params` select the instance: one object per params value. +const Greeter = local("example.greeter", { + version: 1, + params: { language: str() }, + members: { language: str(), greet: method(obj({ name: str() }), str()) }, +}); +// Empty params: one object for the whole host. +const Clock = local("example.clock", { version: 1, params: {}, members: { started: int() } }); + +// A facet is a plugin's setup for one host. It serves types with factories; a factory only declares and returns the +// implementation: a watchable per state member, an async function per method. +const facet = defineFacet((host) => { + host.serve(Clock, () => { + console.log("clock: created at load"); + return { started: constant(Date.now()) }; + }); + host.serve(Greeter, ({ language }) => { + console.log(`greeter ${language}: created by its first get`); + return { + language: constant(language), + greet: async ({ name }) => (language === "de" ? `Hallo ${name}` : `Hello ${name}`), + }; + }); +}); + +// onError receives background errors, such as a watch listener that throws. +const host = createHost({ onError: (error) => console.error("background error:", error) }); +// load runs every setup and activates the types with empty params before it resolves. +await host.load([{ name: "@example/greeter", facet }], context); + +// A reference is live once its get resolves: its state members have values, its methods can be called. +const german = await host.get(Greeter, { language: "de" }, context); +console.log("language:", german.language.value); +console.log("greet:", await german.greet({ name: "Mario" }, context)); + +// Arguments are checked against the contract at the object. +try { + // @ts-expect-error: `name` must be a string + await german.greet({ name: 42 }, context); +} catch (error) { + if (isChordError(error, "invalid")) console.log("refused:", error.reason, error.message); +} + +// The same params share one object; releasing the last reference ends it. +const again = await host.get(Greeter, { language: "de" }, context); +german.release(); +again.release(); +try { + german.language.value; +} catch (error) { + if (isChordError(error, "released")) console.log("after release:", error.message); +} + +// close ends every object. +await host.close(context); diff --git a/packages/chord/test/next/examples/02-documents.ts b/packages/chord/test/next/examples/02-documents.ts new file mode 100644 index 000000000..1387cf0dc --- /dev/null +++ b/packages/chord/test/next/examples/02-documents.ts @@ -0,0 +1,79 @@ +// Documents and transactions: an object's in-memory JSON state, written atomically. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/02-documents.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; + +const context = BACKGROUND_CONTEXT; + +// A singleton document: one per object, `initial()` until written. +const Todos = defineState("todos", { + schema: obj({ items: arr(obj({ id: int(), text: str() })), next: int() }), + version: 1, + initial: () => ({ items: [], next: 1 }), +}); +// A keyed document: many per object, one per string key, absent until written. +const Notes = defineState("notes", { + schema: obj({ body: str() }), + version: 1, + initial: () => ({ body: "" }), + keyed: true, +}); + +const Store = local("example.todos", { + version: 1, + params: {}, + members: { + todos: obj({ items: arr(obj({ id: int(), text: str() })), next: int() }), + add: method(obj({ text: str(), note: opt(str()) }), int()), + note: method(obj({ id: int() }), str()), + clear: method(), + }, +}); + +const facet = defineFacet((host) => { + host.serve(Store, (_, self) => { + const todos = self.state(Todos); + const notes = self.state(Notes); + return { + // A singleton handle is a watchable, so it can be a state member as it is. + todos, + add: async ({ text, note }, ctx) => { + // One transaction over two documents: both change together, or neither does. + const id = await self.tx(async (tx) => { + const draft = await tx.draft(todos); + const id = draft.next++; + draft.items.push({ id, text }); + if (note !== undefined) (await tx.draft(notes, String(id))).body = note; + // Return plain values: drafts are revoked once the transaction settles. + return id; + }, ctx); + return id; + }, + // Read keyed documents outside transactions; an absent one is `undefined`. + note: async ({ id }, ctx) => (await notes.get(String(id), ctx))?.body ?? "(none)", + clear: async (ctx) => { + // Shorthands for one document: retire resets a singleton to `initial()`. + await todos.retire(ctx); + for await (const key of notes.keys(ctx)) await notes.retire(key, ctx); + return null; + }, + }; + }); +}); + +const host = createHost({ onError: (error) => console.error("background error:", error) }); +await host.load([{ name: "@example/todos", facet }], context); +const store = await host.get(Store, {}, context); + +const first = await store.add({ text: "buy milk", note: "the oat one" }, context); +await store.add({ text: "call Anna" }, context); +console.log("todos:", store.todos.value.items); +console.log("note of", first, "->", await store.note({ id: first }, context)); + +await store.clear(context); +console.log("after clear:", store.todos.value, await store.note({ id: first }, context)); + +await host.close(context); diff --git a/packages/chord/test/next/examples/03-derives.ts b/packages/chord/test/next/examples/03-derives.ts new file mode 100644 index 000000000..3d756253c --- /dev/null +++ b/packages/chord/test/next/examples/03-derives.ts @@ -0,0 +1,66 @@ +// Derives: values computed from documents and from other objects' members, recomputed only when needed. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/03-derives.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { mapShared } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; + +const NotesDoc = defineState("notes", { + schema: obj({ list: arr(obj({ text: str() })) }), + version: 1, + initial: () => ({ list: [] }), +}); + +const Notes = local("example.notes", { + version: 1, + params: {}, + members: { titles: arr(str()), count: int(), add: method(obj({ text: str() })) }, +}); +const Badge = local("example.badge", { version: 1, params: { user: str() }, members: { label: str() } }); + +const facet = defineFacet((host) => { + host.serve(Notes, (_, self) => { + const notes = self.state(NotesDoc); + // mapShared calls its function once per note object: unchanged notes keep the same title, so the derive's + // diff after an append is one insert, not a comparison of every title. + const title = mapShared((note: { text: string }) => { + console.log(" title computed for:", note.text); + return note.text.slice(0, 12); + }); + return { + // A derive declares its sources; its compute is synchronous and pure. + titles: self.derive([notes], ([n]) => title(n.list)), + count: self.derive([notes], ([n]) => n.list.length), + add: async ({ text }, ctx) => { + await notes.change((draft) => { + draft.list.push({ text }); + }, ctx); + return null; + }, + }; + }); + // A derive over another object's member, through a reference declared with self.get. + host.serve(Badge, ({ user }, self) => { + const notes = self.get(Notes, {}); + return { label: self.derive([notes.count], ([count]) => `${user}: ${count} notes`) }; + }); +}); + +const host = createHost({ onError: (error) => console.error("background error:", error) }); +await host.load([{ name: "@example/notes", facet }], context); +const notes = await host.get(Notes, {}, context); +const badge = await host.get(Badge, { user: "mario" }, context); + +await notes.add({ text: "buy milk and bread" }, context); +await notes.add({ text: "call Anna" }, context); +// Nobody watches `titles`, so the commits only marked it; reading computes it, each title once. +console.log("titles:", notes.titles.value); +await notes.add({ text: "water plants" }, context); +console.log("titles:", notes.titles.value); +console.log("badge:", badge.label.value); + +await host.close(context); diff --git a/packages/chord/test/next/examples/04-watches.ts b/packages/chord/test/next/examples/04-watches.ts new file mode 100644 index 000000000..f6f81972c --- /dev/null +++ b/packages/chord/test/next/examples/04-watches.ts @@ -0,0 +1,101 @@ +// Watches: follow a state member from its current value through the commits that change it; a busy listener gets +// what changed meanwhile folded into one call; a derive that throws gives a `failed` call, the watch stays open, and +// its next value arrives as a root replacement (it heals on a later change). +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/04-watches.ts +import { BACKGROUND_CONTEXT, withCancel } from "../../../src/context/index.ts"; +import { int, local, method, obj } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; + +const context = BACKGROUND_CONTEXT; + +const Count = defineState("count", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const Counter = local("example.counter", { + version: 1, + params: {}, + members: { count: obj({ n: int() }), double: int(), half: int(), bump: method(), reset: method() }, +}); + +const facet = defineFacet((host) => { + host.serve(Counter, (_, self) => { + const count = self.state(Count); + return { + count, + double: self.derive([count], ([c]) => c.n * 2), + half: self.derive([count], ([c]) => { + if (c.n % 2 === 1) throw new Error(`${c.n} has no half`); + return c.n / 2; + }), + bump: async (ctx) => { + await count.change((draft) => { + draft.n++; + }, ctx); + return null; + }, + reset: async (ctx) => { + await count.retire(ctx); + return null; + }, + }; + }); +}); + +// A derive whose compute throws is reported once per failure episode (`derive_failed`). +const host = createHost({ + onError: (error) => { + if (error.kind === "derive_failed") console.log(`reported: ${error.object}.${error.member}: ${error.error}`); + else console.error("background error:", error); + }, +}); +await host.load([{ name: "@example/counter", facet }], context); +const counter = await host.get(Counter, {}, context); + +// A watch holds the value at acquisition; its listener then gets the changes, asynchronously and in order, never +// overlapping itself. A listener that keeps up gets one call per commit: the new value and the ops that turn the +// previous value into it. +const watch = await counter.count.watch(context); +console.log("acquired:", watch.value); +watch.start(async (call) => { + if (call.kind === "value") console.log("count frame:", call.value, JSON.stringify(call.ops)); +}); + +// A derive's frame carries the diff of its old and new value. +const { context: cancellable, cancel } = withCancel(context); +const doubled = await counter.double.watch(cancellable); +doubled.start(async (call) => { + if (call.kind === "value") console.log("double frame:", call.value, JSON.stringify(call.ops)); +}); + +// A failing derive: one `failed` call per failure, the watch stays open; the next value replaces the root. +const half = await counter.half.watch(context); +half.start(async (call) => { + if (call.kind === "failed") console.log("half failed:", (call.error as Error).message); + else console.log("half frame:", call.value, JSON.stringify(call.ops)); +}); + +// A slow listener: commits made while it is busy fold into its next call, with the newest value and the ops since +// its previous call (an `s` on one path replaces the one before it). +const slow = await counter.count.watch(context); +slow.start(async (call) => { + if (call.kind !== "value") return; + console.log("slow frame:", call.value, JSON.stringify(call.ops)); + await new Promise((resolve) => setTimeout(resolve, 20)); +}); + +await counter.bump(context); +await new Promise((resolve) => setTimeout(resolve, 0)); +await counter.bump(context); +await counter.bump(context); +await new Promise((resolve) => setTimeout(resolve, 50)); +// Retiring the singleton resets it to initial(): watches get a root replacement. +await counter.reset(context); +await new Promise((resolve) => setTimeout(resolve, 50)); + +// Cancelling the watch's context ends it; stop() ends it too. Releasing the reference ends the watches taken through it. +cancel(); +console.log("double watch:", await doubled.closed); +counter.release(); +console.log("count watch:", await watch.closed); + +await host.close(context); diff --git a/packages/chord/test/next/examples/05-errors.ts b/packages/chord/test/next/examples/05-errors.ts new file mode 100644 index 000000000..7ea09797d --- /dev/null +++ b/packages/chord/test/next/examples/05-errors.ts @@ -0,0 +1,93 @@ +// Errors: what Chord refuses and how a caller tells the cases apart (`ChordError` with `code` and `reason`). +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/05-errors.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet, type HostError } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; + +/** Prints how `work` failed: a ChordError's code, reason and message, or a plain error's message. */ +async function attempt(label: string, work: () => unknown): Promise { + try { + await work(); + console.log(`${label}: ok`); + } catch (error) { + if (isChordError(error)) + console.log(`${label}: ${error.code}${error.reason ? `/${error.reason}` : ""} - ${error.message}`); + else console.log(`${label}: ${error instanceof Error ? error.message : String(error)}`); + } +} + +const Echo = local("example.echo", { + version: 1, + params: { id: int() }, + members: { id: int(), echo: method(obj({ text: str() }), str()) }, +}); +// Two types that get each other while they activate: a cycle. +const Ping = local("example.ping", { version: 1, params: {}, members: { n: int() } }); +const Pong = local("example.pong", { version: 1, params: {}, members: { n: int() } }); +const Broken = local("example.broken", { version: 1, params: { id: int() }, members: { n: int() } }); +const Sloppy = local("example.sloppy", { version: 1, params: { id: int() }, members: { n: int() } }); + +const facet = defineFacet((host) => { + host.serve(Echo, ({ id }) => ({ id: constant(id), echo: async ({ text }) => text })); + host.serve(Broken, () => { + throw new Error("broken: missing configuration"); + }); + // Plugins are bundled without type checks; Chord checks the implementation's members when the object activates. + host.serve(Sloppy, () => ({ n: constant(1), extra: constant(2) }) as never); +}); +// Ping and Pong are served by a second plugin: types with empty params activate at load. +const cycle = defineFacet((host) => { + host.serve(Ping, (_, self) => ({ n: self.get(Pong, {}).n })); + host.serve(Pong, (_, self) => ({ n: self.get(Ping, {}).n })); +}); + +// Failed activations are reported to onError; 07-failures.ts shows the other reports. +const onError = (error: HostError): void => { + if (error.kind === "activation_failed") console.log(`reported: ${error.object} failed ${error.failure.kind}`); + else console.log("reported:", error.kind); +}; + +// A dependency cycle fails every object on it broken, with invalid/cycle naming the chain. Load still resolves: the +// start barrier waits for each object's first outcome, live or failed. +const host = createHost({ onError }); +await host.load( + [ + { name: "@example/errors", facet }, + { name: "@example/cycle", facet: cycle }, + ], + context, +); +await attempt("get in a cycle", () => host.get(Ping, {}, context)); +const echo = await host.get(Echo, { id: 1 }, context); + +// Arguments are checked against the contract at the object: wrong types and unknown fields are invalid/args. +// @ts-expect-error: text must be a string +await attempt("wrong argument", () => echo.echo({ text: 1 }, context)); +// @ts-expect-error: an unknown field +await attempt("unknown field", () => echo.echo({ text: "hi", extra: true }, context)); + +// Params select the object and are checked on every get: invalid/params. +// @ts-expect-error: id must be an integer +await attempt("bad params", () => host.get(Echo, { id: "one" }, context)); + +// An implementation with a member its contract does not declare fails activation. +await attempt("unknown member", () => host.get(Sloppy, { id: 1 }, context)); + +// A get needs the provider's version and fingerprint; another declaration of the type does not fit. +const EchoV2 = local("example.echo", { version: 2, params: { id: int() }, members: { id: int() } }); +await attempt("contract mismatch", () => host.get(EchoV2, { id: 1 }, context)); + +// A factory that throws fails broken: the get rejects at once with its error, and no object is left behind. +await attempt("throwing factory", () => host.get(Broken, { id: 1 }, context)); + +// After release, every use of a reference throws released. +echo.release(); +await attempt("released reference", () => echo.echo({ text: "hi" }, context)); + +await host.close(context); +await attempt("get after close", () => host.get(Echo, { id: 1 }, context)); diff --git a/packages/chord/test/next/examples/06-natives.ts b/packages/chord/test/next/examples/06-natives.ts new file mode 100644 index 000000000..8e74f06d6 --- /dev/null +++ b/packages/chord/test/next/examples/06-natives.ts @@ -0,0 +1,96 @@ +// Natives: values an object opens during activation and closes when it ends (a file watcher, a connection, a timer). +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/06-natives.ts +import { setTimeout as sleep } from "node:timers/promises"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, local, method, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; + +const Clock = local("example.clock", { + version: 1, + params: { zone: str() }, + members: { zone: str(), now: method(undefined, str()) }, +}); +const Leaky = local("example.leaky", { version: 1, params: {}, members: { n: int() } }); + +const facet = defineFacet((host) => { + host.serve(Clock, ({ zone }, self) => { + // `self.open(open, close)` declares a native; it opens after the references are live, in declaration order. + const config = self.open( + async () => { + console.log(`${zone}: open config`); + return { prefix: `[${zone}]` }; + }, + (_config, reason) => console.log(`${zone}: close config (${reason.kind})`), + ); + // A later opener reads an earlier native's value. Its context is aborted if the object ends while it opens. + const ticker = self.open( + async () => { + console.log(`${zone}: open ticker, config ${config.value.prefix}`); + const ticker = { ticks: 0, timer: setInterval(() => ticker.ticks++, 10) }; + return ticker; + }, + (ticker, reason) => { + clearInterval(ticker.timer); + console.log(`${zone}: close ticker (${reason.kind})`); + }, + ); + // The factory only declares: a value is there once its open resolved. + try { + config.value; + } catch (error) { + if (isChordError(error, "not_ready")) console.log(`${zone}: in the factory, ${error.message}`); + } + return { + zone: constant(zone), + now: async () => `${config.value.prefix} ${ticker.value.ticks > 0 ? "ticking" : "starting"}`, + }; + }); + // Closes that throw or hang: they never stop the other closes. + host.serve(Leaky, (_, self) => { + self.open( + async () => "socket", + () => { + throw new Error("socket close failed"); + }, + ); + self.open( + async () => "worker", + () => new Promise(() => {}), + { closeTimeout: 50 }, + ); + return { n: constant(1) }; + }); +}); + +// onError gets what fails in the background: here a close past its closeTimeout. +const host = createHost({ + onError: (error) => console.log("reported:", error.kind, "object" in error ? error.object : ""), +}); +await host.load([{ name: "@example/natives", facet }], context); + +const vienna = await host.get(Clock, { zone: "vienna" }, context); +console.log("now:", await vienna.now(context)); +await sleep(30); +console.log("now:", await vienna.now(context)); + +// The last reference gone, the object ends: natives close in reverse order, with reason `end`. +vienna.release(); +await sleep(0); + +// host.close ends every object at once and closes their natives with reason `close`; it rejects with every close +// that threw. +await host.get(Clock, { zone: "tokyo" }, context); +try { + await host.close(context); +} catch (error) { + if (error instanceof AggregateError) + console.log( + "host.close:", + error.errors.map((e: Error) => e.message), + ); +} diff --git a/packages/chord/test/next/examples/07-failures.ts b/packages/chord/test/next/examples/07-failures.ts new file mode 100644 index 000000000..2d8bb9697 --- /dev/null +++ b/packages/chord/test/next/examples/07-failures.ts @@ -0,0 +1,96 @@ +// Failures: how an activation fails (transient or broken), how Chord retries it, and what getters see. +// Run from packages/chord (takes about 3 s: the retries wait 1 s, then 2 s): +// node --conditions=source --experimental-strip-types test/next/examples/07-failures.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, local, str } from "../../../src/next/contracts/index.ts"; +import { broken, isChordError } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet, type HostError } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; +const start = Date.now(); +const at = (): string => `${((Date.now() - start) / 1000).toFixed(1)} s`; + +// A connection to a remote service, which is down for the first two attempts. +const Remote = local("example.remote", { version: 1, params: {}, members: { status: str() } }); +// An inbox per user, which needs the connection. +const Inbox = local("example.inbox", { version: 1, params: { user: str() }, members: { status: str() } }); +// A mirror whose configuration is missing: retrying cannot help. +const Mirror = local("example.mirror", { version: 1, params: { id: int() }, members: { status: str() } }); +// A service whose open never finishes. +const Stuck = local("example.stuck", { version: 1, params: { id: int() }, members: { status: str() } }); + +let attempts = 0; +const facet = defineFacet((host) => { + host.serve(Remote, (_, self) => { + const attempt = ++attempts; + self.open( + async () => { + console.log(`${at()}: remote, attempt ${attempt}`); + // A throwing open is transient: Chord retries it after 1 s, then 2 s, 4 s, ... up to 60 s. + if (attempt <= 2) throw new Error("connection refused"); + return { connected: true }; + }, + () => {}, + ); + return { status: constant("connected") }; + }); + host.serve(Inbox, ({ user }, self) => { + // A dependency that fails fails this object transient too; it is retried the moment the dependency is live. + const remote = self.get(Remote, {}); + console.log(`${at()}: inbox ${user}, attempt`); + return { status: self.derive([remote.status], ([status]) => `${user}: ${status}`) }; + }); + host.serve(Mirror, (_, self) => { + self.open( + async () => { + // broken(error) says retrying is pointless: no retry, and getters are rejected at once. + throw broken(new Error("no mirror configured")); + }, + () => {}, + ); + return { status: constant("mirroring") }; + }); + host.serve(Stuck, (_, self) => { + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { status: constant("never") }; + }); +}); + +// Every failed attempt is reported, with its object and cause. +const onError = (error: HostError): void => { + if (error.kind === "activation_failed") { + console.log(`${at()}: reported ${error.object}, ${error.failure.kind}: ${error.failure.error.message}`); + } +}; + +/** Prints how a get ended. */ +async function attempt(label: string, work: () => Promise<{ status: { value: string } }>): Promise { + try { + const reference = await work(); + console.log(`${at()}: ${label}: ${reference.status.value}`); + } catch (error) { + if (isChordError(error)) console.log(`${at()}: ${label}: ${error.code} - ${error.message}`); + } +} + +// Every attempt must be live within readyTimeout (here 500 ms); getters wait up to it too, unless they say otherwise. +const host = createHost({ onError, readyTimeout: 500 }); +// The start barrier: load resolves once the remote (empty params: it lives as long as the host) had its first outcome. +await host.load([{ name: "@example/failures", facet }], context); +console.log(`${at()}: loaded`); + +// A broken failure rejects the get at once. +await attempt("mirror", () => host.get(Mirror, { id: 1 }, context)); + +// An attempt not live within readyTimeout is abandoned and fails transient; this get waits longer (700 ms) and then +// rejects with the last failure's error. +await attempt("stuck", () => host.get(Stuck, { id: 1 }, context, { readyTimeout: 700 })); + +// Transient failures keep a get waiting through the retries, up to its deadline: the inbox goes live with the remote. +await attempt("inbox", () => host.get(Inbox, { user: "mario" }, context, { readyTimeout: 10_000 })); + +await host.close(context); diff --git a/packages/chord/test/next/examples/08-remote.ts b/packages/chord/test/next/examples/08-remote.ts new file mode 100644 index 000000000..c88d81e15 --- /dev/null +++ b/packages/chord/test/next/examples/08-remote.ts @@ -0,0 +1,111 @@ +// Remote objects over one link: a server and a client host, a remote contract with two variants, a factory choosing +// the variant per principal over a shared local store, and the client's replicas following the server's commits. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/08-remote.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { mapShared } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; + +// The app's principals: who calls. Links stamp them; a caller never sets its own. +type P = { kind: "owner" } | { kind: "guest"; session: string } | { kind: "server" }; + +const Note = obj({ text: str(), author: str() }); +const PublicNote = obj({ text: str() }); + +// A remote contract: one object per (type, params, principal key), in the variant its factory chooses. +const Notes = remote("example.notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { state: { notes: arr(Note), count: int() }, methods: { add: method(obj({ text: str() })) } }, + reader: { state: { notes: arr(PublicNote), count: int() } }, + }, +}); + +// The shared store: the notes of a session once, behind per-principal views (design §6). Local types never cross a link. +const NoteStore = local("example.note-store", { + version: 1, + params: { session: str() }, + members: { notes: arr(Note), add: method(Note) }, +}); + +const NotesDoc = defineState("notes", { schema: obj({ list: arr(Note) }), version: 1, initial: () => ({ list: [] }) }); + +// A redaction computed once per store element, so a reader's derive keeps unchanged notes identical. +const redact = mapShared((note: { text: string; author: string }) => ({ text: note.text })); + +const facet = defineFacet

((host) => { + host.serve(NoteStore, (_, self) => { + const doc = self.state(NotesDoc); + return { + notes: self.derive([doc], ([d]) => d.list), + add: async (note, ctx) => { + await doc.change((draft) => void draft.list.push(note), ctx); + return null; + }, + }; + }); + // A remote factory gets the principal its object is for and returns one variant with exactly its members. + host.serve(Notes, ({ session }, principal, self) => { + const store = self.get(NoteStore, { session }); + const count = self.derive([store.notes], ([n]) => n.length); + if (principal.kind === "guest") { + return { variant: "reader", members: { notes: self.derive([store.notes], ([n]) => redact(n)), count } }; + } + return { + variant: "editor", + members: { + notes: store.notes, + count, + add: async ({ text }, ctx) => store.add({ text, author: principal.kind }, ctx), + }, + }; + }); +}); + +// 1. The server: it acts as `server` for its own requests. +const server = createHost

({ principal: { kind: "server" }, onError: (error) => console.error("server:", error) }); +await server.load([{ name: "@example/notes", facet }], context); + +// 2. A client (a tab): no principal of its own. The entry point authenticated it as a guest of S1 and stamps that on +// the link; the client connects before it loads (its plugins may reference server types). +const [up, down] = memoryLinkPair(); +server.accept(up, { principal: { kind: "guest", session: "S1" } }, context); +const client = createHost

({ onError: (error) => console.error("client:", error) }); +client.connect(down, { principal: { kind: "server" } }, context); +await client.load([], context); + +// 3. The server writes as an editor; the guest reads the redacted view over the link. +const own = await server.get(Notes, { session: "S1" }, context); +if (own.variant === "editor") await own.add({ text: "hello" }, context); +const notes = await client.get(Notes, { session: "S1" }, context); +console.log("guest variant:", notes.variant); +console.log("guest notes:", JSON.stringify(notes.notes.value), "count:", notes.count.value); + +// 4. A commit on the server reaches the client as one state message: both members change in the same batch. +const watch = await notes.notes.watch(context); +watch.start(async (call) => { + if (call.kind === "value") console.log("frame:", JSON.stringify(call.value), "ops:", JSON.stringify(call.ops)); +}); +if (own.variant === "editor") await own.add({ text: "world" }, context); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log("count after the commit:", notes.count.value); + +// 5. TypeScript narrows on `variant`: a reader has no `add` (a forged call would fail `invalid/member` at the server). +if (notes.variant === "reader") { + // @ts-expect-error: `add` is not a member of the reader variant + void notes.add; +} + +// 6. Closing the server closes its links first: the client's reference goes loading and keeps its last values until +// a new link holds it again (10-drops.ts). Closing the client releases it and ends its watches. +await server.close(context); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log("after the server closed:", notes.state.value, "count:", notes.count.value); +await client.close(context); +console.log("after the client closed:", notes.state.value, await watch.closed); diff --git a/packages/chord/test/next/examples/09-calls.ts b/packages/chord/test/next/examples/09-calls.ts new file mode 100644 index 000000000..01dfbbe14 --- /dev/null +++ b/packages/chord/test/next/examples/09-calls.ts @@ -0,0 +1,107 @@ +// Calls over a link: results, bytes as raw segments, ChordErrors and plugin errors, the caller's principal and +// incarnation in `ctx`, cancellation, and release rejecting the calls made through a reference. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/09-calls.ts +import { BACKGROUND_CONTEXT, withCancel } from "../../../src/context/index.ts"; +import { bytes, int, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { isChordError, PluginError } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "owner" } | { kind: "server" }; + +const Files = remote("example.files", { + version: 1, + params: {}, + variants: { + owner: { + methods: { + whoami: method(undefined, str()), + // A `bytes()` leaf: the argument and result travel as raw segments, never base64. + upload: method(obj({ name: str(), data: bytes() }), int()), + download: method(obj({ name: str() }), bytes()), + quota: method(), + slow: method(), + }, + }, + }, +}); + +const stored = new Map(); + +const facet = defineFacet

((host) => { + host.serve(Files, () => ({ + variant: "owner", + members: { + // ctx.principal is the caller as its link stamped it; ctx.incarnation the calling host's run (R20). + whoami: async (ctx) => `${ctx.principal?.kind} from ${ctx.incarnation.slice(0, 8)}…`, + upload: async ({ name, data }) => { + stored.set(name, data); + return data.byteLength; + }, + download: async ({ name }) => stored.get(name) ?? new Uint8Array(), + // A PluginError crosses with its code, message and details; any other throw as code `internal`. + quota: async () => { + throw new PluginError("quota", "the store is full", { limit: 1 }); + }, + // A handler's context is aborted when its caller cancels or releases the reference. + slow: (ctx) => + new Promise((resolve) => + ctx.abortSignal?.addEventListener("abort", () => { + console.log("server: slow aborted:", (ctx.abortSignal?.reason as Error).message); + resolve(null); + }), + ), + }, + })); +}); + +const server = createHost

({ principal: { kind: "server" }, onError: (error) => console.error("server:", error) }); +await server.load([{ name: "@example/files", facet }], context); +const [up, down] = memoryLinkPair(); +server.accept(up, { principal: { kind: "owner" } }, context); +const client = createHost

({ onError: (error) => console.error("client:", error) }); +client.connect(down, { principal: { kind: "server" } }, context); +await client.load([], context); + +const files = await client.get(Files, {}, context); + +// 1. A call resolves with its result; a method without arguments takes only the context. +console.log("whoami:", await files.whoami(context)); + +// 2. Bytes: 1 MiB goes out as one raw segment next to the JSON part. +const data = new Uint8Array(1024 * 1024).fill(7); +console.log("uploaded bytes:", await files.upload({ name: "chart.png", data }, context)); +const back = await files.download({ name: "chart.png" }, context); +console.log("downloaded:", back.byteLength, "bytes, a", back.constructor.name); + +// 3. Errors: arguments are checked at the provider (a ChordError); a plugin's own error keeps its code. +try { + // @ts-expect-error: `name` must be a string + await files.upload({ name: 1, data }, context); +} catch (error) { + if (isChordError(error, "invalid")) console.log("refused:", error.reason, error.message); +} +try { + await files.quota(context); +} catch (error) { + if (error instanceof PluginError) console.log("plugin error:", error.code, error.message, error.details); +} + +// 4. Cancellation: the caller is rejected at once, the provider aborts the handler. +const { context: cancellable, cancel } = withCancel(context); +const slow = files.slow(cancellable); +setTimeout(() => cancel(new Error("the user gave up")), 5); +await slow.catch((error: unknown) => console.log("client: slow rejected:", (error as Error).message)); + +// 5. Release rejects the calls made through the reference; the provider aborts their handlers. +const pending = files.slow(context); +await new Promise((resolve) => setTimeout(resolve, 5)); +files.release(); +await pending.catch((error: unknown) => console.log("client: pending call:", (error as Error).message)); +await new Promise((resolve) => setTimeout(resolve, 5)); + +await client.close(context); +await server.close(context); diff --git a/packages/chord/test/next/examples/10-drops.ts b/packages/chord/test/next/examples/10-drops.ts new file mode 100644 index 000000000..701961c45 --- /dev/null +++ b/packages/chord/test/next/examples/10-drops.ts @@ -0,0 +1,107 @@ +// Surviving drops: a reference held before any link exists, `ref.state` as it follows the link, calls parked while +// loading and sent in order once live, a call in flight when the link drops, and the re-hold after `connect` again. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/10-drops.ts +import { BACKGROUND_CONTEXT, type Context } from "../../../src/context/index.ts"; +import { arr, int, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "owner" } | { kind: "server" }; + +const Notes = remote("example.notes", { + version: 1, + params: { session: str() }, + variants: { + editor: { + state: { notes: arr(str()) }, + methods: { add: method(obj({ text: str() }), int()), slow: method() }, + }, + }, +}); + +const NotesDoc = defineState("notes", { schema: obj({ list: arr(str()) }), version: 1, initial: () => ({ list: [] }) }); + +const facet = defineFacet

((host) => { + // Lingers for a minute after its last reference: a client that reconnects within it finds its notes again. + host.serve( + Notes, + (_, _principal, self) => { + const doc = self.state(NotesDoc); + return { + variant: "editor", + members: { + notes: self.derive([doc], ([d]) => d.list), + add: async ({ text }, ctx) => { + await doc.change((draft) => void draft.list.push(text), ctx); + return doc.value.list.length; + }, + slow: (ctx) => new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", () => resolve(null))), + }, + }; + }, + { linger: 60_000 }, + ); +}); + +const server = createHost

({ principal: { kind: "server" }, onError: (error) => console.error("server:", error) }); +await server.load([{ name: "@example/notes", facet }], context); +const client = createHost

({ onError: (error) => console.error("client:", error) }); +await client.load([], context); + +/** A new link between the hosts: the server stamps the client as its owner. Returns the server's end. */ +function connect(server: Host

, client: Host

, context: Context): Link { + const [up, down] = memoryLinkPair(); + server.accept(up, { principal: { kind: "owner" } }, context); + client.connect(down, { principal: { kind: "server" } }, context); + return up; +} + +const settle = () => new Promise((resolve) => setTimeout(resolve, 10)); + +// 1. A hold before any link: a reference at once, `loading`, its variant unknown. Calls on it park. +const notes = client.hold(Notes, { session: "S1" }, context); +console.log("held without a link:", notes.state.value, "variant:", notes.variant); +const watch = await notes.state.watch(context); +watch.start(async (call) => { + if (call.kind === "value") console.log(" ref.state →", call.value.kind); +}); +const parked = notes.add({ text: "written while loading" }, context); + +// 2. Connect: the reference is held over the link, goes live, and the parked call is sent. +let link = connect(server, client, context); +console.log("parked call's result once live:", await parked); +console.log("notes:", notes.notes.value, "variant:", notes.variant); + +// 3. The link drops (the server's end closes it). A call in flight fails `unavailable` and is never resent; the +// reference goes loading and keeps its last values. +const inFlight = notes.slow(context).catch((error: Error) => `rejected: ${error.message}`); +await settle(); +link.close(); +console.log("in-flight call:", await inFlight); +await settle(); +console.log("after the drop:", notes.state.value, "last notes:", notes.notes.value); + +// 4. Calls made while the link is down park; they wait at most the reference's readyTimeout (60 s here). +const first = notes.add({ text: "first while down" }, context); +const second = notes.add({ text: "second while down" }, context); + +// 5. The app connects again (its link adapter would): every reference is held again with fresh ids, its members get +// the new snapshot as one frame `[["r", value]]`, then the parked calls are sent in the order they were made. +const frames = await notes.notes.watch(context); +frames.start(async (call) => { + if (call.kind === "value") console.log(" notes frame:", JSON.stringify(call.ops)); +}); +link = connect(server, client, context); +console.log("parked results, in order:", await first, await second); +await settle(); +console.log("notes after the re-hold:", notes.notes.value); + +// 6. Release: `ref.state` is `ended` (its error says the holder released it) and its watch ends. +notes.release(); +console.log("watch ended:", await watch.closed, "state:", notes.state.value); +await client.close(context); +await server.close(context); diff --git a/packages/chord/test/next/examples/11-linger-and-retry.ts b/packages/chord/test/next/examples/11-linger-and-retry.ts new file mode 100644 index 000000000..391d55f64 --- /dev/null +++ b/packages/chord/test/next/examples/11-linger-and-retry.ts @@ -0,0 +1,109 @@ +// `linger` and `ref.retry()`: an object that stays warm after its last reference, failed references with `retryAt`, +// a Retry button that skips the backoff, and a broken object renewed by a retry. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/11-linger-and-retry.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { broken } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type RefState } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; + +const Draft = local("example.draft", { + version: 1, + params: { conversation: str() }, + members: { text: str(), type: method(obj({ text: str() })) }, +}); + +const Mirror = local("example.mirror", { version: 1, params: { id: int() }, members: { status: str() } }); + +const DraftDoc = defineState("draft", { schema: obj({ text: str() }), version: 1, initial: () => ({ text: "" }) }); + +// What the mirror's open finds: nothing configured, then a server that is down, then one that answers. +const remote = { configured: false, up: false }; + +const facet = defineFacet((host) => { + // 1. A draft lingers 200 ms after its last reference: a tab that closes and reopens quickly finds its text. + host.serve( + Draft, + ({ conversation }, self) => { + console.log(` (draft ${conversation} activated)`); + const doc = self.state(DraftDoc); + return { + text: self.derive([doc], ([d]) => d.text), + type: async ({ text }, ctx) => { + await doc.change((draft) => { + draft.text = text; + }, ctx); + return null; + }, + }; + }, + { linger: 200 }, + ); + host.serve(Mirror, (_, self) => { + self.open( + async () => { + // `broken`: retrying cannot help until someone changes something (no automatic retry). + if (!remote.configured) throw broken(new Error("no mirror configured")); + // A plain throw is transient: retried with backoff (1 s, doubling). + if (!remote.up) throw new Error("mirror unreachable"); + return null; + }, + () => {}, + ); + return { status: constant("mirroring") }; + }); +}); + +const host = createHost({ onError: (error) => console.log(` onError: ${error.kind}`) }); +await host.load([{ name: "@example/app", facet }], context); +const wait = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +/** A state for the log: a failure with its kind, when it retries by itself, and its error. */ +function describe(state: RefState): string { + if (state.kind !== "failed") return state.kind; + const retry = state.retryAt === undefined ? "no automatic retry" : `retries in ${state.retryAt - Date.now()} ms`; + return `failed, ${state.cause.kind}, ${retry}: ${state.cause.error.message}`; +} + +// 2. Linger: release, get again within 200 ms, the same object and its in-memory document. +let draft = await host.get(Draft, { conversation: "c1" }, context); +await draft.type({ text: "hello" }, context); +draft.release(); +await wait(100); +draft = await host.get(Draft, { conversation: "c1" }, context); +console.log("within linger:", JSON.stringify(draft.text.value)); +draft.release(); +await wait(300); +draft = await host.get(Draft, { conversation: "c1" }, context); +console.log("after linger:", JSON.stringify(draft.text.value)); +draft.release(); + +// 3. A held reference follows its object: broken first (no retryAt), so nothing happens by itself. +const mirror = host.hold(Mirror, { id: 1 }, context); +const watch = await mirror.state.watch(context); +watch.start(async (call) => { + if (call.kind === "value") console.log(` ref.state → ${describe(call.value)}`); +}); +await wait(10); + +// 4. The user fixes the configuration and presses Retry: the broken object is renewed and tries again. The server is +// still down: a transient failure, retried automatically at retryAt. +remote.configured = true; +mirror.retry(); +await wait(10); + +// 5. Retry again (a Retry button): the backoff window is skipped; this time it goes live. +remote.up = true; +mirror.retry(); +await wait(10); +console.log("status:", mirror.status.value); + +// 6. A retry while live does nothing. +mirror.retry(); +await wait(10); +mirror.release(); +await host.close(context); diff --git a/packages/chord/test/next/examples/12-routes.ts b/packages/chord/test/next/examples/12-routes.ts new file mode 100644 index 000000000..7074bd979 --- /dev/null +++ b/packages/chord/test/next/examples/12-routes.ts @@ -0,0 +1,160 @@ +// Routes: host kinds and addresses, a contract's placement, a server routing a session's objects to an in-process +// worker (one `resolve` per address, `authorize` per principal), a guest's `ctx.get` on the server reaching the worker +// as the guest, and a worker that drops and is resolved again. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/12-routes.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = + | { kind: "owner" } + | { kind: "guest"; session: string } + | { kind: "server" } + | { kind: "worker"; session: string }; + +// Kinds of addressable hosts; `Session({ id: "S1" })` is an address, its key `session|S1`. +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); + +// Notes live on their session's worker: the caller computes the address and sends it with its hold. +const Notes = remote("example.notes", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { + editor: { + state: { notes: arr(str()) }, + methods: { add: method(obj({ text: str() }), int()), whoami: method(undefined, str()) }, + }, + reader: { state: { notes: arr(str()) }, methods: { whoami: method(undefined, str()) } }, + }, +}); + +// The lobby lives on the server (no placement): its handler gets notes for whoever calls it. +const Lobby = remote("example.lobby", { + version: 1, + params: {}, + variants: { visitor: { methods: { peek: method(obj({ session: str() }), str()) } } }, +}); + +// The worker's own store of its session's notes, shared by every principal's `Notes` object. +const NoteStore = local("example.note-store", { + version: 1, + params: {}, + members: { list: arr(str()), add: method(obj({ text: str() }), int()) }, +}); + +const NotesDoc = defineState("notes", { schema: obj({ list: arr(str()) }), version: 1, initial: () => ({ list: [] }) }); + +/** The worker's facet: an owner edits, anyone else reads; `whoami` shows whom a call acts for. */ +const workerFacet = defineFacet

((host) => { + host.serve(NoteStore, (_, self) => { + const doc = self.state(NotesDoc); + return { + list: self.derive([doc], ([d]) => d.list), + add: async ({ text }, ctx) => { + await doc.change((draft) => void draft.list.push(text), ctx); + return doc.value.list.length; + }, + }; + }); + host.serve(Notes, (_, principal, self) => { + const store = self.get(NoteStore, {}); + const whoami = async (ctx: { readonly principal: P | undefined }) => JSON.stringify(ctx.principal); + if (principal.kind !== "owner") return { variant: "reader", members: { notes: store.list, whoami } }; + return { variant: "editor", members: { notes: store.list, add: (args, ctx) => store.add(args, ctx), whoami } }; + }); +}); + +const serverFacet = defineFacet

((host) => { + host.serve(Lobby, () => ({ + variant: "visitor", + members: { + // `ctx.get` acts as the caller: over the route, the worker sees the guest, not the server. + peek: async ({ session }, ctx) => { + const notes = await ctx.get(Notes, { session }); + return `${notes.variant} variant, ${await notes.whoami(ctx)}`; + }, + }, + })); +}); + +// 1. The server: an address, and a route for sessions. `resolve` starts an in-process worker over a memory link and +// returns the link at once (the worker connects first, then loads, W6). `authorize` admits owners everywhere and a +// guest only into its own session. +const server = createHost

({ address: Server(), principal: { kind: "server" }, onError: console.error }); +let resolves = 0; +let workerLink: Link | undefined; +server.route( + Session, + { + authorize: (principal, address) => + principal.kind === "owner" || (principal.kind === "guest" && principal.session === address.params.id), + resolve: async (address) => { + console.log(`server: resolving ${address.key} (resolve #${++resolves})`); + const [up, down] = memoryLinkPair(); + const worker = createHost

({ + address, + principal: { kind: "worker", session: address.params.id }, + onError: console.error, + }); + worker.connect(down, { principal: { kind: "server" }, trusted: true }, context); + void worker.load([{ name: "@example/worker", facet: workerFacet }], context); + workerLink = up; + return { link: up, principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, + }, + context, +); +await server.load([{ name: "@example/server", facet: serverFacet }], context); + +/** A tab on the server, stamped as `principal` by the server's entry point. */ +async function tabAs(principal: P): Promise> { + const [up, down] = memoryLinkPair(); + server.accept(up, { principal }, context); + const tab = createHost

({ onError: console.error }); + tab.connect(down, { principal: { kind: "server" } }, context); + await tab.load([], context); + return tab; +} + +// 2. The owner's tab gets S1's notes: tab → server (relays) → route → worker. A second get shares the worker. +const owner = await tabAs({ kind: "owner" }); +const notes = await owner.get(Notes, { session: "S1" }, context); +if (notes.variant === "editor") await notes.add({ text: "milk" }, context); +const again = await owner.get(Notes, { session: "S1" }, context); +console.log("owner:", notes.variant, again.notes.value, "resolves so far:", resolves); + +// 3. A guest of S1 reads S1 and is refused S2 by `authorize` (no worker is started for it). +const guest = await tabAs({ kind: "guest", session: "S1" }); +const read = await guest.get(Notes, { session: "S1" }, context); +console.log("guest:", read.variant, read.notes.value, "acts as", await read.whoami(context)); +await guest.get(Notes, { session: "S2" }, context).catch((error) => { + console.log("guest, S2:", isChordError(error) ? error.code : error, "resolves so far:", resolves); +}); + +// 4. The guest calls the lobby on the server; its handler's `ctx.get` goes over the route as the guest. +const lobby = await guest.get(Lobby, {}, context); +console.log("guest via the lobby:", await lobby.peek({ session: "S1" }, context)); + +// 5. The worker drops: references go loading, the route resolves the address again after its backoff (1 s), and +// the new worker (a fresh run, empty notes here) binds them again. +const states: string[] = []; +const watch = await notes.state.watch(context); +watch.start(async (call) => { + if (call.kind === "value") states.push(call.value.kind); +}); +await new Promise((resolve) => setTimeout(resolve, 0)); +workerLink!.close(); +while (states.length < 2 || states.at(-1) !== "live") await new Promise((resolve) => setTimeout(resolve, 50)); +console.log("after the drop:", states.join(" → "), "notes:", notes.notes.value, "resolves:", resolves); + +await owner.close(context); +await guest.close(context); +await server.close(context); diff --git a/packages/chord/test/next/examples/13-relay.ts b/packages/chord/test/next/examples/13-relay.ts new file mode 100644 index 000000000..6203d3f52 --- /dev/null +++ b/packages/chord/test/next/examples/13-relay.ts @@ -0,0 +1,146 @@ +// A relay over real transports: a tab on a loopback WebSocket, a server that relays, and a worker per session forked +// over IPC. The worker gets its start data from its start message and serves it as `SessionInfo` through its core +// facet (§18); the server routes by address and parses no part. +// Run from packages/chord (the same file is the worker when its first argument is `worker`): +// node --conditions=source --experimental-strip-types test/next/examples/13-relay.ts +import { fork } from "node:child_process"; +import type { AddressInfo } from "node:net"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; +import { WebSocketServer } from "ws"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, hostKind, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { type IpcPort, ipcLink, webSocketLink } from "../../../src/next/link/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "owner" } | { kind: "server" } | { kind: "worker"; session: string }; + +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); + +const Init = obj({ id: str(), title: str(), pid: int() }); + +/** Start data: a local object with empty params (host lifetime) and constant state, served by the core facet. */ +const SessionInfo = local("example.session-info", { version: 1, params: {}, members: { init: Init } }); + +const Notes = remote("example.notes", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { + owner: { state: { info: Init, notes: arr(str()) }, methods: { add: method(obj({ text: str() }), int()) } }, + }, +}); + +/** What the server sends a worker it forked: the app's own message, not the link's. */ +type Start = { readonly id: string; readonly title: string }; + +/** The worker's core facet, made from its start message (base code, loaded under a fixed name). */ +const coreWorkerFacet = (init: { id: string; title: string; pid: number }) => + defineFacet

((host) => { + host.serve(SessionInfo, () => ({ init: constant(init) })); + }); + +const NotesDoc = defineState("notes", { schema: obj({ list: arr(str()) }), version: 1, initial: () => ({ list: [] }) }); + +/** A plugin's facet on the worker: its notes, and the start data read through a reference. */ +const notesFacet = defineFacet

((host) => { + host.serve(Notes, (_, _principal, self) => { + const doc = self.state(NotesDoc); + const info = self.get(SessionInfo, {}); + return { + variant: "owner", + members: { + info: info.init, + notes: self.derive([doc], ([d]) => d.list), + add: async ({ text }, ctx) => { + await doc.change((draft) => void draft.list.push(text), ctx); + return doc.value.list.length; + }, + }, + }; + }); +}); + +if (process.argv[2] === "worker") { + // The worker. 1. Its link first: it queues the link's messages until the host listens, and skips the app's own. + const link = ipcLink(process as unknown as IpcPort); + const start = await new Promise((resolve) => { + const listener = (message: unknown) => { + if (message instanceof Uint8Array) return; + process.off("message", listener); + // The start message the server's `resolve` sent: the app's own shape. + resolve(message as Start); + }; + process.on("message", listener); + }); + // 2. Its host at its address: connect first (W6), then load the core facet with the start data, then the plugins. + const worker = createHost

({ + address: Session({ id: start.id }), + principal: { kind: "worker", session: start.id }, + onError: console.error, + }); + worker.connect(link, { principal: { kind: "server" }, trusted: true }, context); + await worker.load( + [ + { name: "@example/core", facet: coreWorkerFacet({ ...start, pid: process.pid }) }, + { name: "@example/notes", facet: notesFacet }, + ], + context, + ); + // The server closing the link ends the worker. + link.onClose(() => process.exit(0)); +} else { + // The server. 3. A route forks a worker per session, sends it its start message and returns the IPC link at once. + const server = createHost

({ address: Server(), principal: { kind: "server" }, onError: console.error }); + server.route( + Session, + { + authorize: (principal) => principal.kind === "owner", + // No `signal` for the fork: the context is aborted when the host closes, which would kill the worker (and emit + // `error` on the child); a link Chord closes, at `host.close` or because it arrived after it, ends the worker. + resolve: async (address) => { + const child = fork(fileURLToPath(import.meta.url), ["worker"], { + serialization: "advanced", + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + }); + child.send({ id: address.params.id, title: `Session ${address.params.id}` } satisfies Start); + console.log(`server: forked pid ${child.pid} for ${address.key}`); + return { link: ipcLink(child), principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, + }, + context, + ); + await server.load([], context); + + // 4. Tabs reach the server over a WebSocket; its entry point stamps each as the owner here. + const wss = new WebSocketServer({ host: "127.0.0.1", port: 0 }); + await new Promise((resolve) => wss.once("listening", resolve)); + wss.on("connection", (socket) => server.accept(webSocketLink(socket), { principal: { kind: "owner" } }, context)); + const { port } = wss.address() as AddressInfo; + + // 5. A tab: Node's own WebSocket, as a browser's. Its holds go tab → server (relay) → worker. + const tab = createHost

({ onError: console.error }); + tab.connect(webSocketLink(new WebSocket(`ws://127.0.0.1:${port}`)), { principal: { kind: "server" } }, context); + await tab.load([], context); + const s1 = await tab.get(Notes, { session: "S1" }, context); + console.log("tab: S1 info", s1.info.value, "server pid", process.pid); + await s1.add({ text: "milk" }, context); + await s1.add({ text: "bread" }, context); + await new Promise((resolve) => setTimeout(resolve, 10)); + console.log("tab: S1 notes", s1.notes.value); + + // 6. Another session is another worker process. + const s2 = await tab.get(Notes, { session: "S2" }, context); + console.log("tab: S2 info", s2.info.value, "notes", s2.notes.value); + + // 7. Closing the server closes its route links; each worker exits when its link closes. + await tab.close(context); + await server.close(context); + await new Promise((resolve) => wss.close(resolve)); + console.log("closed"); +} diff --git a/packages/chord/test/next/examples/14-flow-control.ts b/packages/chord/test/next/examples/14-flow-control.ts new file mode 100644 index 000000000..4eba6e1ee --- /dev/null +++ b/packages/chord/test/next/examples/14-flow-control.ts @@ -0,0 +1,208 @@ +// Flow control: a slow tab. At most two of a provider's own `bound`/`state` messages are unacknowledged per origin; +// while the window is full the provider folds the ops of every commit into its pending queues and sends them as one +// message when an `ack` frees a slot. A relay folds and drops nothing: it forwards at once under the tab's origin and +// returns each `ack` to the worker that sent the message, so a slow tab folds at the worker, alone. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/14-flow-control.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { hostKind, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { binaryCodec, describeMessage, type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "owner" } | { kind: "server" } | { kind: "worker"; session: string }; + +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); + +/** The worker's own document of a streamed reply, and its one write. */ +const Reply = local("example.reply", { + version: 1, + params: {}, + members: { reply: obj({ text: str() }), append: method(str()) }, +}); + +/** What a tab holds: the reply, on its session's worker. */ +const View = remote("example.view", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { owner: { state: { reply: obj({ text: str() }) } } }, +}); + +const ReplyDoc = defineState("reply", { schema: obj({ text: str() }), version: 1, initial: () => ({ text: "" }) }); + +const workerFacet = defineFacet

((host) => { + host.serve(Reply, (_, self) => { + const doc = self.state(ReplyDoc); + return { + reply: doc, + append: async (token, ctx) => { + await doc.change((draft) => { + draft.text += token; + }, ctx); + return null; + }, + }; + }); + host.serve(View, (_params, _principal, self) => ({ + variant: "owner", + members: { reply: self.get(Reply, {}).reply }, + })); +}); + +/** A worker for `id`, loaded. */ +async function worker(id: string): Promise> { + const host = createHost

({ + address: Session({ id }), + principal: { kind: "worker", session: id }, + onError: console.error, + }); + await host.load([{ name: "@example/worker", facet: workerFacet }], context); + return host; +} + +/** A link end that logs, as `label`, every `bound`, `state` and `ack` its host receives. */ +function tap(link: Link, label: string): Link { + return { + send: (message) => link.send(message), + onMessage: (listener) => + link.onMessage((message) => { + const decoded = binaryCodec.decode(message); + const shown = decoded.kind === "ack" || decoded.kind === "bound" || decoded.kind === "state"; + if (shown) console.log(` ${label} ←`, describeMessage(decoded).slice(0, 110)); + listener(message); + }), + close: () => link.close(), + onClose: (listener) => link.onClose(listener), + }; +} + +/** A slow tab's end of its link: while paused, what arrives waits (so the tab acknowledges nothing). */ +class Pausable implements Link { + paused = false; + readonly #link: Link; + readonly #waiting: Uint8Array[] = []; + #listener: (message: Uint8Array) => void = () => {}; + + constructor(link: Link) { + this.#link = link; + link.onMessage((message) => { + if (this.paused) this.#waiting.push(message); + else this.#listener(message); + }); + } + + resume(): void { + this.paused = false; + for (const message of this.#waiting.splice(0)) this.#listener(message); + } + + send(message: Uint8Array): void { + this.#link.send(message); + } + + onMessage(listener: (message: Uint8Array) => void): void { + this.#listener = listener; + } + + close(): void { + this.#link.close(); + } + + onClose(listener: () => void): void { + this.#link.onClose(listener); + } +} + +/** A tab connected over `link`, loaded. */ +async function tab(link: Link): Promise> { + const host = createHost

({ onError: console.error }); + host.connect(link, { principal: { kind: "server" } }, context); + await host.load([], context); + return host; +} + +const settle = () => new Promise((resolve) => setTimeout(resolve, 10)); + +// 1. Folding at the provider: a tab directly on a worker. Paused, it acknowledges nothing; the worker sends two +// `state` messages (the window), then folds 1,000 appends into its pending queue. +console.log("a slow tab on the provider:"); +const w1 = await worker("S1"); +const [up, down] = memoryLinkPair(); +w1.accept(up, { principal: { kind: "owner" } }, context); +const slow = new Pausable(down); +const t1 = await tab(tap(slow, "tab")); +const view1 = await t1.get(View, { session: "S1" }, context); +const reply1 = await w1.get(Reply, {}, context); +slow.paused = true; +for (let i = 0; i < 1_000; i++) await reply1.append(" tok", context); +await settle(); +console.log( + " paused: the tab shows", + view1.reply.value.text.length, + "characters, the worker has", + reply1.reply.value.text.length, +); + +// 2. Resumed: the tab applies and acknowledges the two, and the freed slot carries one `state` with one folded append. +slow.resume(); +await settle(); +console.log(" resumed: the tab shows", view1.reply.value.text.length, "characters"); + +// 3. The same slow tab through a server: its holds reach the worker with the tab's ids under the tab link's origin +// (`["@",1,…]`), the worker's window for that origin fills and it folds there; a second, fast tab is another origin and +// receives every commit. +console.log("a slow tab through a relay:"); +const server = createHost

({ address: Server(), principal: { kind: "server" }, onError: console.error }); +let w2: Host

| undefined; +server.route( + Session, + { + authorize: () => true, + resolve: async (address) => { + const [a, b] = memoryLinkPair(); + w2 = await worker(address.params.id); + w2.connect(tap(b, "worker"), { principal: { kind: "server" }, trusted: true }, context); + return { link: a, principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, + }, + context, +); +await server.load([], context); +const [up2, down2] = memoryLinkPair(); +server.accept(up2, { principal: { kind: "owner" } }, context); +const slow2 = new Pausable(down2); +const t2 = await tab(tap(slow2, "tab")); +const view2 = await t2.get(View, { session: "S2" }, context); +const [up3, down3] = memoryLinkPair(); +server.accept(up3, { principal: { kind: "owner" } }, context); +const t3 = await tab(down3); +const view3 = await t3.get(View, { session: "S2" }, context); +let fast = 0; +const watch = await view3.reply.watch(context); +watch.start(async () => { + fast++; +}); +const reply2 = await w2!.get(Reply, {}, context); +slow2.paused = true; +for (let i = 0; i < 5; i++) await reply2.append(` t${i}`, context); +await settle(); +console.log(" paused: the slow tab shows", view2.reply.value, "the worker has", reply2.reply.value); +console.log(" the fast tab saw", fast, "changes:", view3.reply.value); + +// 4. Resumed: the slow tab acknowledges the two; the relay returns each `ack` to the worker with the origin, and the +// worker's freed slot carries one `state` with the three folded appends. +slow2.resume(); +await settle(); +console.log(" resumed: the slow tab shows", view2.reply.value); + +await t1.close(context); +await w1.close(context); +await watch.stop(); +await t2.close(context); +await t3.close(context); +await server.close(context); +await w2!.close(context); diff --git a/packages/chord/test/next/examples/15-revocation.ts b/packages/chord/test/next/examples/15-revocation.ts new file mode 100644 index 000000000..1a004ffa6 --- /dev/null +++ b/packages/chord/test/next/examples/15-revocation.ts @@ -0,0 +1,142 @@ +// Revocation: the app says which revocations apply to a principal (`revokedBy`, the same function on every host); +// `host.revoke(r)` ends that principal's objects at once, whatever references remain, closes its child links (tabs) and +// sends `revoke` down every route link, where each worker applies it with its own `revokedBy`. Nothing is replayed. +// What ends gets the error `forbidden` with reason `revoked` (a UI can say "access revoked"): holders get it in `gone`, +// running handlers in their aborted context. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/15-revocation.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, hostKind, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import type { ChordError } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; + +const context = BACKGROUND_CONTEXT; + +/** The app's principals: a guest holds a grant to one session; the server and its workers act as themselves. */ +type P = + | { kind: "owner" } + | { kind: "guest"; grant: string; session: string } + | { kind: "server" } + | { kind: "worker"; session: string }; + +/** What the app revokes: a grant, or everything of a session. */ +type Revocation = { grant: string } | { session: string }; + +/** The same function on every host: a guest is revoked by its grant and by its session. */ +const revokedBy = (p: P): Revocation[] => (p.kind === "guest" ? [{ grant: p.grant }, { session: p.session }] : []); + +const Server = hostKind("server"); +const Session = hostKind("session", { id: str() }); + +const Notes = remote("example.notes", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { + all: { state: { notes: arr(str()) }, methods: { add: method(obj({ text: str() })), follow: method() } }, + }, +}); + +const NotesDoc = defineState("notes", { schema: obj({ list: arr(str()) }), version: 1, initial: () => ({ list: [] }) }); + +const notesFacet = defineFacet

((host) => { + host.serve(Notes, (_, principal, self) => { + const doc = self.state(NotesDoc); + console.log(` worker: Notes for ${principal.kind}${principal.kind === "guest" ? ` ${principal.grant}` : ""}`); + return { + variant: "all", + members: { + notes: self.derive([doc], ([d]) => d.list), + add: async ({ text }, ctx) => { + await doc.change((draft) => void draft.list.push(text), ctx); + return null; + }, + // A long-running call: it ends when its context is aborted, and says why. + follow: async (ctx) => { + await new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", resolve)); + const { code, reason } = ctx.abortSignal!.reason as ChordError; + console.log(` worker: ${principal.kind}'s follow aborted: ${code}/${reason}`); + return null; + }, + }, + }; + }); +}); + +// 1. A server with `revokedBy`, routing sessions to in-process workers (trusted links) with the same `revokedBy`. +const server = createHost({ + address: Server(), + principal: { kind: "server" }, + revokedBy, + onError: console.error, +}); +const workers: Host[] = []; +server.route( + Session, + { + authorize: (p, address) => p.kind === "owner" || (p.kind === "guest" && p.session === address.params.id), + resolve: async (address) => { + const [up, down] = memoryLinkPair(); + const worker = createHost({ + address, + principal: { kind: "worker", session: address.params.id }, + revokedBy, + onError: console.error, + }); + worker.connect(up, { principal: { kind: "server" }, trusted: true }, context); + void worker.load([{ name: "@example/notes", facet: notesFacet }], context); + workers.push(worker); + return { link: down, principal: { kind: "worker", session: address.params.id }, trusted: true }; + }, + }, + context, +); +await server.load([], context); + +/** A tab whose link the server's entry point stamped with `principal`. */ +async function tab(principal: P) { + const host = createHost

({ onError: console.error }); + await host.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal }, context); + host.connect(b, { principal: { kind: "server" } }, context); + return host; +} + +// 2. Two tabs of guest grant g1 (one person, two windows) and the owner's tab, all on session S1. +const [one, two, owner] = await Promise.all([ + tab({ kind: "guest", grant: "g1", session: "S1" }), + tab({ kind: "guest", grant: "g1", session: "S1" }), + tab({ kind: "owner" }), +]); +const guestNotes = await one.get(Notes, { session: "S1" }, context); +await two.get(Notes, { session: "S1" }, context); +const ownerNotes = await owner.get(Notes, { session: "S1" }, context); +await guestNotes.add({ text: "from the guest" }, context); +const following = guestNotes.follow(context).catch((error: ChordError) => error.code); +// The call reaches the worker and runs there. +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log("worker S1 references:", workers[0]!.references(Notes).value.length); + +// 3. Revoke the grant: the worker gets `revoke { grant: "g1" }` and ends the guest's objects there (their running +// calls aborted with forbidden/revoked), and both guest tabs' links close. +server.revoke({ grant: "g1" }, context); +await new Promise((resolve) => setTimeout(resolve, 10)); +console.log("guest tab:", guestNotes.state.value.kind, "(its link closed; a new one needs a new authentication)"); +console.log("guest's follow call:", await following); +// The owner's Notes is its own object (identities include the principal), untouched. +console.log("owner tab:", ownerNotes.state.value.kind, ownerNotes.notes.value); +console.log( + "worker S1 references:", + workers[0]!.references(Notes).value.map(({ principal }) => principal?.kind), +); + +// 4. A worker resolved later gets no past revocation: its objects are new, and the revoked links are closed already. +const s2 = await owner.get(Notes, { session: "S2" }, context); +console.log("owner on S2:", s2.state.value.kind); + +await Promise.all([one, two, owner].map((host) => host.close(context))); +await server.close(context); +await Promise.all(workers.map((worker) => worker.close(context))); diff --git a/packages/chord/test/next/examples/16-presence-and-activity.ts b/packages/chord/test/next/examples/16-presence-and-activity.ts new file mode 100644 index 000000000..9091557e0 --- /dev/null +++ b/packages/chord/test/next/examples/16-presence-and-activity.ts @@ -0,0 +1,89 @@ +// Presence and keep-alive counts. `host.references(Type)` is a JSON watchable of the references to objects of a type +// on this host, one entry per reference with the object's params, its principal and its incarnation (two tabs are two +// origins); a derive over it is a viewer count, and a link drop is a departure in the same batch. `host.activity` counts +// what keeps a host busy (holds over links, running calls, jobs, lingering objects), published at most once per +// microtask turn: the app's supervisor decides when an idle worker stops. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/16-presence-and-activity.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "user"; name: string } | { kind: "server" }; + +/** A shared document view: who else is looking (`viewers`), and a slow method. */ +const Doc = remote("example.doc", { + version: 1, + params: { id: str() }, + variants: { v: { state: { viewers: int() }, methods: { think: method(undefined, int()) } } }, +}); + +// 1. The server serves `Doc`; its factory derives the viewer count from `host.references(Doc)` (on the facet's host). +const server = createHost

({ principal: { kind: "server" }, onError: console.error }); +const facet = defineFacet

((host) => { + host.serve( + Doc, + ({ id }, _principal, self) => ({ + variant: "v", + members: { + viewers: self.derive([host.references(Doc)], ([refs]) => refs.filter((ref) => ref.params.id === id).length), + think: async (ctx) => { + await new Promise((resolve) => setTimeout(resolve, 20)); + return ctx.principal?.kind === "user" ? 42 : 0; + }, + }, + }), + { linger: 50 }, + ); +}); +await server.load([{ name: "@example/doc", facet }], context); + +// 2. The supervisor's view: every publication of the counts. +const activity = await server.activity.watch(context); +activity.start(async (call) => { + if (call.kind === "value") console.log(" activity:", JSON.stringify(call.value)); +}); + +/** A tab of `name` on a new link; returns the tab and the server's end of its link. */ +async function tab(name: string) { + const host = createHost

({ onError: console.error }); + await host.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "user", name } }, context); + host.connect(b, { principal: { kind: "server" } }, context); + return { host, link: a }; +} + +// 3. Ada opens the document in two tabs, Bob in one: three references, three incarnations, two objects (one per principal); +// the viewer count is over every object of the document. +const [ada1, ada2, bob] = await Promise.all([tab("ada"), tab("ada"), tab("bob")]); +const doc = await ada1.host.get(Doc, { id: "d1" }, context); +await ada2.host.get(Doc, { id: "d1" }, context); +await bob.host.get(Doc, { id: "d1" }, context); +await new Promise((resolve) => setTimeout(resolve, 5)); +console.log("viewers seen by ada:", doc.viewers.value); +const present = server + .references(Doc) + .value.map( + ({ principal, incarnation }) => `${principal?.kind === "user" ? principal.name : "?"} @ ${incarnation.slice(-4)}`, + ); +console.log("references:", present); + +// 4. A call running counts as activity until it settles. +await doc.think(context); + +// 5. Bob's link drops: a departure, at once. Bob's own `Doc` object (identities include the principal) lingers. +bob.link.close(); +await new Promise((resolve) => setTimeout(resolve, 5)); +console.log("viewers after bob's link dropped:", doc.viewers.value); + +// 6. Ada closes both tabs: the object lingers (50 ms) with no references, then ends; then everything is zero. +await ada1.host.close(context); +await ada2.host.close(context); +await new Promise((resolve) => setTimeout(resolve, 100)); +console.log("references at the end:", server.references(Doc).value.length); +await activity.stop(); +await server.close(context); diff --git a/packages/chord/test/next/examples/17-limits-and-compression.ts b/packages/chord/test/next/examples/17-limits-and-compression.ts new file mode 100644 index 000000000..970dce626 --- /dev/null +++ b/packages/chord/test/next/examples/17-limits-and-compression.ts @@ -0,0 +1,189 @@ +// Size limits per part and compression at the origin. A host with a synchronous compressor (`nodeCompression` on +// Node) deflates each JSON part above 16 KiB once, when it encodes it; relays forward it as it is; the end that owns it +// inflates it, within its link's decoded limit. A browser has no synchronous deflate: it sends uncompressed and +// inflates with `DecompressionStream`, later messages of its link waiting in order. Only a part that shrinks is sent +// deflated, and never hold params. Clients never send compressed parts: a host sends its call arguments up its parent +// link uncompressed, and a link the server accepted (a child) refuses compressed parts as a protocol error (a small +// part could inflate to 64 MiB there). A part past a limit fails only its own hold or call (`too_large/part`); the +// link stays. Limits and the threshold count UTF-8 bytes. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/17-limits-and-compression.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { type Compression, type Link, memoryLinkPair, type PartLimits } from "../../../src/next/link/index.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "user" } | { kind: "server" }; + +const Entry = obj({ id: str(), text: str() }); + +/** The shared store of a conversation: its entries, once, and the method that appends to them. */ +const Store = local("example.store", { + version: 1, + params: { id: str() }, + members: { entries: arr(Entry), append: method(Entry) }, +}); + +/** What a tab holds: the conversation's entries, and appending. */ +const Conversation = remote("example.conversation", { + version: 1, + params: { id: str() }, + variants: { v: { state: { entries: arr(Entry) }, methods: { append: method(Entry) } } }, +}); + +const Entries = defineState("entries", { + schema: obj({ list: arr(Entry) }), + version: 1, + initial: () => ({ list: [] }), +}); + +const facet = defineFacet

((host) => { + host.serve(Store, (_, self) => { + const doc = self.state(Entries); + return { + entries: self.derive([doc], ([d]) => d.list), + append: async (entry, ctx) => { + await doc.change((draft) => void draft.list.push(entry), ctx); + return null; + }, + }; + }); + host.serve(Conversation, ({ id }, _principal, self) => { + const store = self.get(Store, { id }); + return { variant: "v", members: { entries: store.entries, append: (entry, ctx) => store.append(entry, ctx) } }; + }); +}); + +/** A link end counting the bytes its host receives. */ +function counted(link: Link, bytes: { received: number }): Link { + return { + send: (message) => link.send(message), + onMessage: (listener) => + link.onMessage((message) => { + bytes.received += message.byteLength; + listener(message); + }), + close: () => link.close(), + onClose: (listener) => link.onClose(listener), + }; +} + +/** + * A server compressing with `compression` and a tab without a compressor (a browser) on a link the server accepts with + * `limits`; the tab's own link to the server has `tabLimits`. + */ +async function setUp(compression: Compression | undefined, limits?: PartLimits, tabLimits?: PartLimits) { + const server = createHost

({ principal: { kind: "server" }, compression, onError: console.error }); + await server.load([{ name: "@example/conversation", facet }], context); + const tab = createHost

({ onError: console.error }); + await tab.load([], context); + const [a, b] = memoryLinkPair(); + const bytes = { received: 0 }; + server.accept(a, { principal: { kind: "user" }, limits }, context); + tab.connect(counted(b, bytes), { principal: { kind: "server" }, limits: tabLimits }, context); + return { server, tab, bytes }; +} + +/** Words for entries that compress like prose (about 3x), not like a repeated string. */ +const WORDS = "the a model tool call file read write test error result agent session note code run build".split(" "); + +/** Entry `i`'s text: about 1 KB of words picked by a fixed pseudo-random sequence. */ +function text(i: number): string { + let seed = i * 7919 + 1; + const words: string[] = []; + for (let length = 0; length < 1000; length += words.at(-1)!.length + 1) { + seed = (seed * 1103515245 + 12345) % 2147483648; + const word = WORDS[seed % WORDS.length]!; + words.push(seed % 5 === 0 ? `${word}${seed % 1000}` : word); + } + return words.join(" "); +} + +/** Fills the server's conversation `id` with 500 entries of about 1 KB (its store kept by this reference). */ +async function fill(server: Host

, id: string): Promise { + const conversation = await server.get(Store, { id }, context); + for (let i = 0; i < 500; i++) await conversation.append({ id: `e${i}`, text: text(i) }, context); +} + +// 1. A 500 KB snapshot to a browser tab, without and with compression at the server. +for (const compression of [undefined, nodeCompression]) { + const { server, tab, bytes } = await setUp(compression); + await fill(server, "c1"); + const started = performance.now(); + const conversation = await tab.get(Conversation, { id: "c1" }, context); + const ms = (performance.now() - started).toFixed(1); + const label = compression === undefined ? "uncompressed" : "deflated at the server, inflated asynchronously"; + console.log( + `snapshot ${label}: ${conversation.entries.value.length} entries, ${(bytes.received / 1024).toFixed(1)} KiB received, ${ms} ms`, + ); + await tab.close(context); + await server.close(context); +} + +// 2. Limits per part: the server admits at most 64 KiB per part from this tab. A larger argument fails its call only. +{ + const { server, tab } = await setUp(nodeCompression, { part: 64 * 1024 }); + const conversation = await tab.get(Conversation, { id: "c2" }, context); + try { + await conversation.append({ id: "big", text: "x".repeat(100_000) }, context); + } catch (error) { + console.log( + "a 100 KB argument:", + isChordError(error) ? `${error.code}/${error.reason}: ${error.message}` : error, + ); + } + await conversation.append({ id: "small", text: "fits" }, context); + console.log( + "the link stays: entries", + conversation.entries.value.map(({ id }) => id), + ); + await tab.close(context); + await server.close(context); +} + +// 3. A compression bomb: 50 KB deflated that would inflate to 6 MB, past the tab's decoded limit of 1 MiB. The tab +// stops inflating at the limit; only that hold fails. +{ + const { server, tab } = await setUp(nodeCompression, undefined, { decoded: 1024 * 1024 }); + const bomb = await server.get(Store, { id: "bomb" }, context); + for (let i = 0; i < 6; i++) await bomb.append({ id: `z${i}`, text: "0".repeat(1_000_000) }, context); + await fill(server, "fine"); + try { + await tab.get(Conversation, { id: "bomb" }, context); + } catch (error) { + console.log("the bomb's hold:", isChordError(error) ? `${error.code}/${error.reason}` : error); + } + const fine = await tab.get(Conversation, { id: "fine" }, context); + console.log("another hold on the same link:", fine.entries.value.length, "entries"); + await tab.close(context); + await server.close(context); +} + +// 4. A client with a compressor still sends its arguments up uncompressed: the server, which accepted its link as a +// child, would refuse a compressed part. Its 100 KB argument arrives as JSON, within the server's default limits. +{ + const server = createHost

({ + principal: { kind: "server" }, + compression: nodeCompression, + onError: console.error, + }); + await server.load([{ name: "@example/conversation", facet }], context); + const client = createHost

({ compression: nodeCompression, onError: console.error }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + const bytes = { received: 0 }; + server.accept(counted(a, bytes), { principal: { kind: "user" } }, context); + client.connect(b, { principal: { kind: "server" } }, context); + const conversation = await client.get(Conversation, { id: "c4" }, context); + await conversation.append({ id: "big", text: text(0).repeat(100) }, context); + console.log( + `a compressing client's 100 KB argument: ${(bytes.received / 1024).toFixed(1)} KiB received by the server`, + ); + await client.close(context); + await server.close(context); +} diff --git a/packages/chord/test/next/examples/18-routines.ts b/packages/chord/test/next/examples/18-routines.ts new file mode 100644 index 000000000..8ff6b8d65 --- /dev/null +++ b/packages/chord/test/next/examples/18-routines.ts @@ -0,0 +1,115 @@ +// Routines: an object's standing behaviour, declared in its factory. `self.on(source, fn)` reconciles from a value: the +// first call gets the current value (`[["r", value]]`), then one call per change, never overlapping; a call that +// throws is reported (`routine_failed`) and called again with the whole current value after a backoff (1 s doubling to +// 60 s), or at once when the source changes. `self.onLive(fn)` runs once the object is live: a loop with `sleep` is a +// periodic task; its ctx is aborted when the object ends, which `sleep` turns into a rejection, so nothing is left +// running. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/18-routines.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, bool, local, map, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, sleep } from "../../../src/next/host/index.ts"; + +const context = BACKGROUND_CONTEXT; + +/** The catalog of sessions: another owner's state, which the forwards reconcile from. */ +const Catalog = local("example.catalog", { + version: 1, + params: {}, + members: { sessions: map(obj({ archived: bool() })), archive: method(obj({ session: str() })) }, +}); + +/** Port forwards per session; a forward of an archived session must be closed (an invariant across owners, §9). */ +const ForwardStore = local("example.forwards", { + version: 1, + params: {}, + members: { open: arr(str()), cleanups: arr(str()) }, +}); + +const Sessions = defineState("example.sessions", { + schema: obj({ sessions: map(obj({ archived: bool() })) }), + version: 1, + initial: () => ({ sessions: { s1: { archived: false }, s2: { archived: false } } }), +}); + +const Forwards = defineState("example.forward-list", { + schema: obj({ open: arr(str()), cleanups: arr(str()) }), + version: 1, + initial: () => ({ open: ["s1", "s2"], cleanups: [] }), +}); + +const started = Date.now(); +const at = () => `${Math.round((Date.now() - started) / 100) / 10}s`; + +// 1. The host reports background failures: a routine call that threw arrives as `routine_failed`. +const host = createHost({ + onError: (error) => { + if (error.kind === "routine_failed") { + const { kind, position } = error.routine; + console.log(` [${at()}] ${error.object} ${kind} #${position} failed: ${(error.error as Error).message}`); + } else console.error(error); + }, +}); + +let flaky = true; +const facet = defineFacet((h) => { + h.serve(Catalog, (_, self) => { + const doc = self.state(Sessions); + return { + sessions: self.derive([doc], ([d]) => d.sessions), + archive: async ({ session }, ctx) => { + await doc.change((d) => { + d.sessions[session] = { archived: true }; + }, ctx); + return null; + }, + }; + }); + h.serve(ForwardStore, (_, self) => { + const forwards = self.state(Forwards); + const catalog = self.get(Catalog, {}); + // 2. Reconcile: close every forward of an archived session. Idempotent, from the current value; the first close + // fails once (a flaky remote), and the retry comes with the whole value. + self.on(catalog.sessions, async ({ value: sessions, ops }, ctx) => { + console.log(` [${at()}] on: ${JSON.stringify(ops)}`); + for (const session of forwards.value.open) { + if (!sessions[session]?.archived) continue; + if (flaky) { + flaky = false; + throw new Error(`closing the forward of ${session} timed out`); + } + await forwards.change((d) => { + d.open = d.open.filter((open) => open !== session); + }, ctx); + console.log(` [${at()}] closed the forward of ${session}`); + } + }); + // 3. A periodic task: a loop with `sleep`, ended by the object's end (the abort rejects `sleep`). + self.onLive(async (ctx) => { + for (;;) { + await sleep(400, ctx); + await forwards.change((d) => void d.cleanups.push(at()), ctx); + } + }); + return { + open: self.derive([forwards], ([d]) => d.open), + cleanups: self.derive([forwards], ([d]) => d.cleanups), + }; + }); +}); +await host.load([{ name: "@example/forwards", facet }], context); +const store = await host.get(ForwardStore, {}, context); +const catalog = await host.get(Catalog, {}, context); +await sleep(50, context); + +// 4. Archive s2: the routine's call throws once, is retried after 1 s with the whole value, and closes the forward. +console.log("archiving s2"); +await catalog.archive({ session: "s2" }, context); +await sleep(1_300, context); +console.log("open forwards:", store.open.value); +console.log("periodic cleanups so far:", store.cleanups.value); + +// 5. Closing the host aborts the routines: the periodic loop ends in its `sleep`, nothing is reported. +await host.close(context); +console.log("closed; nothing left running"); diff --git a/packages/chord/test/next/examples/19-jobs.ts b/packages/chord/test/next/examples/19-jobs.ts new file mode 100644 index 000000000..f52ee317a --- /dev/null +++ b/packages/chord/test/next/examples/19-jobs.ts @@ -0,0 +1,211 @@ +// Jobs: owned work a call starts that outlives the call, in Durable's sandwich: intent (`start`, one transaction), run +// (any effect, outside transactions) and finish (one transaction on the current code, with the cleanup). Here a shell +// job of a shared chat store: its state document carries the output, throttled by the author and flushed in a +// `finally`, so the last output reaches `outcome.state` after an abort too; `running` is ordinary state, which +// per-principal views filter (a guest sees no output); `finish` records an entry and chains a second job. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/19-jobs.ts +import { BACKGROUND_CONTEXT, type Context } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, remote, type Static, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, sleep } from "../../../src/next/host/index.ts"; +import { defineJob } from "../../../src/next/jobs/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { mapShared } from "../../../src/next/state/index.ts"; +import { OutputBuffer, Progress } from "../../../src/output.ts"; + +const context = BACKGROUND_CONTEXT; + +type P = { kind: "owner" } | { kind: "guest" } | { kind: "server" }; + +// 1. The job: closed args (checked at every start), a result, a state document per run, one run at a time. +const ShellState = defineState("example.shell.state", { + schema: obj({ output: str() }), + version: 1, + initial: () => ({ output: "" }), +}); +const ShellJob = defineJob("example.shell", { + version: 1, + args: obj({ command: str(), lines: int() }), + result: obj({ exit: int(), output: str() }), + state: ShellState, + concurrency: 1, +}); +/** A second job `finish` chains: it announces a finished command. */ +const NotifyJob = defineJob("example.notify", { version: 1, args: obj({ text: str() }), result: str() }); + +const Entries = defineState("example.entries", { + schema: obj({ list: arr(str()) }), + version: 1, + initial: () => ({ list: [] }), +}); + +/** A shell as its owner sees it; a guest's view has no output. */ +const Shell = obj({ id: str(), phase: str(), command: str(), output: str() }); +const GuestShell = obj({ id: str(), phase: str(), command: str() }); + +/** The shared store: the entries and the shells once, whoever views them (§6). */ +const ChatStore = local("example.chat-store", { + version: 1, + params: {}, + members: { + entries: arr(str()), + shells: arr(Shell), + runShell: method(obj({ command: str(), lines: int(), requestId: str() }), str()), + abortShell: method(obj({ id: str() })), + }, +}); + +/** Per-principal views: the owner runs shells and sees their output, a guest sees which shells run. */ +const Chat = remote("example.chat", { + version: 1, + params: {}, + variants: { + owner: { + state: { entries: arr(str()), shells: arr(Shell) }, + methods: { + runShell: method(obj({ command: str(), lines: int(), requestId: str() }), str()), + abortShell: method(obj({ id: str() })), + }, + }, + guest: { state: { entries: arr(str()), shells: arr(GuestShell) } }, + }, +}); + +/** A fake process: one line every 30 ms; `sleep` rejects when the run is aborted. */ +async function runProcess(lines: number, ctx: Context, onLine: (line: string) => void): Promise { + for (let n = 1; n <= lines; n++) { + await sleep(30, ctx); + onLine(`line ${n}\n`); + } + return 0; +} + +const facet = defineFacet

((host) => { + host.serve(ChatStore, (_, self) => { + const entries = self.state(Entries); + const notify = self.job(NotifyJob, { + run: async ({ text }) => `notified: ${text}`, + finish: async (outcome, tx) => { + if (outcome.kind === "done") (await tx.draft(entries)).list.push(outcome.result); + }, + }); + const shell = self.job(ShellJob, { + run: async ({ lines }, ctx) => { + // The output's bounds are required: a tail of at most 2,000 lines and 64 KiB. + const out = new OutputBuffer({ maxBytes: 64 * 1024, maxLines: 2_000, retain: "tail" }); + // Every state change is a commit and, while watched, a message: the author throttles them (50 ms here). + const write = async () => { + const text = out.snapshot().text; + await ctx.state.change((draft) => { + draft.output = text; + }, ctx); + return text.length; + }; + const progress = new Progress(write, (error) => console.error("progress failed:", error), 50); + try { + const exit = await runProcess(lines, ctx, (line) => { + out.push(line); + progress.mark(); + }); + return { exit, output: out.snapshot().text }; + } finally { + // The last output reaches `outcome.state`, also after an abort (changes are taken until run settles). + await progress.stop(); + await write(); + } + }, + finish: async (outcome, tx) => { + const { command } = outcome.args; + const lines = outcome.state.output.split("\n").length - 1; + const text = + outcome.kind === "done" + ? `${command}: exit ${outcome.result.exit}, ${lines} lines` + : `${command}: interrupted (${outcome.error.kind}), ${lines} lines kept`; + (await tx.draft(entries)).list.push(text); + // Chaining: the next job starts atomically with this outcome (both commit or neither). + tx.start(notify, { text: command }); + }, + }); + // What the views read: one object per run, kept while the run's entry is the same object (`mapShared`). + const toShell = mapShared( + (entry: (typeof shell.running.value)[number]): Static => + entry.phase === "blocked" + ? { id: entry.id, phase: entry.phase, command: "?", output: "" } + : { id: entry.id, phase: entry.phase, command: entry.args.command, output: entry.state.output }, + ); + return { + entries: self.derive([entries], ([d]) => d.list), + shells: self.derive([shell.running], ([running]) => toShell(running)), + runShell: async ({ command, lines, requestId }, ctx) => + (await shell.start({ command, lines }, ctx, { key: requestId })).id, + abortShell: async ({ id }) => { + shell.abort(id); + return null; + }, + }; + }); + host.serve(Chat, (_, principal, self) => { + const store = self.get(ChatStore, {}); + if (principal.kind !== "owner") { + // A guest sees no output: the view drops it (computed once per variant in a real store, §12). + const strip = mapShared( + (s: Static): Static => ({ id: s.id, phase: s.phase, command: s.command }), + ); + return { + variant: "guest", + members: { entries: store.entries, shells: self.derive([store.shells], ([list]) => strip(list)) }, + }; + } + return { + variant: "owner", + members: { + entries: store.entries, + shells: store.shells, + runShell: (args, ctx) => store.runShell(args, ctx), + abortShell: (args, ctx) => store.abortShell(args, ctx), + }, + }; + }); +}); + +// 2. A server and two tabs, the owner's and a guest's. +const server = createHost

({ principal: { kind: "server" }, onError: (error) => console.error(error) }); +await server.load([{ name: "@example/chat", facet }], context); +async function tab(principal: P) { + const host = createHost

({ onError: (error) => console.error(error) }); + await host.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal }, context); + host.connect(b, { principal: { kind: "server" } }, context); + return host.get(Chat, {}, context); +} +const owner = await tab({ kind: "owner" }); +const guest = await tab({ kind: "guest" }); +if (owner.variant !== "owner") throw new Error("expected the owner's variant"); + +// 3. The owner runs a command; its request id dedupes a resent request (the same run comes back). +const id = await owner.runShell({ command: "npm test", lines: 5, requestId: "r1" }, context); +const again = await owner.runShell({ command: "npm test", lines: 5, requestId: "r1" }, context); +console.log("a resent request returns the same run:", again === id); +await sleep(100, context); +console.log("owner sees:", owner.shells.value); +console.log("guest sees:", guest.shells.value); + +// 4. One run at a time: a second command is refused while the first runs. +try { + await owner.runShell({ command: "npm run build", lines: 1, requestId: "r2" }, context); +} catch (error) { + console.log("second run refused:", (error as Error).message); +} +await sleep(300, context); +console.log("entries after the first run:", owner.entries.value); + +// 5. Abort is a request: the run stops at its next `sleep`, its last output is flushed, `finish` records it. +const long = await owner.runShell({ command: "tail -f log", lines: 100, requestId: "r3" }, context); +await sleep(100, context); +await owner.abortShell({ id: long }, context); +await sleep(100, context); +console.log("entries after the abort:", owner.entries.value); +console.log("running now:", owner.shells.value); +await server.close(context); diff --git a/packages/chord/test/next/examples/20-tables.ts b/packages/chord/test/next/examples/20-tables.ts new file mode 100644 index 000000000..5b626fe3a --- /dev/null +++ b/packages/chord/test/next/examples/20-tables.ts @@ -0,0 +1,101 @@ +// Tables: an object's rows in the host's store, declared with `defineTable`, written in transactions (with the +// object's documents, if it has any), read by key, by page with cursors, and counted. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/20-tables.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, bool, int, local, method, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { add, defineTable, eq, openStore, prefix, query, sqliteBackend } from "../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const context = BACKGROUND_CONTEXT; + +// Flat columns (str, int, num, bool, opt(...), json(...)), defaults, a key of non-opt str or int columns, and +// indexes, which end with the key. +const Edits = defineTable("edits", { + version: 1, + columns: { + id: str(), + path: str(), + at: int(), + hits: int({ default: 0 }), + done: bool({ default: false }), + note: opt(str()), + }, + key: ["id"], + indexes: { by_path: ["path", "at"] }, +}); + +const History = local("example.history", { + version: 1, + params: {}, + members: { + record: method(obj({ id: str(), path: str(), at: int() })), + hit: method(str()), + finish: method(str()), + hits: method(str(), int()), + open: method(obj({ after: opt(str()) }), obj({ ids: arr(str()), next: opt(str()) })), + finished: method(obj({}), int()), + }, +}); + +const facet = defineFacet((host) => { + // Tables need a stable identity: a local type with empty params (one object per host). + host.serve(History, (_, self) => { + const edits = self.table(Edits); + // A query of the table: a condition and an order (the key is the tie-break); immutable, built once. + const open = query(Edits) + .where((e) => eq(e.done, false)) + .orderBy((e) => e.at, "desc"); + return { + record: async (row, ctx) => { + // Writes are staged synchronously (a bad row throws here) and commit together in one step. + await self.tx(async (tx) => tx.put(edits, row), ctx); + return null; + }, + hit: async (id, ctx) => { + // A set is values or an expression over the row's columns, computed by the store. + await self.tx(async (tx) => tx.update(edits, { id }, (e) => ({ hits: add(e.hits, 1) })), ctx); + return null; + }, + finish: async (path, ctx) => { + // One statement for every matching row; its count is recorded with the write. + await self.tx(async (tx) => tx.updateWhere(edits, (e) => prefix(e.path, path), { done: true }), ctx); + return null; + }, + hits: async (id, ctx) => (await edits.get({ id }, ctx))?.hits ?? 0, + open: async ({ after }, ctx) => { + const page = await edits.page(open, 1, ctx, after === undefined ? {} : { after }); + return { ids: page.items.map((row) => row.id), ...(page.next === undefined ? {} : { next: page.next }) }; + }, + finished: async (_, ctx) => + edits.count( + query(Edits).where((e) => eq(e.done, true)), + ctx, + ), + }; + }); +}); + +// A store over SQLite (`:memory:` here, a file in an app): the app opens it, gives it to one host, and closes it. +const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context); +const host = createHost({ store, onError: (error) => console.error("background error:", error) }); +await host.load([{ name: "@example/history", facet }], context); +const history = await host.get(History, {}, context); + +for (let i = 0; i < 5; i++) { + await history.record({ id: `e${i}`, path: `src/${i % 2 === 0 ? "a" : "b"}.ts`, at: i }, context); +} +await history.hit("e1", context); +await history.hit("e1", context); +await history.finish("src/a", context); + +console.log("hits of e1:", await history.hits("e1", context)); +// Pages continue from an opaque cursor. +const first = await history.open({}, context); +console.log("open, first page:", first.ids); +console.log("open, next page:", (await history.open({ after: first.next }, context)).ids); +console.log("finished:", await history.finished({}, context)); + +await host.close(context); +await store.close(context); diff --git a/packages/chord/test/next/examples/21-persisted-documents.ts b/packages/chord/test/next/examples/21-persisted-documents.ts new file mode 100644 index 000000000..e9771455b --- /dev/null +++ b/packages/chord/test/next/examples/21-persisted-documents.ts @@ -0,0 +1,91 @@ +// Persisted documents: `persist: true` keeps an object's documents in the host's store, so a new host on the same +// store finds them again; `checkpointWhen` decides when a change is stored as a new base instead of a delta. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/21-persisted-documents.ts +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Reference } from "../../../src/next/host/index.ts"; +import { openStore, sqliteBackend } from "../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const context = BACKGROUND_CONTEXT; + +// A persisted singleton: loaded once when its object activates; every 20th stored change is a new base. +const Tags = defineState("tags", { + schema: obj({ tags: arr(str()) }), + version: 1, + initial: () => ({ tags: [] }), + persist: true, + checkpointWhen: (_value, _ops, { deltasSinceBase }) => deltasSinceBase >= 20, +}); +// A persisted keyed document: each key loaded on its first read, then cached until idle for ten minutes. +const Notes = defineState("notes", { + schema: obj({ body: str() }), + version: 1, + initial: () => ({ body: "" }), + keyed: true, + persist: true, + checkpointWhen: () => false, +}); + +const Notebook = local("example.notebook", { + version: 1, + params: {}, + members: { + tags: obj({ tags: arr(str()) }), + write: method(obj({ id: str(), body: str(), tag: str() })), + read: method(str(), str()), + }, +}); + +const facet = defineFacet((host) => { + // Persisted documents need a stable identity, as tables do: a local type with empty params. + host.serve(Notebook, (_, self) => { + const tags = self.state(Tags); + const notes = self.state(Notes); + return { + tags, + write: async ({ id, body, tag }, ctx) => { + // Both documents commit in one store step, or neither does. + await self.tx(async (tx) => { + (await tx.draft(notes, id)).body = body; + const draft = await tx.draft(tags); + if (!draft.tags.includes(tag)) draft.tags.push(tag); + }, ctx); + return null; + }, + read: async (id, ctx) => (await notes.get(id, ctx))?.body ?? "(none)", + }; + }); +}); + +/** A host on the store at `path`, run until `work` is done with the notebook. */ +async function session(path: string, work: (notebook: Reference) => Promise) { + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context); + const host = createHost({ store, onError: (error) => console.error("background error:", error) }); + await host.load([{ name: "@example/notebook", facet }], context); + await work(await host.get(Notebook, {}, context)); + await host.close(context); + await store.close(context); +} + +const directory = await mkdtemp(join(tmpdir(), "chord-example-")); +const path = join(directory, "notebook.db"); +try { + await session(path, async (notebook) => { + await notebook.write({ id: "groceries", body: "oat milk", tag: "home" }, context); + await notebook.write({ id: "standup", body: "demo the store", tag: "work" }, context); + console.log("first host, tags:", notebook.tags.value.tags); + }); + // A new host on the same file: the documents come back from the store. + await session(path, async (notebook) => { + console.log("second host, tags:", notebook.tags.value.tags); + console.log("second host, standup:", await notebook.read("standup", context)); + }); +} finally { + await rm(directory, { recursive: true, force: true }); +} diff --git a/packages/chord/test/next/examples/22-migrations.ts b/packages/chord/test/next/examples/22-migrations.ts new file mode 100644 index 000000000..f9932070d --- /dev/null +++ b/packages/chord/test/next/examples/22-migrations.ts @@ -0,0 +1,100 @@ +// Table migrations: `evolveTable` appends a version with its edits (rename, add, json before an optional data step; +// drop, index, dropIndex after it). An object whose store holds an older version migrates it when it activates, one +// version at a time, in chunks of 100 rows; a fresh store gets the newest version directly. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/22-migrations.ts +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet, type Reference } from "../../../src/next/host/index.ts"; +import { + defineTable, + evolveTable, + openStore, + query, + sqliteBackend, + type TableDefinition, +} from "../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const context = BACKGROUND_CONTEXT; + +// Version 1, the base, written in full. +const EditsV1 = defineTable("edits", { version: 1, columns: { id: str(), file: str(), at: int() }, key: ["id"] }); +// Version 2: `file` becomes `path`, a `dir` column is added and filled from each row (the data step). +const EditsV2 = evolveTable(EditsV1, (t) => + t + .rename("file", "path") + .add("dir", str({ default: "" })) + .rows((row) => ({ dir: row.path.slice(0, row.path.lastIndexOf("/")) })), +); +// Version 3: `dir` is dropped again, an index is added. Data functions stay as deployed; history only grows. +const EditsV3 = evolveTable(EditsV2, (t) => t.drop("dir").index("by_path", ["path", "at"])); + +const History = local("example.history", { + version: 1, + params: {}, + members: { record: method(obj({ id: str(), at: int() })), dump: method(obj({}), arr(str())) }, +}); + +/** The facet of one deployment: its newest table version, and how it writes a row. */ +function facetOf(definition: TableDefinition, row: (id: string, at: number) => Record) { + return defineFacet((host) => { + host.serve(History, (_, self) => { + const edits = self.table(definition); + return { + record: async ({ id, at }, ctx) => { + // A deployment's rows match its own version (an example shortcut for two versions in one file). + await self.tx(async (tx) => tx.put(edits, row(id, at) as never), ctx); + return null; + }, + dump: async (_, ctx) => { + const lines: string[] = []; + for await (const row of edits.rows(query(definition), ctx)) lines.push(JSON.stringify(row)); + return lines; + }, + }; + }); + }); +} + +/** A host on the store at `path` serving `facet`, run until `work` is done. */ +async function deployment( + path: string, + facet: ReturnType, + work: (history: Reference) => Promise, +) { + const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context); + const host = createHost({ store, onError: (error) => console.error("background error:", error) }); + // Objects with empty params activate at load: the migration runs before `load` resolves. + await host.load([{ name: "@example/history", facet }], context); + await work(await host.get(History, {}, context)); + await host.close(context); + await store.close(context); +} + +const directory = await mkdtemp(join(tmpdir(), "chord-example-")); +const path = join(directory, "history.db"); +try { + await deployment( + path, + facetOf(EditsV1, (id, at) => ({ id, file: `src/${id}.ts`, at })), + async (history) => { + for (let i = 0; i < 3; i++) await history.record({ id: `e${i}`, at: i }, context); + console.log("version 1:", await history.dump({}, context)); + }, + ); + // The next deployment declares version 3: the store goes through 2 (rename, add, rows) and 3 (drop, index). + await deployment( + path, + facetOf(EditsV3, (id, at) => ({ id, path: `lib/${id}.ts`, at })), + async (history) => { + await history.record({ id: "e9", at: 9 }, context); + console.log("version 3:", await history.dump({}, context)); + }, + ); +} finally { + await rm(directory, { recursive: true, force: true }); +} diff --git a/packages/chord/test/next/examples/23-keyed-watches.ts b/packages/chord/test/next/examples/23-keyed-watches.ts new file mode 100644 index 000000000..0a13e62c1 --- /dev/null +++ b/packages/chord/test/next/examples/23-keyed-watches.ts @@ -0,0 +1,84 @@ +// Keyed watches and `at(key)`: a keyed document is watched per key; `handle.at(key)` in the factory makes one key a +// watchable (a state member here) that reads `initial()` while absent. A retirement arrives as `initial()` with +// `retired: true`, so a watcher can tell "deleted" from "changed back". +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/23-keyed-watches.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type KeyedHandle } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; + +const context = BACKGROUND_CONTEXT; + +type Note = { body: string }; + +const Notes = defineState("notes", { + schema: obj({ body: str() }), + version: 1, + initial: (): Note => ({ body: "" }), + keyed: true, +}); + +const Notebook = local("example.notebook", { + version: 1, + params: {}, + members: { + // The note "pinned" as a state member: `initial()` while absent. + pinned: obj({ body: str() }), + write: method(obj({ id: str(), body: str() })), + remove: method(str()), + }, +}); + +// The example's own window into the object: its keyed handle, to watch other keys from outside. +let notes: KeyedHandle | undefined; + +const facet = defineFacet((host) => { + host.serve(Notebook, (_, self) => { + notes = self.state(Notes); + const handle = notes; + return { + // Pinned while this attempt lives, and loaded before the object goes live. + pinned: handle.at("pinned"), + write: async ({ id, body }, ctx) => { + await handle.change( + id, + (draft) => { + draft.body = body; + }, + ctx, + ); + return null; + }, + remove: async (id, ctx) => { + await handle.retire(id, ctx); + return null; + }, + }; + }); +}); + +const host = createHost({ onError: (error) => console.error("background error:", error) }); +await host.load([{ name: "@example/notebook", facet }], context); +const notebook = await host.get(Notebook, {}, context); + +// A watch of the state member: every value call carries the value, its ops and the retired flag. +const pinned = await notebook.pinned.watch(context); +pinned.start(async (call) => { + if (call.kind === "value") console.log("pinned:", call.value, call.retired ? "(retired)" : ""); +}); + +await notebook.write({ id: "pinned", body: "buy milk" }, context); +// A watch of one key; an absent key gives `undefined` (watch it once it exists, or pin it with `at`). +console.log("watch of an absent key:", await notes!.watch("later", context)); +await notebook.write({ id: "later", body: "call Anna" }, context); +const later = (await notes!.watch("later", context))!; +later.start(async (call) => { + if (call.kind === "value") console.log("later:", call.value, call.retired ? "(retired)" : ""); +}); +await notebook.remove("later", context); +await notebook.remove("pinned", context); + +// Frames are delivered after the commit, asynchronously: let them arrive before closing. +await new Promise((resolve) => setTimeout(resolve, 10)); +await host.close(context); diff --git a/packages/chord/test/next/examples/24-keyed-members.ts b/packages/chord/test/next/examples/24-keyed-members.ts new file mode 100644 index 000000000..37f13a08d --- /dev/null +++ b/packages/chord/test/next/examples/24-keyed-members.ts @@ -0,0 +1,64 @@ +// Keyed state members: a `local` contract exposes a keyed document set as one member, `keyed(schema)`. A consumer on +// the same host pins one key with `ref.member.at(key)` in its factory (a watchable, part of its own hydration), or +// reads and watches any key with `ref.member.get/watch(key, ctx)`. One source per key: consumers share it. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/24-keyed-members.ts +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { keyed, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; + +const context = BACKGROUND_CONTEXT; + +const Profile = obj({ name: str(), status: str() }); + +const Profiles = defineState("profiles", { + schema: Profile, + version: 1, + initial: () => ({ name: "", status: "offline" }), + keyed: true, +}); + +// The directory: every user's profile, one keyed member. +const Directory = local("example.directory", { + version: 1, + params: {}, + members: { profiles: keyed(Profile), set: method(obj({ user: str(), name: str(), status: str() })) }, +}); + +// A badge per user: its own object, showing one profile. +const Badge = local("example.badge", { version: 1, params: { user: str() }, members: { label: str() } }); + +const facet = defineFacet((host) => { + host.serve(Directory, (_, self) => { + const profiles = self.state(Profiles); + return { + // The keyed handle is the member as it is. + profiles, + set: async ({ user, name, status }, ctx) => { + await profiles.change(user, (draft) => Object.assign(draft, { name, status }), ctx); + return null; + }, + }; + }); + host.serve(Badge, ({ user }, self) => { + // Pinned for this attempt; the badge goes live once the profile is loaded. + const profile = self.get(Directory, {}).profiles.at(user); + return { label: self.derive([profile], ([p]) => `${p.name || user} (${p.status})`) }; + }); +}); + +const host = createHost({ onError: (error) => console.error("background error:", error) }); +await host.load([{ name: "@example/directory", facet }], context); +const directory = await host.get(Directory, {}, context); +await directory.set({ user: "ana", name: "Ana", status: "online" }, context); + +const badge = await host.get(Badge, { user: "ana" }, context); +console.log("badge:", badge.label.value); +await directory.set({ user: "ana", name: "Ana", status: "away" }, context); +console.log("badge after a change:", badge.label.value); +// Any key, read where it is needed. +console.log("ref.profiles.get(bo):", await directory.profiles.get("bo", context)); + +badge.release(); +await host.close(context); diff --git a/packages/chord/test/next/examples/25-export-import.ts b/packages/chord/test/next/examples/25-export-import.ts new file mode 100644 index 000000000..f8dd8a812 --- /dev/null +++ b/packages/chord/test/next/examples/25-export-import.ts @@ -0,0 +1,102 @@ +// Export and import: `store.export(ctx)` yields the store's state as archive parts (lossless: tables with their rows +// and versions, the live documents), holding the commit queue while it runs; `encodeJsonl` turns them into JSONL. +// `importStore` checks an archive part by part and makes a new store from it in one backend transaction. +// Run from packages/chord: +// node --conditions=source --experimental-strip-types test/next/examples/25-export-import.ts +import { createReadStream, createWriteStream } from "node:fs"; +import { mkdtemp, rm, stat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { pipeline } from "node:stream/promises"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Reference } from "../../../src/next/host/index.ts"; +import { + decodeJsonl, + defineTable, + encodeJsonl, + importStore, + openStore, + type Store, + sqliteBackend, +} from "../../../src/next/store/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const context = BACKGROUND_CONTEXT; + +const Visits = defineTable("visits", { version: 1, columns: { page: str(), count: int() }, key: ["page"] }); +const Title = defineState("title", { + schema: obj({ text: str() }), + version: 1, + initial: () => ({ text: "untitled" }), + persist: true, + checkpointWhen: () => false, +}); + +const Site = local("example.site", { + version: 1, + params: {}, + members: { + title: obj({ text: str() }), + visit: method(str(), int()), + rename: method(str()), + }, +}); + +const facet = defineFacet((host) => { + host.serve(Site, (_, self) => { + const visits = self.table(Visits); + const title = self.state(Title); + return { + title, + visit: async (page, ctx) => + self.tx(async (tx) => { + const count = ((await tx.get(visits, { page }))?.count ?? 0) + 1; + tx.put(visits, { page, count }); + return count; + }, ctx), + rename: async (text, ctx) => { + await title.change((draft) => { + draft.text = text; + }, ctx); + return null; + }, + }; + }); +}); + +/** A host on `store` serving the site, run until `work` is done. */ +async function serve(store: Store, work: (site: Reference) => Promise) { + const host = createHost({ store, onError: (error) => console.error("background error:", error) }); + await host.load([{ name: "@example/site", facet }], context); + await work(await host.get(Site, {}, context)); + await host.close(context); +} + +const directory = await mkdtemp(join(tmpdir(), "chord-example-")); +try { + const original = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(directory, "a.db"))), context); + await serve(original, async (site) => { + await site.rename("Chord", context); + await site.visit("/", context); + await site.visit("/", context); + await site.visit("/docs", context); + }); + // Export to a local file (then upload it, if needed): the queue is held only while the parts are read. + const archive = join(directory, "archive.jsonl"); + await pipeline(encodeJsonl(original.export(context)), createWriteStream(archive)); + console.log("archive:", (await stat(archive)).size, "bytes at seq", original.lastSeq); + await original.close(context); + + // Import into an empty backend: every part is checked; a refused archive leaves the backend empty. + const parts = decodeJsonl(createReadStream(archive)); + const copy = await importStore(sqliteBackend(await openNodeSqliteDatabase(join(directory, "b.db"))), parts, context); + console.log("imported at seq", copy.lastSeq); + await serve(copy, async (site) => { + console.log("title:", site.title.value.text, "- next visit of /:", await site.visit("/", context)); + }); + await copy.close(context); +} finally { + await rm(directory, { recursive: true, force: true }); +} diff --git a/packages/chord/test/next/host/activity.test.ts b/packages/chord/test/next/host/activity.test.ts new file mode 100644 index 000000000..405089d6d --- /dev/null +++ b/packages/chord/test/next/host/activity.test.ts @@ -0,0 +1,146 @@ +// Tests of \`host.activity\` (§3, R26): the counts per kind, one publication per microtask turn with changes, a slow +// watcher, derives over it, nothing scheduled while nobody follows it, lingering. + +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { bool, int, local, method, remote } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { fakeTimers, flushed, framesOf, linked, loaded, Notes, serveNotes, transported } from "../support.ts"; + +const Kept = remote("app.kept", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + +describe("host.activity", () => { + it("counts holds arriving over links and served here, not this host's own references", async () => { + const { server, client } = await linked((h) => void serveNotes(h)); + expect(server.activity.value).toEqual({ references: 0, calls: 0, jobs: 0, lingering: 0 }); + const notes = await client.get(Notes, { session: "S1" }, context); + await server.get(Notes, { session: "S2" }, context); + expect(server.activity.value.references).toBe(1); + notes.release(); + await transported(); + expect(server.activity.value.references).toBe(0); + }); + + it("counts method calls running on this host, local and remote, until they settle", async () => { + const { server, client } = await linked((h) => void serveNotes(h)); + const remoteRef = await client.get(Notes, { session: "S1" }, context); + const localRef = await server.get(Notes, { session: "S1" }, context); + const calls = [remoteRef.wait(context), localRef.wait(context)]; + await transported(); + expect(server.activity.value.calls).toBe(2); + remoteRef.release(); + localRef.release(); + await Promise.allSettled(calls); + await transported(); + expect(server.activity.value.calls).toBe(0); + }); + + it("publishes once per microtask turn with changes; a slow watcher gets the newest counts", async () => { + const Counter = local("app.counter", { version: 1, params: {}, members: { bump: method() } }); + const { host } = await loaded((h) => h.serve(Counter, () => ({ bump: async () => null }))); + const counter = await host.get(Counter, {}, context); + const watch = await host.activity.watch(context); + const frames = framesOf(watch); + // Ten calls started in one turn: ten changes, one publication; they settle in a later turn: one more. + const bumps = Array.from({ length: 10 }, () => counter.bump(context)); + await Promise.all(bumps); + await flushed(); + const counts = (calls: number) => ({ references: 0, calls, jobs: 0, lingering: 0 }); + expect(frames).toEqual([ + [counts(10), [["r", counts(10)]]], + [counts(0), [["r", counts(0)]]], + ]); + }); + + it("gives a slow watcher the newest counts once it is free: its pending frames fold", async () => { + const Counter = local("app.counter", { version: 1, params: {}, members: { bump: method() } }); + const { host } = await loaded((h) => h.serve(Counter, () => ({ bump: async () => null }))); + const counter = await host.get(Counter, {}, context); + const watch = await host.activity.watch(context); + const gate = Promise.withResolvers(); + const seen: unknown[] = []; + watch.start(async (call) => { + if (call.kind !== "value") return; + seen.push(call.value); + if (seen.length === 1) await gate.promise; + }); + // Three more publications while the watcher is busy with the first. + for (let i = 0; i < 2; i++) { + await counter.bump(context); + await flushed(); + } + gate.resolve(); + await flushed(); + const counts = (calls: number) => ({ references: 0, calls, jobs: 0, lingering: 0 }); + expect(seen).toEqual([counts(1), counts(0)]); + }); + + it("schedules nothing while nobody watches or derives from it; a later watch reads the counts then", async () => { + const Counter = local("app.counter", { version: 1, params: {}, members: { bump: method(), wait: method() } }); + const gate = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Counter, () => ({ bump: async () => null, wait: () => gate.promise })), + ); + const counter = await host.get(Counter, {}, context); + const scheduled = vi.spyOn(globalThis, "queueMicrotask"); + try { + await counter.bump(context); + const waiting = counter.wait(context); + await flushed(); + expect(scheduled).not.toHaveBeenCalled(); + const watch = await host.activity.watch(context); + expect(watch.value).toMatchObject({ calls: 1 }); + gate.resolve(null); + await waiting; + expect(scheduled).toHaveBeenCalledTimes(1); + } finally { + scheduled.mockRestore(); + } + }); + + it("feeds derives (base code holding the host): they recompute in the publishing batch", async () => { + const Busy = local("app.busy", { version: 1, params: {}, members: { busy: bool(), work: method() } }); + const host = createHost({ onError: () => {} }); + const facet = defineFacet((h) => + h.serve(Busy, (_, self) => ({ + busy: self.derive([host.activity], ([{ calls }]) => calls > 0), + work: async () => null, + })), + ); + await host.load([{ name: "@test/core", facet }], context); + const busy = await host.get(Busy, {}, context); + const values = framesOf(await busy.busy.watch(context)); + await busy.work(context); + await flushed(); + expect(values.map(([value]) => value)).toEqual([true, false]); + expect(busy.busy.value).toBe(false); + }); + + describe("lingering", () => { + fakeTimers(); + + it("counts live objects kept by their type's linger after their last reference, until they end", async () => { + const { server, client } = await linked((h) => + h.serve(Kept, () => ({ variant: "v", members: { n: constant(1) } }), { linger: 1000 }), + ); + const kept = await client.get(Kept, {}, context); + kept.release(); + await transported(); + expect(server.activity.value).toMatchObject({ references: 0, lingering: 1 }); + // A new reference within the linger takes it back. + const again = await client.get(Kept, {}, context); + expect(server.activity.value).toMatchObject({ references: 1, lingering: 0 }); + again.release(); + await transported(); + expect(server.activity.value.lingering).toBe(1); + await vi.advanceTimersByTimeAsync(1000); + expect(server.activity.value.lingering).toBe(0); + }); + }); + + it("counts no jobs before slice 5", async () => { + const { host } = await loaded(() => {}); + expect(host.activity.value.jobs).toBe(0); + }); +}); diff --git a/packages/chord/test/next/host/attempt.test.ts b/packages/chord/test/next/host/attempt.test.ts new file mode 100644 index 000000000..73983ae0a --- /dev/null +++ b/packages/chord/test/next/host/attempt.test.ts @@ -0,0 +1,848 @@ +import { createHook } from "node:async_hooks"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, opt, remote, str } from "../../../src/next/contracts/index.ts"; +import { broken, isChordError } from "../../../src/next/data/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { + createHost, + defineFacet, + type HostError, + type Reference, + type Self, + type StoreSelf, +} from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import { evolveTable } from "../../../src/next/store/table-evolve.ts"; +import { TableHandle } from "../../../src/next/store/table-handle.ts"; +import { prepareTables } from "../../../src/next/store/table-migration.ts"; +import { failures, fakeTimers, flushed, loaded, memoryStore, metered, rejection, tableOwner } from "../support.ts"; + +const List = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); +const Page = local("app.page", { version: 1, params: { id: int() }, members: { label: str(), slow: method() } }); + +describe("failure kinds", () => { + const Broken = local("app.broken", { version: 1, params: { id: int() }, members: { label: str() } }); + const Bad = defineState("bad", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: "x" }) as never }); + + it.each<[string, (self: Self) => unknown, string]>([ + [ + "a throwing factory", + () => { + throw new Error("factory threw"); + }, + "app.broken(1): factory threw", + ], + ["a missing member", () => ({}), "app.broken(1): label: a state member is a JSON watchable"], + ["an unknown member", () => ({ label: constant(""), extra: 1 }), "app.broken(1): extra: not a member"], + [ + "a native watchable as state (Q14)", + () => ({ label: { value: "x", watch: async () => ({}) } }), + "app.broken(1): label: a state member is a JSON watchable", + ], + [ + "a throwing first derive", + (self) => ({ + label: self.derive([], () => { + throw new Error("derive threw"); + }), + }), + "app.broken(1): derive threw", + ], + [ + "an initial() that fails its schema", + (self) => ({ label: self.derive([self.state(Bad)], () => "") }), + "app.broken(1): bad: initial() does not match the schema", + ], + [ + "an open that throws broken(error)", + (self) => { + self.open( + async () => { + throw broken(new Error("no remote configured")); + }, + () => {}, + ); + return { label: constant("") }; + }, + "app.broken(1): no remote configured", + ], + ])( + "%s fails broken: reported, the get rejects at once with a ChordError, nothing is left", + async (_, factory, message) => { + const served = vi.fn((_: { id: number }, self: Self) => factory(self) as never); + const { host, errors } = await loaded((h) => h.serve(Broken, served)); + const error = await host.get(Broken, { id: 1 }, context).catch((error: unknown) => error); + expect(isChordError(error) && error.message).toBe(message); + await expect(host.get(Broken, { id: 1 }, context)).rejects.toThrow(message); + expect(served).toHaveBeenCalledTimes(2); + expect(failures(errors)).toEqual(["app.broken(1): broken", "app.broken(1): broken"]); + // One report per failure: a derive throwing while its attempt activates reports no derive_failed. + expect(errors.map(({ kind }) => kind)).toEqual(["activation_failed", "activation_failed"]); + }, + ); + + it("an open that throws fails transient: reported, the get waits through retries", async () => { + const { host, errors } = await loaded((h) => + h.serve(Broken, (_, self) => { + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + return { label: constant("") }; + }), + ); + expect(await rejection(host.get(Broken, { id: 1 }, context, { readyTimeout: 10 }))).toEqual({ + code: "unavailable", + message: "app.broken(1): offline", + }); + expect(errors).toMatchObject([ + { + kind: "activation_failed", + object: "app.broken(1)", + failure: { kind: "transient", error: { code: "unavailable" } }, + }, + ]); + await host.close(context); + }); + + describe("an attempt not live within readyTimeout", () => { + fakeTimers(); + + it("is abandoned and fails transient: the opener's context aborted, what opened closed", async () => { + let opener: Context | undefined; + const closes: string[] = []; + const { host, errors } = await loaded( + (h) => + h.serve(Broken, (_, self) => { + self.open( + async () => "a", + (value, reason) => void closes.push(`${value} ${reason.kind}`), + ); + self.open( + async (ctx) => { + opener = ctx; + return new Promise(() => {}); + }, + () => {}, + ); + return { label: constant("") }; + }), + { readyTimeout: 100 }, + ); + const got = rejection(host.get(Broken, { id: 1 }, context, { readyTimeout: 150 })); + await vi.advanceTimersByTimeAsync(100); + expect([opener?.abortSignal?.aborted, closes]).toEqual([true, ["a end"]]); + const notLive = { code: "unavailable", message: "app.broken(1): did not become live within 100 ms" }; + expect(errors).toMatchObject([{ kind: "activation_failed", failure: { kind: "transient", error: notLive } }]); + await vi.advanceTimersByTimeAsync(50); + expect(await got).toEqual(notLive); + }); + }); + + describe("an attempt waiting for its previous instance's teardown", () => { + fakeTimers(); + + it("is bounded by readyTimeout too: abandoned transient without running its factory, retried after it", async () => { + const closing = Promise.withResolvers(); + let factories = 0; + const { host, errors } = await loaded( + (h) => + h.serve(Page, (_, self) => { + factories++; + self.open( + async () => "native", + () => closing.promise, + ); + return { label: constant("p"), slow: async () => null }; + }), + { readyTimeout: 100, closeTimeout: 10_000 }, + ); + (await host.get(Page, { id: 1 }, context)).release(); + const next = host.hold(Page, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(100); + expect(factories).toBe(1); + expect(errors).toMatchObject([ + { + kind: "activation_failed", + failure: { kind: "transient", error: { message: "app.page(1): did not become live within 100 ms" } }, + }, + ]); + closing.resolve(); + await vi.advanceTimersByTimeAsync(1_000); + expect([factories, next.state.value]).toEqual([2, { kind: "live" }]); + }); + }); + + it("a keyed document's initial() failing its schema fails that transaction, not the object", async () => { + const Keys = defineState("keys", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: "x" }) as never, + keyed: true, + }); + const { host, errors } = await loaded((h) => + h.serve(Page, (_, self) => { + const keys = self.state(Keys); + return { + label: constant("live"), + slow: async (ctx) => { + await keys.change("a", () => {}, ctx); + return null; + }, + }; + }), + ); + const page = await host.get(Page, { id: 1 }, context); + await expect(page.slow(context)).rejects.toThrow("keys: initial() does not match the schema"); + expect([page.label.value, errors]).toEqual(["live", []]); + }); + + it("releases the references it acquired", async () => { + const pages = vi.fn((_: { id: number }) => ({ label: constant("p"), slow: async () => null })); + const { host } = await loaded((h) => { + h.serve(Broken, (_, self) => { + const page = self.get(Page, { id: 7 }); + const label = self.derive([page.label], (): string => { + throw new Error("late"); + }); + return { label }; + }); + h.serve(Page, pages); + }); + await expect(host.get(Broken, { id: 1 }, context)).rejects.toThrow("late"); + expect(pages).toHaveBeenCalledTimes(1); + await host.get(Page, { id: 7 }, context); + expect(pages).toHaveBeenCalledTimes(2); + }); +}); + +describe("ending", () => { + it("rejects their callers at once; the handler runs on with its context aborted and cannot commit", async () => { + const started = Promise.withResolvers(); + const proceed = Promise.withResolvers(); + const finished = Promise.withResolvers<{ aborted?: boolean; error?: unknown }>(); + const { host } = await loaded((h) => + h.serve(Page, (_, self) => { + const list = self.state(List); + return { + label: constant(""), + slow: async (ctx) => { + started.resolve(); + await proceed.promise; + const tx = self.tx(async (tx) => (await tx.draft(list)).items.push("x"), context); + finished.resolve({ aborted: ctx.abortSignal?.aborted, error: await rejection(tx) }); + return null; + }, + }; + }), + ); + const page = await host.get(Page, { id: 1 }, context); + const call = page.slow(context); + await started.promise; + page.release(); + const released = { code: "released", reason: "ended", message: "app.page(1): ended" }; + expect(await rejection(call)).toEqual(released); + proceed.resolve(); + expect(await finished.promise).toEqual({ aborted: true, error: released }); + }); + + it("rejects a call whose caller's context is already aborted, before the handler runs", async () => { + const slow = vi.fn(async () => null); + const { host } = await loaded((h) => h.serve(Page, () => ({ label: constant(""), slow }))); + const page = await host.get(Page, { id: 1 }, context); + const { context: cancelled, cancel } = withCancel(context); + cancel(new Error("gone")); + await expect(page.slow(cancelled)).rejects.toThrow("gone"); + expect(slow).not.toHaveBeenCalled(); + }); + + it("waits for the running handlers' own promises (the drain) before the natives close", async () => { + const events: string[] = []; + const started = Promise.withResolvers(); + const proceed = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Page, (_, self) => { + self.open( + async () => "native", + (value, reason) => void events.push(`close ${value} ${reason.kind}`), + ); + return { + label: constant(""), + slow: async () => { + started.resolve(); + await proceed.promise; + events.push("handler settled"); + return null; + }, + }; + }), + ); + const page = await host.get(Page, { id: 1 }, context); + const call = page.slow(context); + await started.promise; + page.release(); + await expect(call).rejects.toThrow("app.page(1): ended"); + await flushed(); + expect(events).toEqual([]); + proceed.resolve(); + await flushed(); + expect(events).toEqual(["handler settled", "close native end"]); + }); + + it("drains a handler that releases its object's last reference before it returns its promise", async () => { + const events: string[] = []; + const proceed = Promise.withResolvers(); + let page: Reference | undefined; + const { host } = await loaded((h) => + h.serve(Page, (_, self) => { + self.open( + async () => null, + () => void events.push("closed"), + ); + return { + label: constant(""), + // Not async: the release runs before the handler has a promise to return. + slow: () => { + page!.release(); + return proceed.promise.then(() => { + events.push("handler settled"); + return null; + }); + }, + }; + }), + ); + page = await host.get(Page, { id: 1 }, context); + const call = page.slow(context); + await expect(call).rejects.toThrow("app.page(1): ended"); + await flushed(); + expect(events).toEqual([]); + proceed.resolve(); + await flushed(); + expect(events).toEqual(["handler settled", "closed"]); + }); + + it("refuses transactions at once; documents stay readable until the closes are done", async () => { + let list: SingletonHandle<{ items: string[] }> | undefined; + const seen = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Page, (_, self) => { + const handle = self.state(List); + list = handle; + self.open( + async () => null, + async () => seen.resolve([handle.value, await rejection(handle.change(() => {}, context))]), + ); + return { label: constant(""), slow: async () => null }; + }), + ); + (await host.get(Page, { id: 1 }, context)).release(); + expect(await seen.promise).toEqual([ + { items: [] }, + { code: "released", reason: "ended", message: "app.page(1): ended" }, + ]); + await flushed(); + expect(() => list!.value).toThrow("app.page(1): ended"); + }); + + describe("the drain", () => { + fakeTimers(); + + it("is bounded by closeTimeout: a handler still running then sees its natives close, which is reported", async () => { + const closed = vi.fn(); + const { host, errors } = await loaded( + (h) => + h.serve(Page, (_, self) => { + self.open(async () => null, closed); + return { label: constant(""), slow: () => new Promise(() => {}) }; + }), + { closeTimeout: 1000 }, + ); + const page = await host.get(Page, { id: 1 }, context); + void page.slow(context).catch(() => {}); + page.release(); + await vi.advanceTimersByTimeAsync(999); + expect(closed).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(closed).toHaveBeenCalledTimes(1); + expect(errors).toEqual([{ kind: "drain_timeout", object: "app.page(1)" }]); + }); + + it("a handler stuck past it no longer counts as a call nor holds its ctx.get references", async () => { + const Item = local("app.item", { version: 1, params: { id: int() }, members: { label: str() } }); + const got = Promise.withResolvers(); + const { host } = await loaded( + (h) => { + h.serve(Item, () => ({ label: constant("item") })); + h.serve(Page, () => ({ + label: constant(""), + slow: async (ctx) => { + await ctx.get(Item, { id: 7 }); + got.resolve(); + return new Promise(() => {}); + }, + })); + }, + { closeTimeout: 1000 }, + ); + const items = host.references(Item); + const page = await host.get(Page, { id: 1 }, context); + void page.slow(context).catch(() => {}); + await got.promise; + expect(host.activity.value.calls).toBe(1); + expect(items.value).toHaveLength(1); + page.release(); + await vi.advanceTimersByTimeAsync(1000); + expect(host.activity.value.calls).toBe(0); + expect(items.value).toEqual([]); + }); + }); +}); + +describe("declarations", () => { + it("are valid only while the factory runs", async () => { + let later: Self | undefined; + const { host } = await loaded((h) => + h.serve(Page, (_, self) => { + later = self; + return { label: constant(""), slow: async () => null }; + }), + ); + await host.get(Page, { id: 1 }, context); + expect(() => later!.state(List)).toThrow("app.page(1): self.state is valid only in the factory"); + expect(() => later!.derive([], () => 1)).toThrow("self.derive is valid only in the factory"); + expect(() => later!.get(Page, { id: 2 })).toThrow("self.get is valid only in the factory"); + expect(() => + later!.open( + async () => null, + () => {}, + ), + ).toThrow("self.open is valid only in the factory"); + }); + + it("refuse a document definition declared twice", async () => { + const Twice = local("app.twice", { version: 1, params: { n: int() }, members: {} }); + const { host } = await loaded((h) => + h.serve(Twice, (_, self) => { + [self.state(List), self.state(List)]; + return {}; + }), + ); + await expect(host.get(Twice, { n: 2 }, context)).rejects.toThrow('app.twice(2): state "list" is declared twice'); + }); +}); + +describe("calls", () => { + it("check arguments with L1 and pass arguments and results by reference", async () => { + const seen: unknown[] = []; + const result = { items: [] }; + const Echo = local("app.echo", { + version: 1, + params: {}, + members: { echo: method(obj({ text: str() }), obj({ items: arr(str()) })), none: method() }, + }); + const { host } = await loaded((h) => + h.serve(Echo, () => ({ + echo: async (args) => { + seen.push(args); + return result; + }, + // A method without arguments gets only the context. + none: async (...args) => { + seen.push(args.length); + return null; + }, + })), + ); + const echo = await host.get(Echo, {}, context); + const args = { text: "t" }; + expect(await echo.echo(args, context)).toBe(result); + expect(await echo.none(context)).toBeNull(); + expect(seen).toEqual([args, 1]); + expect(seen[0]).toBe(args); + // @ts-expect-error: an unknown field + expect(await rejection(echo.echo({ text: "t", extra: 1 }, context))).toMatchObject({ + code: "invalid", + reason: "args", + }); + }); + + it("take a plain value or a synchronous throw from a handler that is no async function", async () => { + const Plain = local("app.plain", { + version: 1, + params: {}, + members: { value: method(undefined, int()), fail: method() }, + }); + const { host } = await loaded((h) => + // Plugin code is bundled without type checks. + h.serve(Plain, () => ({ + value: (() => 7) as never, + fail: (() => { + throw new Error("sync throw"); + }) as never, + })), + ); + const plain = await host.get(Plain, {}, context); + expect(await plain.value(context)).toBe(7); + await expect(plain.fail(context)).rejects.toThrow("sync throw"); + }); + + it("cost one validation per call and one compilation per method", async () => { + const telemetry = await metered(); + // A fresh declaration: compiled checks are cached per contract. + const Fresh = local("app.store", { version: 1, params: {}, members: { add: method(obj({ text: str() })) } }); + const { host } = await loaded((h) => h.serve(Fresh, () => ({ add: async () => null }))); + const store = await host.get(Fresh, {}, context); + await store.add({ text: "a" }, telemetry.context); + await store.add({ text: "b" }, telemetry.context); + expect(telemetry.counts()).toEqual({ "chord.data.compilations": 1, "chord.data.validations": 2 }); + }); +}); + +describe("the call path's cost", () => { + it("makes at most 19 promises per local call (a guard: the drain and the cleanups share one reaction)", async () => { + const Touch = local("app.touch", { version: 1, params: {}, members: { touch: method() } }); + const { host } = await loaded((h) => h.serve(Touch, () => ({ touch: async () => null }))); + const ref = await host.get(Touch, {}, context); + await ref.touch(context); + let promises = 0; + // Every `then` makes a promise: one more reaction per call shows here (HEAD before the cleanups made 20). + const hook = createHook({ + init: (_id, type) => { + if (type === "PROMISE") promises++; + }, + }); + hook.enable(); + try { + for (let call = 0; call < 100; call++) await ref.touch(context); + } finally { + hook.disable(); + } + expect(promises / 100).toBeLessThanOrEqual(19); + await host.close(context); + }); +}); + +describe("remote variants", () => { + const Two = remote("app.two", { + version: 1, + params: {}, + variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { n: int() } } }, + }); + const owner = { kind: "owner" } as const; + + /** A host acting as `owner` whose factory for `Two` returns `implementation`. */ + async function two(implementation: unknown) { + const errors: HostError[] = []; + const host = createHost({ principal: owner, onError: (error) => errors.push(error) }); + const facet = defineFacet((h) => h.serve(Two, () => implementation as never)); + await host.load([{ name: "@test/app", facet }], context); + return { host, errors }; + } + + it("the factory chooses one; the reference narrows on its variant", async () => { + const { host } = await two({ variant: "view", members: { n: constant(1) } }); + const ref = await host.get(Two, {}, context); + expect(ref.variant).toBe("view"); + if (ref.variant === "view") expect(ref.n.value).toBe(1); + }); + + it.each([ + ["an unknown variant", { variant: "other", members: {} }, 'variant "other": not a variant of app.two'], + ["a missing member", { variant: "full", members: { n: constant(1) } }, "bump: a method is a function"], + [ + "an extra member", + { variant: "view", members: { n: constant(1), bump: async () => null } }, + "bump: not a member", + ], + ])("%s fails the activation broken", async (_, implementation, message) => { + const { host, errors } = await two(implementation); + expect(await rejection(host.get(Two, {}, context))).toEqual({ + code: "unavailable", + message: `app.two: ${message}`, + }); + expect(failures(errors)).toEqual(["app.two: broken"]); + }); + + it("a member outside the bound variant fails invalid/member", async () => { + const { host } = await two({ variant: "view", members: { n: constant(1) } }); + const ref = (await host.get(Two, {}, context)) as unknown as { bump(ctx: Context): Promise }; + expect(await rejection(ref.bump(context))).toEqual({ + code: "invalid", + reason: "member", + message: 'app.two (variant view) has no method "bump"', + }); + }); +}); + +describe("tables", () => { + const Edits = defineTable("edits", { + version: 1, + columns: { id: str(), at: int() }, + key: ["id"], + indexes: { by_at: ["at"] }, + }); + const Store = local("app.store", { version: 1, params: {}, members: { n: int() } }); + + it("a fresh table is created at activation at its version, with its indexes, in one commit", async () => { + const store = await memoryStore({ outbox: true }); + const { host } = await loaded( + (h) => + h.serve(Store, (_, self) => { + self.table(Edits); + return { n: constant(1) }; + }), + { + store, + }, + ); + await host.get(Store, {}, context); + expect(store.table("client:app.store:edits")).toMatchObject({ + version: 1, + state: "ready", + indexes: { by_at: ["at"] }, + }); + expect((await store.outbox!.read(0, 10, context)).map(({ writes }) => writes.map(({ kind }) => kind))).toEqual([ + ["table", "index"], + ]); + await host.close(context); + const again = await loaded( + (h) => + h.serve(Store, (_, self) => { + self.table(Edits); + return { n: constant(1) }; + }), + { + store, + }, + ); + await again.host.get(Store, {}, context); + expect(store.lastSeq).toBe(1); + await again.host.close(context); + await store.close(context); + }); + + it.each<[string, (self: StoreSelf) => void, string]>([ + [ + "declared twice", + (self) => { + self.table(Edits); + self.table(Edits); + }, + 'table "edits" is declared twice', + ], + [ + "another version recorded", + (self) => self.table(defineTable("edits", { version: 2, columns: { id: str() }, key: ["id"] })), + "older than the history's base 2", + ], + [ + "another definition at the same version", + (self) => self.table(defineTable("edits", { version: 1, columns: { id: str(), at: int() }, key: ["id"] })), + "recorded with another definition", + ], + ])("fails the activation broken: %s", async (_, declare, message) => { + const store = await memoryStore(); + const first = await loaded( + (h) => + h.serve(Store, (_, self) => { + self.table(Edits); + return { n: constant(1) }; + }), + { + store, + }, + ); + await first.host.get(Store, {}, context); + await first.host.close(context); + const { host } = await loaded( + (h) => + h.serve(Store, (_, self) => { + declare(self); + return { n: constant(1) }; + }), + { store }, + ); + expect(await rejection(host.get(Store, {}, context))).toMatchObject({ + message: expect.stringContaining(message), + }); + await host.close(context); + await store.close(context); + }); + + it("needs a local type with empty params (P3) and a host with a store", async () => { + const { host } = await loaded((h) => { + h.serve(Page, (_, self) => { + // @ts-expect-error: only a local type with empty params has tables (P3) + const declare = () => self.table(Edits); + void declare; + (self as unknown as StoreSelf).table(Edits); + return { label: constant(""), slow: async () => null }; + }); + h.serve(Store, (_, self) => { + self.table(Edits); + return { n: constant(1) }; + }); + }); + expect(await rejection(host.get(Page, { id: 1 }, context))).toMatchObject({ + message: expect.stringContaining("persist needs a stable identity"), + }); + expect(await rejection(host.get(Store, {}, context))).toMatchObject({ + message: expect.stringContaining("persist needs a store"), + }); + await host.close(context); + }); +}); + +describe("eager migrations at activation (slice 6a §9)", () => { + fakeTimers(); + // The clock is faked, but the zero-delay timers a migration yields with between chunks run every real turn. + let pumping = true; + beforeEach(() => { + pumping = true; + const pump = (): void => { + if (!pumping) return; + vi.advanceTimersByTime(0); + setImmediate(pump); + }; + setImmediate(pump); + }); + afterEach(() => { + pumping = false; + }); + const TABLE = "client:app.store:edits"; + const OPEN = { admit: () => true, adopt: () => {} }; + const EditsV1 = defineTable("edits", { version: 1, columns: { id: str(), at: int() }, key: ["id"] }); + const Store = local("app.store", { version: 1, params: {}, members: { n: int() } }); + const Slow = local("app.slow", { version: 1, params: {}, members: { n: int() } }); + + /** A store whose edits table is at version 1 with `count` rows. */ + async function seeded(count: number): Promise { + const store = await memoryStore(); + await prepareTables([{ definition: EditsV1, table: TABLE }], store, OPEN, "app.store", context); + const staged: StagedWrite[] = []; + const changes = new TableChanges(new TableHandle(EditsV1, TABLE, store, tableOwner()), staged); + for (let i = 0; i < count; i++) changes.put({ id: `e${String(i).padStart(3, "0")}`, at: i }); + await store.commit(staged, OPEN, context); + return store; + } + + /** Waits (real turns of the event loop, the clock faked) until `done`. */ + async function until(done: () => boolean): Promise { + for (let turn = 0; turn < 1_000 && !done(); turn++) await flushed(); + expect(done()).toBe(true); + } + + /** Version 2 adding `note`, whose `rows` runs `each` per row first. */ + function evolved(each: (id: string) => void) { + return evolveTable(EditsV1, (t) => + t.add("note", opt(str())).rows((row) => { + each(row.id); + return { note: `#${row.at}` }; + }), + ); + } + + /** A native whose open never succeeds: it rejects when its context is aborted (the deadline). */ + function hanging(ctx: Context): Promise { + return new Promise((_, reject) => ctx.abortSignal!.addEventListener("abort", () => reject(new Error("aborted")))); + } + + it("pause the readyTimeout deadline and resume it with the budget left", async () => { + const store = await seeded(250); + // Each row takes 10 ms of the fake clock: 2.5 s of migration against a 1 s deadline. + const Edits = evolved(() => vi.advanceTimersByTime(10)); + const loading = loaded( + (h) => + h.serve(Store, (_, self) => { + self.table(Edits); + // The factory takes 600 ms of the budget; the native after the migration never opens. + vi.advanceTimersByTime(600); + self.open(hanging, () => {}); + return { n: constant(1) }; + }), + { store, readyTimeout: 1_000 }, + ); + await until(() => store.table(TABLE)?.version === 2 && store.table(TABLE)?.state === "ready"); + expect(Date.now()).toBe(3_100); + await vi.advanceTimersByTimeAsync(399); + await vi.advanceTimersByTimeAsync(1); + const { host, errors } = await loading; + expect(Date.now()).toBe(3_500); + expect(errors).toMatchObject([ + { + kind: "activation_failed", + failure: { kind: "transient", error: { message: "app.store: did not become live within 1000 ms" } }, + }, + ]); + await host.close(context); + await store.close(context); + }); + + it("a timer firing during a long migration: another activation's deadline keeps its bound", async () => { + const store = await seeded(250); + const Edits = evolved(() => vi.advanceTimersByTime(10)); + // Both types have empty params: the host activates both at load, the start barrier waiting for both. + const { host, errors } = await loaded( + (h) => { + h.serve(Slow, (_, self) => { + self.open(hanging, () => {}); + return { n: constant(1) }; + }); + h.serve(Store, (_, self) => { + self.table(Edits); + return { n: constant(1) }; + }); + }, + { store, readyTimeout: 1_000 }, + ); + expect(Date.now()).toBe(2_500); + expect(errors).toMatchObject([{ kind: "activation_failed", object: "app.slow", failure: { kind: "transient" } }]); + expect((await host.get(Store, {}, context)).n.value).toBe(1); + await host.close(context); + await store.close(context); + }); + + it("a throwing rows fails the object broken, the table migrating; a retry resumes after the committed chunks", async () => { + const store = await seeded(250); + let crash = true; + const patched: string[] = []; + const Edits = evolved((id) => { + if (crash && id === "e150") throw new Error("crash"); + patched.push(id); + }); + const { host, errors } = await loaded( + (h) => + h.serve(Store, (_, self) => { + self.table(Edits); + return { n: constant(1) }; + }), + { store }, + ); + const held = host.hold(Store, {}, context); + await until(() => held.state.value.kind === "failed"); + expect(held.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: { message: expect.stringContaining("crash") } }, + }); + expect(errors).toMatchObject([{ kind: "activation_failed", failure: { kind: "broken" } }]); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "migrating" }); + crash = false; + patched.length = 0; + held.retry(); + await until(() => held.state.value.kind === "live"); + expect(patched).toHaveLength(150); + expect(patched[0]).toBe("e100"); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready" }); + held.release(); + await host.close(context); + await store.close(context); + }); +}); diff --git a/packages/chord/test/next/host/backoff.test.ts b/packages/chord/test/next/host/backoff.test.ts new file mode 100644 index 000000000..f53bdd426 --- /dev/null +++ b/packages/chord/test/next/host/backoff.test.ts @@ -0,0 +1,87 @@ +import { describe, expect, it, vi } from "vitest"; +import { Backoff } from "../../../src/next/host/backoff.ts"; +import { fakeTimers } from "../support.ts"; + +/** The length of the window `failed` opens for `identity`, measured until `due` runs. */ +function window(backoff: Backoff, due: ReturnType, identity: string): number { + const calls = due.mock.calls.length; + const start = Date.now(); + backoff.failed(identity); + while (due.mock.calls.length === calls) vi.advanceTimersByTime(100); + return Date.now() - start; +} + +describe("backoff", () => { + fakeTimers(); + + it("opens a window of 1 s after the first failure, doubling up to 60 s, and calls due at its end", () => { + const due = vi.fn(); + const backoff = new Backoff(due); + const windows = Array.from({ length: 9 }, () => window(backoff, due, "a")); + expect(windows).toEqual([1000, 2000, 4000, 8000, 16000, 32000, 60000, 60000, 60000]); + expect(due.mock.calls.every(([identity]) => identity === "a")).toBe(true); + }); + + it("tells when a running window ends: a failed reference's retryAt", () => { + const backoff = new Backoff(() => {}); + vi.advanceTimersByTime(5_000); + backoff.failed("a"); + expect([backoff.until("a"), backoff.until("b")]).toEqual([6_000, undefined]); + vi.advanceTimersByTime(1_000); + expect(backoff.until("a")).toBeUndefined(); + backoff.failed("a"); + expect(backoff.until("a")).toBe(8_000); + backoff.skip("a"); + expect(backoff.until("a")).toBeUndefined(); + }); + + it("is waiting only inside a window", () => { + const backoff = new Backoff(() => {}); + backoff.failed("a"); + expect([backoff.waiting("a"), backoff.waiting("b")]).toEqual([true, false]); + vi.advanceTimersByTime(1000); + expect(backoff.waiting("a")).toBe(false); + }); + + it("skip ends a window without due and keeps its delay; delete starts over", () => { + const due = vi.fn(); + const backoff = new Backoff(due); + backoff.failed("a"); + backoff.skip("a"); + vi.advanceTimersByTime(5000); + expect([backoff.waiting("a"), due.mock.calls.length]).toEqual([false, 0]); + expect(window(backoff, due, "a")).toBe(2000); + backoff.delete("a"); + expect(window(backoff, due, "a")).toBe(1000); + }); + + it("ended keeps a running window and drops a passed one", () => { + const due = vi.fn(); + const backoff = new Backoff(due); + backoff.failed("a"); + backoff.ended("a"); + expect(backoff.waiting("a")).toBe(true); + vi.advanceTimersByTime(1000); + backoff.ended("a"); + expect(window(backoff, due, "a")).toBe(1000); + }); + + it("a failure inside a running window replaces it: one due, after the doubled delay", () => { + const due = vi.fn(); + const backoff = new Backoff(due); + backoff.failed("a"); + backoff.failed("a"); + vi.advanceTimersByTime(1999); + expect([due.mock.calls.length, vi.getTimerCount()]).toEqual([0, 1]); + vi.advanceTimersByTime(1); + expect(due).toHaveBeenCalledTimes(1); + }); + + it("clear forgets every identity with its timer", () => { + const backoff = new Backoff(() => {}); + backoff.failed("a"); + backoff.failed("b"); + backoff.clear(); + expect([backoff.waiting("a"), backoff.waiting("b"), vi.getTimerCount()]).toEqual([false, false, 0]); + }); +}); diff --git a/packages/chord/test/next/host/ctx.test.ts b/packages/chord/test/next/host/ctx.test.ts new file mode 100644 index 000000000..0d4cc480b --- /dev/null +++ b/packages/chord/test/next/host/ctx.test.ts @@ -0,0 +1,173 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withAbortSignal } from "../../../src/context/index.ts"; +import { int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { withLinkedAbortSignal } from "../../../src/next/host/ctx.ts"; +import type { ChordHost } from "../../../src/next/host/host.ts"; +import type { Ctx, Host, Reference } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { flushed, loaded, rawParent, rejection, type TestPrincipal } from "../support.ts"; + +const Item = local("app.item", { version: 1, params: { id: int() }, members: { id: int() } }); +const Desk = local("app.desk", { + version: 1, + params: {}, + members: { use: method(undefined, int()), leak: method(), late: method(), who: method(undefined, str()) }, +}); + +/** A desk whose handlers use `ctx.get`; what they got and their contexts are recorded. */ +async function desk() { + const seen: { ctx?: Ctx; item?: Reference; late?: Promise; factories: number } = { + factories: 0, + }; + const { host } = await loaded((h) => { + h.serve(Item, ({ id }) => { + seen.factories++; + return { id: constant(id) }; + }); + h.serve(Desk, () => ({ + use: async (ctx) => { + const item = await ctx.get(Item, { id: 1 }); + return item.id.value; + }, + leak: async (ctx) => { + seen.ctx = ctx; + seen.item = await ctx.get(Item, { id: 2 }); + return null; + }, + late: async (ctx) => { + seen.late = ctx.get(Item, { id: 3 }).catch((error: unknown) => error); + return null; + }, + who: async (ctx) => `${ctx.principal?.kind ?? "none"} ${ctx.incarnation}`, + })); + }); + return { host, seen, desk: await host.get(Desk, {}, context) }; +} + +describe("ctx", () => { + it("ctx.get references are released when the handler settles", async () => { + const { desk: ref, seen } = await desk(); + expect(await ref.use(context)).toBe(1); + await ref.leak(context); + expect(() => seen.item!.id.value).toThrow("app.item: released"); + // The object ended with its last reference: a new get creates it anew. + await ref.use(context); + expect(seen.factories).toBe(3); + }); + + it("ctx.get after the handler settled throws released", async () => { + const { desk: ref, seen } = await desk(); + await ref.leak(context); + expect(await rejection(seen.ctx!.get(Item, { id: 1 }))).toEqual({ + code: "released", + reason: "settled", + message: "app.item: ctx.get after its handler settled", + }); + }); + + it("ctx.get of a handler whose call was aborted checks the signal first: nothing is created", async () => { + let factories = 0; + let got: Promise | undefined; + const gate = Promise.withResolvers(); + const { host } = await loaded((h) => { + h.serve(Item, ({ id }) => { + factories++; + return { id: constant(id) }; + }); + h.serve(Desk, () => ({ + use: async (ctx) => { + await gate.promise; + got = ctx.get(Item, { id: 1 }).catch((error: unknown) => error); + return 0; + }, + leak: async () => null, + late: async () => null, + who: async () => "", + })); + }); + const ref = await host.get(Desk, {}, context); + const caller = new AbortController(); + const call = ref.use(withAbortSignal(caller.signal, context)).catch(() => {}); + await flushed(); + caller.abort(new Error("caller gave up")); + gate.resolve(); + await call; + await flushed(); + expect(await got).toEqual(new Error("caller gave up")); + expect(factories).toBe(0); + }); + + it("a ctx.get finishing after its handler settled is released at once (race 11)", async () => { + const { desk: ref, seen } = await desk(); + await ref.late(context); + expect(await seen.late).toMatchObject({ + code: "released", + reason: "settled", + message: "app.item: ctx.get after its handler settled", + }); + await flushed(); + // The late reference's object ended with it. + expect(seen.factories).toBe(1); + }); + + it("a ctx.get still waiting when its handler settles gives up: its reference is released at once", async () => { + const Far = remote("app.far", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const Starter = local("app.starter", { version: 1, params: {}, members: { start: method() } }); + let pending: Promise | undefined; + const { host } = await loaded((h) => + h.serve(Starter, () => ({ + start: async (ctx) => { + pending = ctx.get(Far, {}).catch((error: unknown) => error); + return null; + }, + })), + ); + // A host of any principal type takes a parent link. + const parent = await rawParent(host as Host); + await (await host.get(Starter, {}, context)).start(context); + const { hold } = await parent.next("hold"); + expect(await parent.next("release")).toMatchObject({ hold }); + expect(await pending).toMatchObject({ + code: "released", + reason: "settled", + message: "app.far: ctx.get after its handler settled", + }); + }); + + it("a local handler's ctx.principal is the host's, its incarnation this host's", async () => { + const { desk: ref, host } = await desk(); + expect(await ref.who(context)).toBe(`none ${(host as ChordHost).incarnation}`); + }); +}); + +describe("linked abort signals", () => { + it("abort with the caller's signal or the call's, the caller's reason first", () => { + for (const first of ["caller", "call"] as const) { + const [caller, call] = [new AbortController(), new AbortController()]; + const { context: linked } = withLinkedAbortSignal(call.signal, withAbortSignal(caller.signal, context)); + (first === "caller" ? caller : call).abort(first); + expect(linked.abortSignal?.reason).toBe(first); + } + const [caller, call] = [new AbortController(), new AbortController()]; + call.abort("call"); + caller.abort("caller"); + const { context: linked } = withLinkedAbortSignal(call.signal, withAbortSignal(caller.signal, context)); + expect(linked.abortSignal?.reason).toBe("caller"); + }); + + it("detach from both sources on release: a later abort reaches nothing", () => { + const [caller, call] = [new AbortController(), new AbortController()]; + const linked = withLinkedAbortSignal(call.signal, withAbortSignal(caller.signal, context)); + linked.release(); + caller.abort("late"); + call.abort("late"); + expect(linked.context.abortSignal?.aborted).toBe(false); + }); + + it("is the call's own signal without a caller's", () => { + const call = new AbortController(); + const linked = withLinkedAbortSignal(call.signal, context); + expect(linked.context.abortSignal).toBe(call.signal); + linked.release(); + }); +}); diff --git a/packages/chord/test/next/host/deadline.test.ts b/packages/chord/test/next/host/deadline.test.ts new file mode 100644 index 000000000..23b6e5093 --- /dev/null +++ b/packages/chord/test/next/host/deadline.test.ts @@ -0,0 +1,42 @@ +// Tests of an activation's deadline (slice 6a §9): it expires once after its budget; paused time does not count; a +// cleared deadline never expires. + +import { describe, expect, it, vi } from "vitest"; +import { Deadline } from "../../../src/next/host/deadline.ts"; +import { fakeTimers } from "../support.ts"; + +describe("deadline", () => { + fakeTimers(); + + it("expires once after its budget", () => { + const expired = vi.fn(); + new Deadline(100, expired); + vi.advanceTimersByTime(99); + expect(expired).not.toHaveBeenCalled(); + vi.advanceTimersByTime(1_000); + expect(expired).toHaveBeenCalledTimes(1); + }); + + it("paused time does not count: resumed with the budget it had left", () => { + const expired = vi.fn(); + const deadline = new Deadline(100, expired); + vi.advanceTimersByTime(60); + deadline.pause(); + vi.advanceTimersByTime(1_000); + deadline.resume(); + vi.advanceTimersByTime(39); + expect(expired).not.toHaveBeenCalled(); + vi.advanceTimersByTime(1); + expect(expired).toHaveBeenCalledTimes(1); + }); + + it("a cleared deadline never expires, also when resumed", () => { + const expired = vi.fn(); + const deadline = new Deadline(100, expired); + deadline.pause(); + deadline.clear(); + deadline.resume(); + vi.advanceTimersByTime(1_000); + expect(expired).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/chord/test/next/host/facet.test.ts b/packages/chord/test/next/host/facet.test.ts new file mode 100644 index 000000000..54fe1113d --- /dev/null +++ b/packages/chord/test/next/host/facet.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, expectTypeOf, it } from "vitest"; +import { int, local } from "../../../src/next/contracts/index.ts"; +import { defineFacet, type Facet, type FacetHost, type Host } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { loaded } from "../support.ts"; + +const Info = local("app.info", { version: 1, params: {}, members: { n: int() } }); + +describe("facets", () => { + it("declare only while their setup runs", async () => { + let later: FacetHost | undefined; + await loaded((host) => { + later = host; + }); + expect(() => later!.serve(Info, () => ({ n: constant(1) }))).toThrow( + "@test/app: host.serve is valid only in the facet's setup", + ); + }); + + it("only declare: a setup takes the host alone, which has no documents, routines or natives", () => { + expectTypeOf["setup"]>>().toEqualTypeOf<[FacetHost<{ kind: "owner" }>]>(); + expectTypeOf().not.toHaveProperty("state"); + expectTypeOf().not.toHaveProperty("on"); + expectTypeOf().not.toHaveProperty("onLive"); + }); + + it("are typed by exactly the app's principal type", () => { + type App = { kind: "owner" } | { kind: "guest"; session: string }; + const facet = defineFacet(() => {}); + expectTypeOf(facet).toEqualTypeOf>(); + type Load = Host["load"]; + expectTypeOf[0]>().toEqualTypeOf< + readonly { readonly name: string; readonly facet: Facet }[] + >(); + expectTypeOf>().toExtend>(); + expectTypeOf>().not.toExtend>(); + expectTypeOf>().not.toExtend>(); + }); +}); diff --git a/packages/chord/test/next/host/host.test.ts b/packages/chord/test/next/host/host.test.ts new file mode 100644 index 000000000..96d2482c4 --- /dev/null +++ b/packages/chord/test/next/host/host.test.ts @@ -0,0 +1,944 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { broken } from "../../../src/next/data/index.ts"; +import { + type Ctx, + createHost, + defineFacet, + type FacetHost, + type HostError, + type Self, +} from "../../../src/next/host/index.ts"; +import type { ChordHost } from "../../../src/next/host/internal.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import { type ChordStore, openStore } from "../../../src/next/store/store.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { + failures, + fakeTimers, + flushed, + forEachCodec, + linked, + loaded, + Notes, + PlacedNotes, + partValue, + RawPeer, + rejection, + Session, + serveNotes, + type TestPrincipal, + testCodec, + transported, +} from "../support.ts"; + +const Info = local("app.info", { version: 1, params: {}, members: { title: str() } }); +const Counter = local("app.counter", { version: 1, params: {}, members: { count: int(), bump: method() } }); +const Page = local("app.page", { version: 1, params: { id: int() }, members: { id: int() } }); +const Svc = local("app.svc", { version: 1, params: { id: int() }, members: { n: int() } }); +const Core = local("app.core", { version: 1, params: {}, members: { n: int() } }); + +const noErrors = { onError: () => {} }; + +/** Serves `Svc`: one native whose open waits for `gate` and whose close records its value and reason in `closes`. */ +function serveGated(h: FacetHost, gate: Promise, closes: string[]): void { + h.serve(Svc, ({ id }, self) => { + self.open( + async () => { + await gate; + return `svc ${id}`; + }, + (value, reason) => void closes.push(`${value} ${reason.kind}`), + ); + return { n: constant(id) }; + }); +} + +describe("load", () => { + it("runs every setup and activates the types with empty params before it resolves", async () => { + const factories = { info: vi.fn(() => ({ title: constant("t") })), page: vi.fn() }; + const facet = defineFacet((host) => { + host.serve(Info, factories.info); + host.serve(Page, factories.page); + }); + const host = createHost(noErrors); + await host.load([{ name: "@app/info", facet }], context); + expect(factories.info).toHaveBeenCalledTimes(1); + expect(factories.page).not.toHaveBeenCalled(); + await host.close(context); + }); + + it("refuses invalid and duplicate plugin names, double serves and a second load", async () => { + const empty = defineFacet(() => {}); + const info = defineFacet((host) => host.serve(Info, () => ({ title: constant("t") }))); + await expect(createHost(noErrors).load([{ name: "Bad Name", facet: empty }], context)).rejects.toThrow( + '"Bad Name" is not a valid plugin name', + ); + const twice = [ + { name: "a", facet: empty }, + { name: "a", facet: empty }, + ]; + await expect(createHost(noErrors).load(twice, context)).rejects.toThrow('plugin "a" is loaded twice'); + const served = [ + { name: "a", facet: info }, + { name: "b", facet: info }, + ]; + await expect(createHost(noErrors).load(served, context)).rejects.toThrow("app.info is served twice"); + const host = createHost(noErrors); + await host.load([], context); + await expect(host.load([], context)).rejects.toThrow("load runs once per host"); + }); + + it("makes a get before it resolves wait for it", async () => { + const host = createHost(noErrors); + const got = host.get(Info, {}, context); + await host.load( + [{ name: "a", facet: defineFacet((h) => h.serve(Info, () => ({ title: constant("t") }))) }], + context, + ); + expect((await got).title.value).toBe("t"); + }); +}); + +describe("the start barrier", () => { + fakeTimers(); + + it("resolves once every host-lifetime object had its first outcome; a failed one is reported and retried", async () => { + let attempts = 0; + const { host, errors } = await loaded((h) => { + h.serve(Info, () => ({ title: constant("t") })); + h.serve(Counter, (_, self) => { + const attempt = ++attempts; + self.open( + async () => { + if (attempt === 1) throw new Error("offline"); + return null; + }, + () => {}, + ); + return { count: constant(attempt), bump: async () => null }; + }); + }); + expect(failures(errors)).toEqual(["app.counter: transient"]); + const counter = host.get(Counter, {}, context); + await vi.advanceTimersByTimeAsync(1000); + expect((await counter).count.value).toBe(2); + }); + + it("is not held by a hanging open: its attempt is abandoned at readyTimeout", async () => { + const host = createHost({ ...noErrors, readyTimeout: 100 }); + const facet = defineFacet((h) => + h.serve(Info, (_, self) => { + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { title: constant("t") }; + }), + ); + let done = false; + void host.load([{ name: "a", facet }], context).then(() => { + done = true; + }); + await vi.advanceTimersByTimeAsync(99); + expect(done).toBe(false); + await vi.advanceTimersByTimeAsync(1); + expect(done).toBe(true); + }); +}); + +describe("get", () => { + it("refuses an unserved type and a contract mismatch", async () => { + const { host } = await loaded((h) => h.serve(Info, () => ({ title: constant("t") }))); + expect(await rejection(host.get(Page, { id: 1 }, context))).toEqual({ + code: "unavailable", + message: "app.page is not served here", + }); + const InfoV2 = local("app.info", { version: 2, params: {}, members: { title: str() } }); + const Changed = local("app.info", { version: 1, params: {}, members: { title: int() } }); + expect(await rejection(host.get(InfoV2, {}, context))).toMatchObject({ code: "contract_mismatch" }); + expect(await rejection(host.get(Changed, {}, context))).toMatchObject({ code: "contract_mismatch" }); + const Same = local("app.info", { version: 1, params: {}, members: { title: str() } }); + expect((await host.get(Same, {}, context)).title.value).toBe("t"); + }); + + it("checks params with L1 (invalid/params)", async () => { + const { host } = await loaded((h) => h.serve(Page, ({ id }) => ({ id: constant(id) }))); + // @ts-expect-error: a string is not an int + expect(await rejection(host.get(Page, { id: "x" }, context))).toMatchObject({ + code: "invalid", + reason: "params", + }); + }); +}); + +describe("deadlines of gets", () => { + fakeTimers(); + + it("reject with unavailable while no attempt has failed (the last failure: see attempt.test.ts)", async () => { + const gate = Promise.withResolvers(); + const { host } = await loaded((h) => serveGated(h, gate.promise, [])); + const waiting = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 500 })); + await vi.advanceTimersByTimeAsync(500); + expect(await waiting).toEqual({ code: "unavailable", message: "app.svc(1): did not become live within 500 ms" }); + }); + + it("include the wait for load", async () => { + const host = createHost(noErrors); + const waiting = rejection(host.get(Info, {}, context, { readyTimeout: 100 })); + await vi.advanceTimersByTimeAsync(100); + expect(await waiting).toEqual({ code: "unavailable", message: "app.info: did not become live within 100 ms" }); + }); + + it("are per getter: the first to pass releases only its own reference, the attempt goes on for the other", async () => { + const gate = Promise.withResolvers(); + const closes: string[] = []; + const { host } = await loaded((h) => serveGated(h, gate.promise, closes), { readyTimeout: 10_000 }); + const early = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 100 })); + const late = host.get(Svc, { id: 1 }, context, { readyTimeout: 1000 }); + await vi.advanceTimersByTimeAsync(100); + expect(await early).toMatchObject({ code: "unavailable" }); + gate.resolve(); + expect((await late).n.value).toBe(1); + expect(closes).toEqual([]); + }); +}); + +describe("close", () => { + it("ends every object, the last to go live first, and rejects new calls", async () => { + const Outer = local("app.outer", { version: 1, params: {}, members: { title: str() } }); + const { host } = await loaded((h) => { + h.serve(Info, () => ({ title: constant("info") })); + h.serve(Outer, (_, self) => ({ title: self.get(Info, {}).title })); + }); + const ends: string[] = []; + const outer = await host.get(Outer, {}, context); + const info = await host.get(Info, {}, context); + void (await outer.title.watch(context)).closed.then(() => ends.push("outer")); + void (await info.title.watch(context)).closed.then(() => ends.push("info")); + await host.close(context); + expect(ends).toEqual(["outer", "info"]); + expect(await rejection(host.get(Info, {}, context))).toEqual({ + code: "released", + reason: "closed", + message: "the host is closed", + }); + expect(() => info.title.value).toThrow("app.info: ended"); + }); + + it("fails a get waiting for load", async () => { + const host = createHost(noErrors); + const got = host.get(Info, {}, context); + await host.close(context); + expect(await rejection(got)).toEqual({ code: "released", reason: "closed", message: "the host is closed" }); + }); + + it("ends every object at once: calls fail released before any close ran", async () => { + const closing = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Counter, (_, self) => { + self.open( + async () => null, + () => closing.promise, + ); + return { count: constant(1), bump: async () => null }; + }), + ); + const counter = await host.get(Counter, {}, context); + const closed = host.close(context); + expect(await rejection(counter.bump(context))).toEqual({ + code: "released", + reason: "ended", + message: "app.counter: ended", + }); + closing.resolve(); + await closed; + }); + + it("aborts a pending open and does not wait for it; its late value is closed with close", async () => { + const gate = Promise.withResolvers(); + const closes: string[] = []; + const { host } = await loaded((h) => serveGated(h, gate.promise, closes)); + const got = rejection(host.get(Svc, { id: 1 }, context)); + await flushed(); + await host.close(context); + expect(await got).toEqual({ code: "released", reason: "ended", message: "app.svc(1): ended" }); + gate.resolve(); + await flushed(); + expect(closes).toEqual(["svc 1 close"]); + }); + + it("joins an ending running in the background", async () => { + const closing = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Svc, ({ id }, self) => { + self.open( + async () => null, + () => closing.promise, + ); + return { n: constant(id) }; + }), + ); + (await host.get(Svc, { id: 1 }, context)).release(); + let closed = false; + const close = host.close(context).then(() => { + closed = true; + }); + await flushed(); + expect(closed).toBe(false); + closing.resolve(); + await close; + }); + + it("closes object by object, a dependent before its dependencies", async () => { + const View = local("app.view", { version: 1, params: { id: int() }, members: { n: int() } }); + const events: string[] = []; + /** A native whose close records when it starts and, a turn later, when it ends. */ + const open = (self: Self, name: string): void => { + self.open( + async () => null, + async () => { + events.push(`${name} closing`); + await flushed(); + events.push(`${name} closed`); + }, + ); + }; + const { host } = await loaded((h) => { + h.serve(Svc, ({ id }, self) => { + open(self, `svc ${id}`); + return { n: constant(id) }; + }); + h.serve(View, ({ id }, self) => { + open(self, `view ${id}`); + return { n: self.get(Svc, { id }).n }; + }); + }); + await host.get(View, { id: 1 }, context); + await host.get(View, { id: 2 }, context); + await host.close(context); + expect(events).toEqual([ + "view 2 closing", + "view 2 closed", + "svc 2 closing", + "svc 2 closed", + "view 1 closing", + "view 1 closed", + "svc 1 closing", + "svc 1 closed", + ]); + }); +}); + +describe("close, its order and its races", () => { + it("returns one closing to every caller: a second close waits for the first", async () => { + const closing = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Counter, (_, self) => { + self.open( + async () => null, + () => closing.promise, + ); + return { count: constant(1), bump: async () => null }; + }), + ); + const first = host.close(context); + let second = false; + void host.close(context).then(() => { + second = true; + }); + await flushed(); + expect(second).toBe(false); + closing.resolve(); + await first; + await flushed(); + expect(second).toBe(true); + }); + + it("tears down what a failed dependent holds with close, and waits for it", async () => { + const Pair = local("app.pair", { version: 1, params: { id: int() }, members: { n: int() } }); + const closing = Promise.withResolvers(); + const closes: string[] = []; + const { host } = await loaded((h) => { + h.serve(Svc, ({ id }, self) => { + self.open( + async () => null, + async (_value, reason) => { + closes.push(`svc ${reason.kind}`); + await closing.promise; + }, + ); + return { n: constant(id) }; + }); + h.serve(Core, (_, self) => { + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + return { n: constant(0) }; + }); + h.serve(Pair, ({ id }, self) => ({ + n: self.derive([self.get(Svc, { id }).n, self.get(Core, {}).n], ([n]) => n), + })); + }); + // The pair fails on the core (R18) and keeps its references: svc(1), live before the pair, held by it alone. + const svc = await host.get(Svc, { id: 1 }, context); + void host.get(Pair, { id: 1 }, context).catch(() => {}); + await flushed(); + svc.release(); + let closed = false; + const close = host.close(context).then(() => { + closed = true; + }); + await flushed(); + expect([closes, closed]).toEqual([["svc close"], false]); + closing.resolve(); + await close; + }); + + it("goes on when onError throws: failures are still classified, teardowns still finish", async () => { + const host = createHost({ + onError: () => { + throw new Error("onError threw"); + }, + }); + const facet = defineFacet((h) => + h.serve(Svc, ({ id }, self) => { + self.open( + async () => { + if (id === 2) throw broken(new Error("no config")); + return null; + }, + () => { + throw new Error("close threw"); + }, + ); + return { n: constant(id) }; + }), + ); + await host.load([{ name: "a", facet }], context); + // The background end reports its throwing close; the next instance waits for that teardown. + (await host.get(Svc, { id: 1 }, context)).release(); + expect((await host.get(Svc, { id: 1 }, context)).n.value).toBe(1); + expect(await rejection(host.get(Svc, { id: 2 }, context))).toEqual({ + code: "unavailable", + message: "app.svc(2): no config", + }); + }); + + describe("with a backoff window running", () => { + fakeTimers(); + + it("ends an object waiting for its window and clears every timer", async () => { + const { host } = await loaded((h) => + h.serve(Svc, (_, self) => { + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + return { n: constant(0) }; + }), + ); + const first = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 100 })); + await vi.advanceTimersByTimeAsync(100); + await first; + const waiting = rejection(host.get(Svc, { id: 1 }, context)); + await flushed(); + await host.close(context); + expect(await waiting).toEqual({ code: "released", reason: "ended", message: "app.svc(1): ended" }); + expect(vi.getTimerCount()).toBe(0); + }); + }); +}); + +describe("getters", () => { + it("one cancelled while another waits for the same object: the other gets it, the object lives", async () => { + const { context: cancellable, cancel } = withCancel(context); + const factory = vi.fn((_: { id: number }) => { + cancel(new Error("cancelled")); + return { id: constant(1) }; + }); + const { host } = await loaded((h) => h.serve(Page, factory)); + const [cancelled, kept] = await Promise.allSettled([ + host.get(Page, { id: 1 }, cancellable), + host.get(Page, { id: 1 }, context), + ]); + expect(cancelled).toMatchObject({ status: "rejected", reason: new Error("cancelled") }); + expect(kept.status === "fulfilled" && kept.value.id.value).toBe(1); + await host.get(Page, { id: 1 }, context); + expect(factory).toHaveBeenCalledTimes(1); + }); +}); + +forEachCodec(() => { + describe("principals and where a reference goes", () => { + const Who = remote("app.who", { version: 1, params: { id: int() }, variants: { v: { state: { who: str() } } } }); + const Outer = remote("app.outer", { version: 1, params: {}, variants: { v: { state: { who: str() } } } }); + const Inner = local("app.inner", { version: 1, params: { id: int() }, members: { who: str() } }); + const Desk = local("app.desk", { version: 1, params: {}, members: { who: method(undefined, str()) } }); + const Gate = remote("app.gate", { + version: 1, + params: {}, + variants: { v: { methods: { who: method(undefined, str()) } } }, + }); + type P = TestPrincipal; + + /** Serves every type above: `Who` shows the principal its object is for. */ + function serveAll(h: FacetHost

, factories: string[]) { + h.serve(Who, ({ id }, principal) => { + factories.push(`${id} ${principal.kind}`); + return { variant: "v", members: { who: constant(principal.kind) } }; + }); + h.serve(Outer, (_, _principal, self) => ({ variant: "v", members: { who: self.get(Who, { id: 3 }).who } })); + h.serve(Inner, ({ id }, self) => ({ who: self.get(Who, { id }).who })); + const who = async (ctx: Ctx

) => { + const ref = await ctx.get(Who, { id: 2 }); + return ref.who.value; + }; + h.serve(Desk, () => ({ who })); + h.serve(Gate, () => ({ variant: "v", members: { who } })); + } + + it("host.get acts as the host's principal; a host without one (a client) serves no remote type", async () => { + const factories: string[] = []; + const { server } = await linked((h) => serveAll(h, factories)); + expect((await server.get(Who, { id: 1 }, context)).who.value).toBe("server"); + const client = createHost

({ onError: () => {} }); + const loading = client.load([{ name: "@test/app", facet: defineFacet

((h) => serveAll(h, [])) }], context); + await expect(loading).rejects.toThrow("app.who: a host without a principal serves only local types"); + }); + + it("ctx.get acts as ctx.principal: a guest's call gets the guest's view; a local handler's acts as the host", async () => { + const { server, client } = await linked((h) => serveAll(h, []), { child: { kind: "guest", session: "S1" } }); + expect(await (await client.get(Gate, {}, context)).who(context)).toBe("guest"); + expect(await (await server.get(Desk, {}, context)).who(context)).toBe("server"); + }); + + it("self.get acts as a remote object's principal, a local object's as the host's", async () => { + const { server, client } = await linked((h) => serveAll(h, []), { child: { kind: "guest", session: "S1" } }); + expect((await client.get(Outer, {}, context)).who.value).toBe("guest"); + expect((await server.get(Inner, { id: 4 }, context)).who.value).toBe("server"); + }); + + it("identity includes the principal key: equal keys share one object", async () => { + const factories: string[] = []; + const host = createHost

({ + principal: { kind: "guest", session: "S1" }, + principalKey: (principal) => principal.kind, + onError: () => {}, + }); + await host.load([{ name: "@test/app", facet: defineFacet

((h) => serveAll(h, factories)) }], context); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "guest", session: "S2" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + await host.get(Who, { id: 1 }, context); + peer.send({ + kind: "hold", + hold: 0, + type: Who.name, + version: Who.version, + fingerprint: Who.fingerprint, + params: jsonPart({ id: 1 }), + }); + await peer.next("bound"); + expect(factories).toEqual(["1 guest"]); + const [c, d] = memoryLinkPair(); + host.accept(c, { principal: { kind: "owner" }, codec: testCodec() }, context); + const owner = new RawPeer(d); + owner.send({ + kind: "hold", + hold: 0, + type: Who.name, + version: Who.version, + fingerprint: Who.fingerprint, + params: jsonPart({ id: 1 }), + }); + await owner.next("bound"); + expect(factories).toEqual(["1 guest", "1 owner"]); + }); + + it("remote objects with empty params are not activated by load; they live while referenced", async () => { + const factories: string[] = []; + const Empty = remote("app.empty", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const { server } = await linked((h) => + h.serve(Empty, (_, principal) => { + factories.push(principal.kind); + return { variant: "v", members: { n: constant(1) } }; + }), + ); + expect(factories).toEqual([]); + (await server.get(Empty, {}, context)).release(); + await flushed(); + await server.get(Empty, {}, context); + expect(factories).toEqual(["server", "server"]); + }); + + it("a local type never crosses a link; a remote type without a parent link waits for one", async () => { + const { client } = await linked(() => {}); + expect(await rejection(client.get(Inner, { id: 1 }, context))).toEqual({ + code: "unavailable", + message: "app.inner is not served here", + }); + const alone = createHost

({ onError: () => {} }); + await alone.load([], context); + expect(alone.hold(Who, { id: 1 }, context).state.value).toEqual({ kind: "loading" }); + expect(await rejection(alone.get(Who, { id: 1 }, context, { readyTimeout: 10 }))).toEqual({ + code: "unavailable", + message: "app.who(1): did not become live within 10 ms", + }); + }); + + it("connect takes one open parent link", async () => { + const { client } = await linked(() => {}); + expect(() => + client.connect(memoryLinkPair()[0], { principal: { kind: "server" }, codec: testCodec() }, context), + ).toThrow("connect: this host has an open parent link"); + }); + + it("close closes the links first: the consumer sees the link drop, no gone", async () => { + const { server, client } = await linked((h) => serveAll(h, [])); + const who = await client.get(Who, { id: 1 }, context); + const watch = await who.who.watch(context); + await server.close(context); + await transported(); + expect(who.state.value).toEqual({ kind: "loading" }); + expect(who.who.value).toBe("owner"); + expect(await Promise.race([watch.closed, transported().then(() => "open")])).toBe("open"); + }); + }); +}); + +describe("a load whose setup throws", () => { + it("closes the host as close does, links included, and rejects with the setup's error", async () => { + const host = createHost({ principal: { kind: "server" }, onError: () => {} }); + const [a, b] = memoryLinkPair(); + const [c, d] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + host.connect(c, { principal: { kind: "server" }, codec: testCodec() }, context); + const child = new RawPeer(b); + const parent = new RawPeer(d); + const facet = defineFacet(() => { + throw new Error("setup threw"); + }); + await expect(host.load([{ name: "@test/app", facet }], context)).rejects.toThrow("setup threw"); + await transported(); + expect([child.closed, parent.closed]).toEqual([true, true]); + expect(await rejection(host.get(Page, { id: 1 }, context))).toEqual({ + code: "released", + reason: "closed", + message: "the host is closed", + }); + }); +}); + +describe("acquiring: hold and get", () => { + fakeTimers(); + + /** Serves `Svc`, whose open throws `offline` (broken when `failing.broken`) while `failing.left` is above 0. */ + function serveFlaky(h: FacetHost, failing: { left: number; broken: boolean }): void { + h.serve(Svc, ({ id }, self) => { + self.open( + async () => { + if (failing.left-- <= 0) return; + throw failing.broken ? broken(new Error("offline")) : new Error("offline"); + }, + () => {}, + ); + return { n: constant(id) }; + }); + } + + const offline = { code: "unavailable", message: "app.svc(1): offline" }; + + it("invalid params and a local type not served here: at once", async () => { + const host = createHost({ ...noErrors, principal: { kind: "server" } }); + await host.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); + const params = { code: "invalid", reason: "params" }; + expect(() => host.hold(Notes, { session: 1 } as never, context)).toThrow("params"); + expect(await rejection(host.get(Notes, { session: 1 } as never, context))).toMatchObject(params); + const unserved = { code: "unavailable", message: "app.info is not served here" }; + expect(() => host.hold(Info, {}, context)).toThrow(unserved.message); + expect(await rejection(host.get(Info, {}, context))).toEqual(unserved); + }); + + it("before load resolved: hold is loading, get waits; once it resolved both bind, or an unserved type fails", async () => { + const host = createHost(noErrors); + const held = host.hold(Info, {}, context); + const unserved = host.hold(Page, { id: 1 }, context); + const got = host.get(Info, {}, context); + const refused = rejection(host.get(Page, { id: 1 }, context)); + expect([held.state.value, unserved.state.value]).toEqual([{ kind: "loading" }, { kind: "loading" }]); + await host.load( + [{ name: "@test/app", facet: defineFacet((h) => h.serve(Info, () => ({ title: constant("t") }))) }], + context, + ); + await flushed(); + expect(held.state.value).toEqual({ kind: "live" }); + expect((await got).title.value).toBe("t"); + const error = { code: "unavailable", message: "app.page is not served here" }; + expect(unserved.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error }, + retryAt: undefined, + }); + expect(await refused).toEqual(error); + }); + + it("a hold released before load resolved makes nothing once it did", async () => { + const factory = vi.fn(({ id }: { id: number }) => ({ id: constant(id) })); + const host = createHost(noErrors); + host.hold(Page, { id: 1 }, context).release(); + await host.load([{ name: "@test/app", facet: defineFacet((h) => h.serve(Page, factory)) }], context); + await flushed(); + expect(factory).not.toHaveBeenCalled(); + }); + + it("through transient failures: hold is failed with retryAt, then loading; get waits, then rejects with the last cause", async () => { + const { host } = await loaded((h) => serveFlaky(h, { left: 5, broken: false })); + const held = host.hold(Svc, { id: 1 }, context); + const got = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 1_500 })); + await flushed(); + expect(held.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "transient", error: offline }, + retryAt: 1_000, + }); + await vi.advanceTimersByTimeAsync(1_000); + await flushed(); + expect(held.state.value).toMatchObject({ kind: "failed", retryAt: 3_000 }); + await vi.advanceTimersByTimeAsync(500); + expect(await got).toEqual(offline); + // The getter released only its own reference: the hold follows the object's retries. + await vi.advanceTimersByTimeAsync(1_500); + expect(held.state.value).toMatchObject({ kind: "failed", retryAt: 7_000 }); + }); + + it("a broken failure: hold is failed without retryAt; get releases its reference and rejects at once", async () => { + const { host } = await loaded((h) => serveFlaky(h, { left: 1, broken: true })); + const held = host.hold(Svc, { id: 1 }, context); + expect(await rejection(host.get(Svc, { id: 1 }, context))).toEqual(offline); + expect(held.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" }, retryAt: undefined }); + }); + + it("a cancelled context: no effect on a hold; a get releases its reference and rejects", async () => { + const gate = Promise.withResolvers(); + const closes: string[] = []; + const { host } = await loaded((h) => serveGated(h, gate.promise, closes)); + const { context: cancellable, cancel } = withCancel(context); + const held = host.hold(Svc, { id: 1 }, cancellable); + const got = host.get(Svc, { id: 2 }, cancellable); + cancel(new Error("user cancelled")); + await expect(got).rejects.toThrow("user cancelled"); + gate.resolve(); + await flushed(); + expect(held.state.value).toEqual({ kind: "live" }); + // The getter's object ended with its only reference: its late value was closed. + expect(closes).toEqual(["svc 2 end"]); + }); + + it("host.close: queued calls reject released, references go released, their watches end (race 10)", async () => { + const gate = Promise.withResolvers(); + const { host } = await loaded((h) => serveGated(h, gate.promise, [])); + const local = host.hold(Svc, { id: 1 }, context); + const remoteHost = createHost(noErrors); + await remoteHost.load([], context); + const remote = remoteHost.hold(Notes, { session: "S1" }, context); + const call = remote.wait(context); + const watches = [await local.state.watch(context), await remote.state.watch(context)]; + await Promise.all([host.close(context), remoteHost.close(context)]); + expect(await rejection(call)).toEqual({ code: "released", reason: "closed", message: "the host is closed" }); + expect([local.state.value, remote.state.value]).toMatchObject([{ kind: "ended" }, { kind: "ended" }]); + expect(await Promise.all(watches.map(({ closed }) => closed))).toEqual([{ kind: "ended" }, { kind: "ended" }]); + }); +}); + +forEachCodec(() => { + describe("connect after a drop", () => { + it("throws while the previous parent link is open or still closing, and takes the next once it closed (race 4)", async () => { + const { client, links } = await linked((h) => void serveNotes(h)); + const notes = await client.get(Notes, { session: "S1" }, context); + const next = (): void => + client.connect(memoryLinkPair()[0], { principal: { kind: "server" }, codec: testCodec() }, context); + // The server's end closes now, the client's a microtask later. + links[0].close(); + expect(next).toThrow("connect: this host has an open parent link"); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + expect(next).not.toThrow(); + }); + }); +}); + +forEachCodec(() => { + describe("where a reference or an arriving hold goes (section 1's table)", () => { + /** A worker for session S1 serving `PlacedNotes`, a raw peer as its child, and one as its parent when `parent`. */ + async function worker(options: { readonly parent?: boolean } = {}) { + const errors: HostError[] = []; + const host = createHost({ + address: Session({ id: "S1" }), + principal: { kind: "worker", session: "S1" }, + onError: (error) => errors.push(error), + }); + let parent: RawPeer | undefined; + if (options.parent === true) { + const [a, b] = memoryLinkPair(); + host.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + parent = new RawPeer(b); + } + await host.load( + [{ name: "@test/w", facet: defineFacet((h) => serveNotes(h, PlacedNotes)) }], + context, + ); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + return { host, child: new RawPeer(b), parent, errors }; + } + + /** Sends a hold of `PlacedNotes` (or `type`) for `session`, naming `address` when given. */ + function holdOf(peer: RawPeer, session: string, address?: string, type: string = PlacedNotes.name) { + peer.send({ + kind: "hold", + hold: 0, + type, + version: PlacedNotes.version, + fingerprint: PlacedNotes.fingerprint, + params: jsonPart({ session }), + ...(address === undefined ? {} : { address: { kind: "session", params: { id: address } } }), + }); + } + + const misplaced = { + code: "invalid", + reason: "placement", + message: "app.placed-notes: placed on session, this host is session|S1", + }; + + it("binds a hold of a type served here that names this host's address", async () => { + const { child } = await worker(); + holdOf(child, "S1", "S1"); + expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); + }); + + it("refuses a hold served here whose params place it elsewhere, whatever address it names: invalid/placement (race 5)", async () => { + const { child } = await worker(); + holdOf(child, "S2", "S1"); + expect(await child.next("failed")).toMatchObject({ + hold: 0, + cause: { kind: "broken", error: { kind: "chord", error: misplaced } }, + }); + holdOf(child, "S2"); + expect(await child.next("failed")).toMatchObject({ + hold: 0, + cause: { kind: "broken", error: { kind: "chord", error: misplaced } }, + }); + }); + + it("places a hold served here by its params, not by the address it names (that only routes it)", async () => { + const { child } = await worker(); + holdOf(child, "S1", "S2"); + expect((await child.next("bound")).holds[0]).toMatchObject({ hold: 0 }); + }); + + it("sends a reference made here whose placement is another host over the parent link, though it serves the type", async () => { + const { host, parent } = await worker({ parent: true }); + const elsewhere = host.hold(PlacedNotes, { session: "S2" }, context); + const hold = await parent!.next("hold"); + expect(hold).toMatchObject({ type: PlacedNotes.name, address: { kind: "session", params: { id: "S2" } } }); + expect(elsewhere.state.value).toEqual({ kind: "loading" }); + expect((await host.get(PlacedNotes, { session: "S1" }, context)).variant).toBe("editor"); + }); + + it("sends a hold it cannot serve or route over the parent link, unparsed", async () => { + const { child, parent } = await worker({ parent: true }); + holdOf(child, "S9", undefined, "app.elsewhere"); + const relayed = await parent!.next("hold"); + expect(relayed).toMatchObject({ type: "app.elsewhere" }); + expect(partValue(relayed.params)).toEqual({ session: "S9" }); + }); + + it("never sends a hold from the parent back to it: unavailable (race 8)", async () => { + const { parent } = await worker({ parent: true }); + holdOf(parent!, "S9", undefined, "app.elsewhere"); + expect(await parent!.next("failed")).toMatchObject({ + hold: 0, + cause: { + kind: "broken", + error: { kind: "chord", error: { code: "unavailable", message: "app.elsewhere is not served here" } }, + }, + }); + expect(parent!.kinds()).toEqual(["hello"]); + }); + + it("refuses a hold it cannot serve or route without a parent link: unavailable", async () => { + const { child } = await worker(); + holdOf(child, "S9", undefined, "app.elsewhere"); + expect(await child.next("failed")).toMatchObject({ + cause: { kind: "broken", error: { kind: "chord", error: { code: "unavailable" } } }, + }); + }); + + it("fails a reference broken with what its placement throws", async () => { + const Throwing = remote("app.throwing", { + version: 1, + params: {}, + placement: () => { + throw new Error("placement threw"); + }, + variants: { v: { state: { n: int() } } }, + }); + const client = createHost(noErrors); + await client.load([], context); + expect(await rejection(client.get(Throwing, {}, context))).toMatchObject({ + message: "app.throwing: placement threw", + }); + expect(client.hold(Throwing, {}, context).state.value).toMatchObject({ kind: "failed" }); + }); + }); +}); + +describe("the host's store", () => { + /** A store on `:memory:`, as Chord's layers see it. */ + async function memoryStore(): Promise { + return (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore; + } + + it("belongs to one open host at a time (rule 7)", async () => { + const store = await memoryStore(); + const first = createHost({ store, onError: () => {} }); + expect(() => createHost({ store, onError: () => {} })).toThrow("store: already attached to an open host"); + await first.close(context); + await createHost({ store, onError: () => {} }).close(context); + await store.close(context); + }); + + it("reports storage_failed once and closes the host; the app still closes the store", async () => { + const store = await memoryStore(); + const errors: HostError[] = []; + const host = createHost({ store, onError: (error) => errors.push(error) }); + await host.load([], context); + const broken: StagedWrite = { + order: "write", + apply: async (step) => { + await step.run("INSERT INTO no_such_table VALUES (1)"); + return { kind: "document.retire", id: 0, revision: 0 }; + }, + }; + const fence = { admit: () => true, adopt: () => {} }; + await expect(store.commit([broken], fence, context)).rejects.toMatchObject({ reason: "storage_failed" }); + await expect(store.commit([broken], fence, context)).rejects.toMatchObject({ reason: "storage_failed" }); + // The host closed itself; the app's own close returns that closing. + expect((host as ChordHost).closing).toBe(true); + await host.close(context); + expect(errors.map(({ kind }) => kind)).toEqual(["storage_failed"]); + await store.close(context); + }); +}); diff --git a/packages/chord/test/next/host/keyed-members.test.ts b/packages/chord/test/next/host/keyed-members.test.ts new file mode 100644 index 000000000..d2570dd56 --- /dev/null +++ b/packages/chord/test/next/host/keyed-members.test.ts @@ -0,0 +1,452 @@ +// Tests of keyed members across attempts (slice 6a §1.2, §1.4, §12): the keys references hold on an object's keyed +// members, frozen while it is failed, re-bound by name (loaded first) when its next attempt goes live, released once. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withAbortSignal } from "../../../src/context/index.ts"; +import { int, keyed, local, obj, str } from "../../../src/next/contracts/index.ts"; +import { ChordError } from "../../../src/next/data/index.ts"; +import { defineState, type KeyedHandle } from "../../../src/next/documents/index.ts"; +import type { Self } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { + changesOf, + delivered, + expectCollected, + failLive, + instancesOf, + loaded, + memoryStore, + valuesOf, +} from "../support.ts"; + +const attempted = instancesOf(ChordObject, "attempt"); + +type N = { n: number }; +const Schema = obj({ n: int() }); +const Notes = defineState("notes", { schema: Schema, version: 1, initial: () => ({ n: 0 }), keyed: true }); +const Own = defineState("own", { schema: Schema, version: 1, initial: () => ({ n: 0 }) }); +const Provider = local("app.provider", { version: 1, params: { id: str() }, members: { notes: keyed(Schema) } }); +const Exposer = local("app.exposer", { version: 1, params: {}, members: { notes: keyed(Schema) } }); +/** A viewer with params: it ends with its last reference (an object with empty params lives as long as its host). */ +const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: { value: Schema, double: int() } }); + +/** Providers `a` and `b`, an exposer re-exposing provider `which()`'s notes, and a viewer of its key `k`. */ +async function served(which: () => string) { + const providers: Record> = {}; + const exposers: Self[] = []; + const { host } = await loaded((h) => { + h.serve(Provider, ({ id }, self) => { + const notes = self.state(Notes); + providers[id] = notes; + return { notes }; + }); + h.serve(Exposer, (_, self) => { + exposers.push(self); + self.state(Own); + return { notes: self.get(Provider, { id: which() }).notes }; + }); + h.serve(Viewer, (_, self) => { + const value = self.get(Exposer, {}).notes.at("k"); + return { value, double: self.derive([value], ([v]) => v.n * 2) }; + }); + }); + const a = await host.get(Provider, { id: "a" }, context); + const b = await host.get(Provider, { id: "b" }, context); + const set = (id: string, n: number) => + providers[id]!.change( + "k", + (draft) => { + draft.n = n; + }, + context, + ); + await set("a", 1); + await set("b", 2); + const exposer = await host.get(Exposer, {}, context); + const viewer = await host.get(Viewer, { id: 1 }, context); + const fail = (): void => + failLive(attempted, exposers.at(-1)!, new ChordError({ code: "unavailable", message: "boom" })); + return { host, a, b, exposer, viewer, set, fail, providers }; +} + +describe("a failed object's held keys", () => { + it("keep their last values and get no frames (at, watch, a derive over at); the provider goes on", async () => { + const { host, a, exposer, viewer, set, fail } = await served(() => "a"); + const values = changesOf(await viewer.value.watch(context)); + const keyWatch = (await exposer.notes.watch("k", context))!; + const keyed = changesOf(keyWatch); + const direct = changesOf((await a.notes.watch("k", context))!); + fail(); + await set("a", 3); + await delivered(); + expect(exposer.state.value.kind).toBe("failed"); + expect([viewer.value.value, viewer.double.value, keyWatch.value]).toEqual([{ n: 1 }, 2, { n: 1 }]); + expect([values, keyed]).toEqual([[], []]); + expect(valuesOf(direct)).toEqual([{ n: 3 }]); + await host.close(context); + }); +}); + +describe("re-binding when a new attempt goes live", () => { + it("follows the new attempt's member of the same name (one frame); the old provider's pin is released", async () => { + let which = "a"; + const { host, exposer, viewer, set, fail, providers } = await served(() => which); + const values = changesOf(await viewer.value.watch(context)); + const doubles = changesOf(await viewer.double.watch(context)); + const keyWatch = (await exposer.notes.watch("k", context))!; + const keyed = changesOf(keyWatch); + const pins = (id: string) => + (providers[id] as unknown as { slot: { source(key: string): { pins: number } | undefined } }).slot.source("k") + ?.pins; + expect(pins("a")).toBe(1); + fail(); + which = "b"; + exposer.retry(); + (await host.get(Exposer, {}, context)).release(); + await delivered(); + expect([viewer.value.value, viewer.double.value, keyWatch.value]).toEqual([{ n: 2 }, 4, { n: 2 }]); + expect([pins("a"), pins("b")]).toEqual([undefined, 1]); + await set("b", 4); + await set("a", 5); + await delivered(); + expect(valuesOf(values)).toEqual([{ n: 2 }, { n: 4 }]); + expect(valuesOf(keyed)).toEqual([{ n: 2 }, { n: 4 }]); + expect(valuesOf(doubles)).toEqual([4, 8]); + await host.close(context); + }); + + it("loads a cold persisted key of the new member before the new attempt goes live", async () => { + const Stored = defineState("notes", { + schema: Schema, + version: 1, + initial: () => ({ n: 0 }), + keyed: true, + persist: true, + checkpointWhen: () => false, + }); + const Durable = local("app.durable", { version: 1, params: {}, members: { notes: keyed(Schema) } }); + const store = await memoryStore(); + let memos: KeyedHandle | undefined; + const serveDurable = (h: Parameters[0]>[0]) => + h.serve(Durable, (_, self) => { + memos = self.state(Stored); + return { notes: memos }; + }); + const seed = await loaded(serveDurable, { store }); + await seed.host.get(Durable, {}, context); + await memos!.change( + "k", + (draft) => { + draft.n = 10; + }, + context, + ); + await seed.host.close(context); + // A new host: the key is in the store only. + let runs = 0; + const exposers: Self[] = []; + const { host } = await loaded( + (h) => { + h.serve(Provider, (_, self) => ({ notes: self.state(Notes) })); + serveDurable(h); + h.serve(Exposer, (_, self) => { + exposers.push(self); + self.state(Own); + return { notes: ++runs === 1 ? self.get(Provider, { id: "a" }).notes : self.get(Durable, {}).notes }; + }); + h.serve(Viewer, (_, self) => { + const value = self.get(Exposer, {}).notes.at("k"); + return { value, double: self.derive([value], ([v]) => v.n * 2) }; + }); + }, + { store }, + ); + const exposer = await host.get(Exposer, {}, context); + const viewer = await host.get(Viewer, { id: 1 }, context); + failLive(attempted, exposers[0]!, new ChordError({ code: "unavailable", message: "boom" })); + exposer.retry(); + (await host.get(Exposer, {}, context)).release(); + expect([viewer.value.value, viewer.double.value]).toEqual([{ n: 10 }, 20]); + await host.close(context); + await store.close(context); + }); +}); + +/** A provider with a persisted keyed state (empty params: a stable identity), whose keys a new host loads cold. */ +const ColdProvider = local("app.cold-provider", { version: 1, params: {}, members: { notes: keyed(Schema) } }); +/** A persisted keyed state whose keys a new host loads from the store. */ +const Stored = defineState("notes", { + schema: Schema, + version: 1, + initial: () => ({ n: 0 }), + keyed: true, + persist: true, + checkpointWhen: () => false, +}); + +/** The pins on key `k` of a provider's notes (`undefined`: none). */ +function pinsOf(notes: KeyedHandle): number | undefined { + // The handle's slot keeps the key's source while it is pinned. + return (notes as unknown as { slot: { source(key: string): { pins: number } | undefined } }).slot.source("k")?.pins; +} + +/** + * Provider `a` (its key `k` in the store only, its loads held at `gate` once `hold()` is called) and `b` (in memory), + * an exposer re-exposing `which()`'s notes, and a viewer of key `k`. + */ +async function cold(which: () => string) { + const store = await memoryStore(); + let seeded: KeyedHandle | undefined; + const seed = await loaded( + (h) => + h.serve(ColdProvider, (_, self) => { + seeded = self.state(Stored); + return { notes: seeded }; + }), + { + store, + }, + ); + await seed.host.get(ColdProvider, {}, context); + await seeded!.change( + "k", + (draft) => { + draft.n = 1; + }, + context, + ); + await seed.host.close(context); + // A new host: the key is in the store only. + const providers: Record> = {}; + const exposers: Self[] = []; + const { host, errors } = await loaded( + (h) => { + h.serve(ColdProvider, (_, self) => { + providers.a = self.state(Stored); + return { notes: providers.a }; + }); + h.serve(Provider, ({ id }, self) => { + providers[id] = self.state(Notes); + return { notes: providers[id] }; + }); + h.serve(Exposer, (_, self) => { + exposers.push(self); + self.state(Own); + const notes = which() === "a" ? self.get(ColdProvider, {}).notes : self.get(Provider, { id: "b" }).notes; + return { notes }; + }); + h.serve(Viewer, (_, self) => { + const value = self.get(Exposer, {}).notes.at("k"); + return { value, double: self.derive([value], ([v]) => v.n * 2) }; + }); + }, + { store }, + ); + await host.get(ColdProvider, {}, context); + await host.get(Provider, { id: "b" }, context); + await providers.b!.change( + "k", + (draft) => { + draft.n = 2; + }, + context, + ); + const exposer = await host.get(Exposer, {}, context); + const gate = Promise.withResolvers(); + const entered = Promise.withResolvers(); + const hold = (): void => { + // The slot's persistence reads cold keys from the store; the test holds the read open. + const persistence = (providers.a as unknown as { slot: { persistence: { read(...args: unknown[]): unknown } } }) + .slot.persistence; + const read = persistence.read.bind(persistence); + persistence.read = async (...args: unknown[]) => { + const result = await read(...args); + entered.resolve(); + await gate.promise; + return result; + }; + }; + const fail = (): void => + failLive(attempted, exposers.at(-1)!, new ChordError({ code: "unavailable", message: "boom" })); + const close = async (): Promise => { + await host.close(context); + await store.close(context); + }; + return { host, errors, exposer, providers, hold, gate, entered, fail, close }; +} + +describe("a key whose first pin still loads", () => { + it("the object fails and re-binds meanwhile: the consumer retries on the new member, the late pin is released", async () => { + let which = "a"; + const { host, errors, exposer, providers, hold, gate, entered, fail, close } = await cold(() => which); + hold(); + const loading = host.get(Viewer, { id: 1 }, context); + await entered.promise; + fail(); + which = "b"; + exposer.retry(); + (await host.get(Exposer, {}, context)).release(); + gate.resolve(); + const viewer = await loading; + await delivered(); + expect([viewer.value.value, viewer.double.value]).toEqual([{ n: 2 }, 4]); + expect([pinsOf(providers.a!), pinsOf(providers.b!)]).toEqual([undefined, 1]); + expect(errors.filter((error) => error.kind === "activation_failed").length).toBe(1); + const values = changesOf(await viewer.value.watch(context)); + await providers.a!.change( + "k", + (draft) => { + draft.n = 3; + }, + context, + ); + await providers.b!.change( + "k", + (draft) => { + draft.n = 4; + }, + context, + ); + await delivered(); + expect(valuesOf(values)).toEqual([{ n: 4 }]); + await close(); + }); + + it("the object fails and stays failed: the consumer's activation fails transient, never live on it", async () => { + const { host, errors, exposer, providers, hold, gate, entered, fail, close } = await cold(() => "a"); + hold(); + const abort = new AbortController(); + const loading = host.get(Viewer, { id: 1 }, withAbortSignal(abort.signal, context)); + await entered.promise; + fail(); + gate.resolve(); + await delivered(); + await new Promise((resolve) => setTimeout(resolve, 10)); + expect(exposer.state.value.kind).toBe("failed"); + const failure = errors.find((error) => error.kind === "activation_failed"); + expect(failure).toMatchObject({ failure: { kind: "transient" } }); + abort.abort(); + await expect(loading).rejects.toThrow(); + expect(pinsOf(providers.a!)).toBeUndefined(); + await close(); + }); + + it("the object ends meanwhile: the consumer's wait rejects, the late pin is released", async () => { + const { host, providers, hold, gate, entered, close } = await cold(() => "a"); + hold(); + const loading = host.get(Viewer, { id: 1 }, context); + await entered.promise; + const closing = host.close(context); + gate.resolve(); + await expect(loading).rejects.toThrow(); + await closing; + expect(pinsOf(providers.a!)).toBeUndefined(); + await close(); + }); +}); + +describe("a keyed watch whose object closes while it is set up", () => { + it("rejects with the object's end error, whichever microtask the close lands in", async () => { + for (let gap = 0; gap < 40; gap++) { + let notes: KeyedHandle | undefined; + const { host } = await loaded((h) => + h.serve(Exposer, (_, self) => { + notes = self.state(Notes); + return { notes }; + }), + ); + const exposer = await host.get(Exposer, {}, context); + await notes!.change( + "a", + (draft) => { + draft.n = 1; + }, + context, + ); + const watching = exposer.notes.watch("a", context).then( + () => "watch", + (error: unknown) => `${(error as ChordError).code}/${(error as ChordError).reason}`, + ); + for (let turn = 0; turn < gap; turn++) await Promise.resolve(); + await host.close(context); + expect(["watch", "released/ended"]).toContain(await watching); + } + }); +}); + +describe("chains and releases", () => { + it("a key re-exposed through two objects freezes with the middle one and follows its next attempt", async () => { + const Outer = local("app.outer", { version: 1, params: {}, members: { notes: keyed(Schema) } }); + const OuterViewer = local("app.outer-viewer", { version: 1, params: {}, members: { value: Schema } }); + let which = "a"; + const middles: Self[] = []; + const providers: Record> = {}; + const { host } = await loaded((h) => { + h.serve(Provider, ({ id }, self) => { + const notes = self.state(Notes); + providers[id] = notes; + return { notes }; + }); + h.serve(Exposer, (_, self) => { + middles.push(self); + self.state(Own); + return { notes: self.get(Provider, { id: which }).notes }; + }); + h.serve(Outer, (_, self) => ({ notes: self.get(Exposer, {}).notes })); + h.serve(OuterViewer, (_, self) => ({ value: self.get(Outer, {}).notes.at("k") })); + }); + for (const id of ["a", "b"]) { + await host.get(Provider, { id }, context); + await providers[id]!.change( + "k", + (draft) => { + draft.n = id === "a" ? 1 : 2; + }, + context, + ); + } + const middle = await host.get(Exposer, {}, context); + const viewer = await host.get(OuterViewer, {}, context); + const values = changesOf(await viewer.value.watch(context)); + failLive(attempted, middles[0]!, new ChordError({ code: "unavailable", message: "boom" })); + await providers.a!.change( + "k", + (draft) => { + draft.n = 3; + }, + context, + ); + await delivered(); + expect([viewer.value.value, values]).toEqual([{ n: 1 }, []]); + which = "b"; + middle.retry(); + (await host.get(Exposer, {}, context)).release(); + await delivered(); + expect(viewer.value.value).toEqual({ n: 2 }); + expect(valuesOf(values)).toEqual([{ n: 2 }]); + await host.close(context); + }); + + it("a consumer released while the object re-activates leaves no hold behind; nothing is kept for it", async () => { + let which = "a"; + const refs = await (async () => { + const { host, exposer, viewer, fail, providers } = await served(() => which); + const pins = (id: string) => + (providers[id] as unknown as { slot: { source(key: string): { pins: number } | undefined } }).slot.source( + "k", + )?.pins; + fail(); + which = "b"; + exposer.retry(); + viewer.release(); + (await host.get(Exposer, {}, context)).release(); + // The viewer's teardown releases its references. + await new Promise((resolve) => setTimeout(resolve, 10)); + expect([pins("a"), pins("b")]).toEqual([undefined, undefined]); + const kept = [new WeakRef(viewer), new WeakRef(providers.a!)]; + await host.close(context); + return kept; + })(); + await expectCollected(...refs); + }); +}); diff --git a/packages/chord/test/next/host/member-source.test.ts b/packages/chord/test/next/host/member-source.test.ts new file mode 100644 index 000000000..77939396b --- /dev/null +++ b/packages/chord/test/next/host/member-source.test.ts @@ -0,0 +1,507 @@ +// Tests of member sources (slice 6a §1.2, §1.4): a local object's state members as references and other objects' +// derives follow them across attempts: passed on while live, frozen while failed, re-bound by name when live again. + +import { describe, expect, it } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, keyed, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { ChordError } from "../../../src/next/data/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Self, type StoreSelf } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { resolveWatchable } from "../../../src/next/host/reference.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import type { Source, Subscriber } from "../../../src/next/state/internal.ts"; +import { + changesOf, + delivered, + expectCollected, + failLive, + holdNextCommit, + instancesOf, + linked, + loaded, + memoryStore, + metered, + transported, + valuesOf, +} from "../support.ts"; + +const attempted = instancesOf(ChordObject, "attempt"); + +const List = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); +const Store = local("app.store", { + version: 1, + params: {}, + members: { list: obj({ items: arr(str()) }), count: int(), add: method(obj({ text: str() })) }, +}); +const Viewer = local("app.count-viewer", { version: 1, params: {}, members: { tens: int() } }); + +/** Serves `Store` (a list, its count, `add`) and `Viewer` (a derive over the store's count); `attempts` are the store's. */ +async function served(options: Parameters[1] = {}) { + const attempts: Self[] = []; + const lists: SingletonHandle<{ items: string[] }>[] = []; + const { host, errors } = await loaded((h) => { + h.serve(Store, (_, self) => { + attempts.push(self); + const list = self.state(List); + lists.push(list); + return { + list, + count: self.derive([list], ([l]) => l.items.length), + add: async ({ text }: { text: string }, ctx: Context) => { + await list.change((draft) => void draft.items.push(text), ctx); + return null; + }, + }; + }); + h.serve(Viewer, (_, self) => ({ tens: self.derive([self.get(Store, {}).count], ([n]) => n * 10) })); + }, options); + const store = await host.get(Store, {}, context); + const fail = (): void => + failLive(attempted, attempts.at(-1)!, new ChordError({ code: "unavailable", message: "boom" })); + return { host, errors, store, attempts, lists, fail }; +} + +describe("following the live attempt", () => { + it("passes the member's frames on as they are, and keeps a derive member lazy while nothing watches it", async () => { + const { telemetry, counts } = await metered(); + const { host, store, lists: handles } = await served({ telemetry }); + expect(store.count.value).toBe(0); + const computes = (): number => counts()["chord.derive.computes"] ?? 0; + const before = computes(); + expect(before).toBeGreaterThan(0); + await store.add({ text: "a" }, context); + await store.add({ text: "b" }, context); + // Unwatched: marked only, computed by the next read. + expect(computes()).toBe(before); + expect(store.count.value).toBe(2); + const watch = await store.count.watch(context); + const counted = changesOf(watch); + const lists = changesOf(await store.list.watch(context)); + const own = changesOf(await handles[0]!.watch(context)); + await store.add({ text: "c" }, context); + await delivered(); + expect(counted).toEqual([{ kind: "value", value: 3, ops: [["r", 3]], retired: false }]); + // The slot's own change, not a copy. + expect(lists).toHaveLength(1); + expect(lists[0]).toBe(own[0]); + await watch.stop(); + await delivered(); + const stopped = computes(); + await store.add({ text: "d" }, context); + expect(computes()).toBe(stopped); + await host.close(context); + }); +}); + +describe("a failed object's members", () => { + it("keep their last values and get no frames; a new watch opens at the last value (slice 6a §1.4)", async () => { + const { host, store, lists, fail } = await served(); + await store.add({ text: "a" }, context); + const counted = changesOf(await store.count.watch(context)); + fail(); + await delivered(); + expect(store.state.value.kind).toBe("failed"); + const late = await store.count.watch(context); + expect(late.value).toBe(1); + const lately = changesOf(late); + const listed = changesOf(await store.list.watch(context)); + // The failed attempt's handle is sealed; the slot changes only through a new attempt. + expect(lists[0]!.value).toEqual({ items: ["a"] }); + store.retry(); + await (await host.get(Store, {}, context)).add({ text: "b" }, context); + await delivered(); + expect(valuesOf(counted)).toEqual([2]); + expect(valuesOf(listed)).toEqual([{ items: ["a", "b"] }]); + expect(valuesOf(lately)).toEqual([2]); + await host.close(context); + }); + + it("re-exposed from another object freeze with the re-exposing object; the provider's source goes on", async () => { + const Note = obj({ title: str() }); + const Memo = defineState("memo", { schema: Note, version: 1, initial: () => ({ title: "" }), keyed: true }); + const Own = defineState("own", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const Provider = local("app.provider", { + version: 1, + params: {}, + members: { n: int(), notes: keyed(Note), set: method(int()) }, + }); + const Exposer = local("app.exposer", { version: 1, params: {}, members: { n: int(), note: Note } }); + let exposer: Self | undefined; + const { host } = await loaded((h) => { + h.serve(Provider, (_, self) => { + const memos = self.state(Memo); + const n = self.derive([memos.at("a")], ([note]) => note.title.length); + return { + n, + notes: memos, + set: async (length, ctx) => { + await memos.change( + "a", + (draft) => { + draft.title = "x".repeat(length); + }, + ctx, + ); + return null; + }, + }; + }); + h.serve(Exposer, (_, self) => { + exposer = self; + self.state(Own); + const provider = self.get(Provider, {}); + return { n: provider.n, note: provider.notes.at("a") }; + }); + }); + const provider = await host.get(Provider, {}, context); + const exposed = await host.get(Exposer, {}, context); + await provider.set(1, context); + const exposedN = changesOf(await exposed.n.watch(context)); + const exposedNote = changesOf(await exposed.note.watch(context)); + const providerN = changesOf(await provider.n.watch(context)); + failLive(attempted, exposer!, new ChordError({ code: "unavailable", message: "boom" })); + await delivered(); + await provider.set(7, context); + await delivered(); + expect(exposed.state.value.kind).toBe("failed"); + expect([exposed.n.value, exposed.note.value]).toEqual([1, { title: "x" }]); + expect([exposedN, exposedNote]).toEqual([[], []]); + expect(provider.n.value).toBe(7); + expect(valuesOf(providerN)).toEqual([7]); + await host.close(context); + }); +}); + +describe("the window of a live failure", () => { + it("members following another object's sources freeze at once, while an admitted commit of the object runs", async () => { + const Counter = defineState("counter", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const Persisted = defineState("own", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: 0 }), + persist: true, + checkpointWhen: () => false, + }); + const Provider = local("app.provider", { version: 1, params: {}, members: { n: int(), set: method(int()) } }); + const Exposer = local("app.exposer", { version: 1, params: {}, members: { n: int(), own: obj({ n: int() }) } }); + const store = await memoryStore(); + let exposer: StoreSelf | undefined; + let own: SingletonHandle<{ n: number }> | undefined; + const { host } = await loaded( + (h) => { + h.serve(Provider, (_, self) => { + const doc = self.state(Counter); + return { + n: self.derive([doc], ([d]) => d.n), + set: async (n, ctx) => { + await doc.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }); + h.serve(Exposer, (_, self) => { + exposer = self; + own = self.state(Persisted); + return { n: self.get(Provider, {}).n, own }; + }); + }, + { store }, + ); + const provider = await host.get(Provider, {}, context); + const exposed = await host.get(Exposer, {}, context); + await provider.set(1, context); + const frames = changesOf(await exposed.n.watch(context)); + const owned = changesOf(await exposed.own.watch(context)); + const held = holdNextCommit(store); + const change = own!.change((draft) => { + draft.n = 5; + }, context); + await held.entered; + failLive(attempted, exposer!, new ChordError({ code: "unavailable", message: "boom" })); + await provider.set(9, context); + await delivered(); + expect([exposed.n.value, frames]).toEqual([1, []]); + held.release(); + await change; + await delivered(); + // The object's own admitted commit still publishes once. + expect(valuesOf(owned)).toEqual([{ n: 5 }]); + await host.close(context); + await store.close(context); + }); +}); + +describe("re-binding when a new attempt goes live", () => { + it("recovers a consumer's derive that failed on the provider's earlier error, and the derives downstream", async () => { + const Counted = local("app.counted", { version: 1, params: {}, members: { count: int(), add: method(str()) } }); + const Tens = local("app.tens", { version: 1, params: {}, members: { tens: int(), hundreds: int() } }); + const selves: Self[] = []; + const { host } = await loaded((h) => { + h.serve(Counted, (_, self) => { + selves.push(self); + const first = selves.length === 1; + const list = self.state(List); + return { + count: self.derive([list], ([l]) => { + if (first && l.items.length === 1) throw new Error("count broke"); + return l.items.length; + }), + add: async (text, ctx) => { + await list.change((draft) => void draft.items.push(text), ctx); + return null; + }, + }; + }); + h.serve(Tens, (_, self) => { + const tens = self.derive([self.get(Counted, {}).count], ([n]) => n * 10); + return { tens, hundreds: self.derive([tens], ([n]) => n * 10) }; + }); + }); + const counted = await host.get(Counted, {}, context); + const tens = await host.get(Tens, {}, context); + await counted.add("a", context); + expect(() => tens.tens.value).toThrow("count broke"); + expect(() => tens.hundreds.value).toThrow("count broke"); + failLive(attempted, selves[0]!, new ChordError({ code: "unavailable", message: "boom" })); + counted.retry(); + (await host.get(Counted, {}, context)).release(); + expect([tens.tens.value, tens.hundreds.value]).toEqual([10, 100]); + const frames = changesOf(await tens.hundreds.watch(context)); + await counted.add("b", context); + await delivered(); + expect(valuesOf(frames)).toEqual([200]); + await host.close(context); + }); + + it("re-binds a consumer's derive over a retried provider's derive member, and its watches get frames again", async () => { + const { host, store, fail } = await served(); + const viewer = await host.get(Viewer, {}, context); + const tens = changesOf(await viewer.tens.watch(context)); + const counted = changesOf(await store.count.watch(context)); + await store.add({ text: "a" }, context); + await delivered(); + fail(); + await delivered(); + expect([store.count.value, viewer.tens.value]).toEqual([1, 10]); + store.retry(); + await (await host.get(Store, {}, context)).add({ text: "b" }, context); + await delivered(); + expect(viewer.state.value.kind).toBe("live"); + expect([store.count.value, viewer.tens.value]).toEqual([2, 20]); + expect(valuesOf(tens)).toEqual([10, 20]); + expect(valuesOf(counted)).toEqual([1, 2]); + await host.close(context); + }); + + it("sends one frame from the last value when the new attempt's member differs", async () => { + const Shown = local("app.shown", { version: 1, params: {}, members: { label: str() } }); + const labels = ["old", "new"]; + let attempt = 0; + const selves: Self[] = []; + const { host } = await loaded((h) => + h.serve(Shown, (_, self) => { + selves.push(self); + self.state(List); + const label = labels[attempt++]!; + return { label: self.derive([], () => label) }; + }), + ); + const shown = await host.get(Shown, {}, context); + const frames = changesOf(await shown.label.watch(context)); + failLive(attempted, selves[0]!, new ChordError({ code: "unavailable", message: "boom" })); + shown.retry(); + (await host.get(Shown, {}, context)).release(); + await delivered(); + expect(shown.label.value).toBe("new"); + expect(frames).toEqual([{ kind: "value", value: "new", ops: [["r", "new"]], retired: false }]); + await host.close(context); + }); +}); + +describe("subscribers", () => { + it("a hold over a link on a remote object re-exposing a local member gets frames again after the local retry", async () => { + const Count = local("app.count", { version: 1, params: {}, members: { n: obj({ n: int() }), add: method() } }); + const View = remote("app.view", { version: 1, params: {}, variants: { v: { state: { n: obj({ n: int() }) } } } }); + const Counter = defineState("counter", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const selves: Self[] = []; + const { server, client } = await linked((h) => { + h.serve(Count, (_, self) => { + selves.push(self); + const counter = self.state(Counter); + return { + n: self.derive([counter], ([c]) => ({ n: c.n * 10 })), + add: async (ctx: Context) => { + await counter.change((draft) => void draft.n++, ctx); + return null; + }, + }; + }); + h.serve(View, (_, _principal, self) => ({ variant: "v", members: { n: self.get(Count, {}).n } })); + }); + const count = await server.get(Count, {}, context); + const view = await client.get(View, {}, context); + if (view.variant !== "v") throw new Error("expected variant v"); + const frames = changesOf(await view.n.watch(context)); + await count.add(context); + await transported(); + failLive(attempted, selves[0]!, new ChordError({ code: "unavailable", message: "boom" })); + count.retry(); + await (await server.get(Count, {}, context)).add(context); + await transported(); + expect(view.n.value).toEqual({ n: 20 }); + expect(valuesOf(frames)).toEqual([{ n: 10 }, { n: 20 }]); + await client.close(context); + await server.close(context); + }); +}); + +describe("subscribers, directly", () => { + it("are fed the member's calls: a failure, then its recovery at the re-bind as a root replacement", async () => { + const Counted = local("app.counted", { version: 1, params: {}, members: { count: int(), add: method(str()) } }); + const selves: Self[] = []; + const { host } = await loaded((h) => + h.serve(Counted, (_, self) => { + selves.push(self); + const first = selves.length === 1; + const list = self.state(List); + return { + count: self.derive([list], ([l]) => { + if (first && l.items.length === 2) throw new Error("count broke"); + return l.items.length; + }), + add: async (text, ctx) => { + await list.change((draft) => void draft.items.push(text), ctx); + return null; + }, + }; + }), + ); + const counted = await host.get(Counted, {}, context); + const source = resolveWatchable(counted.count) as Source; + const events: unknown[] = []; + const subscriber: Subscriber = { + advance: (call) => + void events.push(call.kind === "value" ? [call.value, call.ops] : (call.error as Error).message), + end: (end) => void events.push(end.kind), + }; + expect(source.value).toBe(0); + source.subscribe(subscriber); + await counted.add("a", context); + await counted.add("b", context); + failLive(attempted, selves[0]!, new ChordError({ code: "unavailable", message: "boom" })); + counted.retry(); + await (await host.get(Counted, {}, context)).add("c", context); + expect(counted.count.value).toBe(3); + expect(events).toEqual([[1, [["r", 1]]], "count broke", [2, [["r", 2]]], [3, [["r", 3]]]]); + await host.close(context); + }); +}); + +describe("a member the new attempt's variant lacks", () => { + it("fails: its watch gets one failure call and stays open, reads throw invalid/member; a later attempt brings it back", async () => { + const Two = remote("app.two", { + version: 1, + params: {}, + variants: { full: { state: { n: int() } }, view: { state: { m: int() } } }, + }); + const owner = { kind: "owner" } as const; + const host = createHost({ principal: owner, onError: () => {} }); + const selves: Self[] = []; + const facet = defineFacet((h) => + h.serve(Two, (_, _principal, self) => { + selves.push(self); + self.state(List); + return selves.length === 2 + ? { variant: "view", members: { m: constant(2) } } + : { variant: "full", members: { n: constant(selves.length) } }; + }), + ); + await host.load([{ name: "@test/app", facet }], context); + const two = await host.get(Two, {}, context); + if (two.variant !== "full") throw new Error("expected the full variant"); + const n = two.n; + const calls = changesOf(await n.watch(context)); + const boom = new ChordError({ code: "unavailable", message: "boom" }); + failLive(attempted, selves[0]!, boom); + two.retry(); + (await host.get(Two, {}, context)).release(); + await delivered(); + expect(calls).toEqual([ + { kind: "failed", error: expect.objectContaining({ code: "invalid", reason: "member" }) }, + ]); + expect(() => n.value).toThrow('app.two (variant view) has no state member "n"'); + // The same as over a link: the member comes back with the next attempt that has it. + failLive(attempted, selves[1]!, boom); + two.retry(); + (await host.get(Two, {}, context)).release(); + await delivered(); + expect(n.value).toBe(3); + expect(calls.at(-1)).toEqual({ kind: "value", value: 3, ops: [["r", 3]], retired: false }); + await host.close(context); + }); +}); + +describe("an ended object's members", () => { + it("mark the derives over them: a consumer's derive keeps no cached value, its read throws released", async () => { + const Paged = local("app.paged", { version: 1, params: { id: int() }, members: { n: int() } }); + const Reader = local("app.reader", { version: 1, params: {}, members: { twice: int() } }); + const { host } = await loaded((h) => { + h.serve(Paged, () => ({ n: constant(4) })); + h.serve(Reader, (_, self) => ({ twice: self.derive([self.get(Paged, { id: 1 }).n], ([n]) => n * 2) })); + }); + const reader = await host.get(Reader, {}, context); + expect(reader.twice.value).toBe(8); + for (const object of attempted) { + const paged = object.deref(); + if (paged?.label === "app.paged(1)") paged.end(); + } + expect(() => reader.twice.value).toThrow(expect.objectContaining({ code: "released" })); + await host.close(context); + }); +}); + +describe("member sources and memory", () => { + it("a failed attempt is collected after a retry while the object, its member source and a retained watch live", async () => { + const attempts: WeakRef[] = []; + const { host } = await loaded((h) => + h.serve(Store, (_, self) => { + attempts.push(new WeakRef(self)); + const list = self.state(List); + return { + list, + count: self.derive([list], ([l]) => l.items.length), + add: async ({ text }, ctx) => { + await list.change((draft) => void draft.items.push(text), ctx); + return null; + }, + }; + }), + ); + const store = await host.get(Store, {}, context); + const frames = changesOf(await store.count.watch(context)); + failLive(attempted, attempts[0]!.deref()!, new ChordError({ code: "unavailable", message: "boom" })); + store.retry(); + (await host.get(Store, {}, context)).release(); + await store.add({ text: "a" }, context); + await delivered(); + expect(valuesOf(frames)).toEqual([1]); + await expectCollected(attempts[0]!); + expect(attempts[1]!.deref()).toBeDefined(); + await host.close(context); + }); + + it("an ended object's member sources are collected with it", async () => { + const refs = await (async () => { + const { host, store } = await served(); + const source = resolveWatchable(store.count); + expect(source?.value).toBe(0); + const ref = new WeakRef(source!); + await host.close(context); + return [ref]; + })(); + await expectCollected(...refs); + }); +}); diff --git a/packages/chord/test/next/host/members.test.ts b/packages/chord/test/next/host/members.test.ts new file mode 100644 index 000000000..2efc060e0 --- /dev/null +++ b/packages/chord/test/next/host/members.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method } from "../../../src/next/contracts/index.ts"; +import { loaded } from "../support.ts"; + +describe("the member check (Q14)", () => { + it("refuses an inherited property for a member", async () => { + const Named = local("app.named", { version: 1, params: { id: int() }, members: { toString: method() } }); + const { host } = await loaded((h) => h.serve(Named, () => ({}) as never)); + await expect(host.get(Named, { id: 1 }, context)).rejects.toThrow( + "app.named(1): toString: a method is a function", + ); + }); + + it("names a derive in errors by its member when exposed, else by its position", async () => { + const Named = local("app.named", { version: 1, params: { n: int() }, members: { label: int() } }); + const bad = { value: 1 }; + const { host } = await loaded((h) => + h.serve(Named, ({ n }, self) => { + const inner = self.derive(n === 1 ? [] : [bad as never], () => 1); + return { label: self.derive(n === 1 ? [inner, bad as never] : [inner], () => 2) }; + }), + ); + await expect(host.get(Named, { n: 1 }, context)).rejects.toThrow( + "app.named(1): label: source 1 is not a JSON watchable", + ); + await expect(host.get(Named, { n: 2 }, context)).rejects.toThrow( + "app.named(2): derive #1: source 0 is not a JSON watchable", + ); + }); +}); diff --git a/packages/chord/test/next/host/native.test.ts b/packages/chord/test/next/host/native.test.ts new file mode 100644 index 000000000..d2ad9ae97 --- /dev/null +++ b/packages/chord/test/next/host/native.test.ts @@ -0,0 +1,244 @@ +import { describe, expect, expectTypeOf, it, vi } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local } from "../../../src/next/contracts/index.ts"; +import { isChordError } from "../../../src/next/data/index.ts"; +import type { FacetHost, Native } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { failures, fakeTimers, flushed, loaded, rejection } from "../support.ts"; + +const Res = local("app.res", { version: 1, params: { id: int() }, members: { n: int() } }); + +describe("opening", () => { + it("opens one after another in declaration order; a later opener reads an earlier handle", async () => { + const events: string[] = []; + const first = Promise.withResolvers(); + const { host } = await loaded((h) => + h.serve(Res, (_, self) => { + const a = self.open( + async () => { + events.push("open a"); + return first.promise; + }, + () => {}, + ); + self.open( + async () => { + events.push(`open b with ${a.value}`); + return "b"; + }, + () => {}, + ); + return { n: constant(1) }; + }), + ); + const got = host.get(Res, { id: 1 }, context); + await flushed(); + expect(events).toEqual(["open a"]); + first.resolve("a"); + await got; + expect(events).toEqual(["open a", "open b with a"]); + }); + + it("opens nothing for an object ended before its first open started", async () => { + const opens = vi.fn(async () => null); + const { host } = await loaded((h) => + h.serve(Res, (_, self) => { + self.open(opens, () => {}); + return { n: constant(1) }; + }), + ); + host.hold(Res, { id: 1 }, context).release(); + await flushed(); + expect(opens).not.toHaveBeenCalled(); + }); + + it("throws not_ready before its open resolved and released once its close started", async () => { + const reads: unknown[] = []; + const read = (native: Native): void => { + try { + reads.push(native.value); + } catch (error) { + reads.push(isChordError(error) ? error.toJSON() : error); + } + }; + const { host } = await loaded((h) => + h.serve(Res, (_, self) => { + const a: Native = self.open( + async () => { + read(b); + return "a"; + }, + () => read(a), + ); + const b: Native = self.open( + async () => "b", + () => read(a), + ); + read(a); + return { n: constant(1) }; + }), + ); + (await host.get(Res, { id: 1 }, context)).release(); + await flushed(); + expect(reads).toEqual([ + // In the factory, and b in a's opener. + { code: "not_ready", message: "app.res(1) native #1: not open yet" }, + { code: "not_ready", message: "app.res(1) native #2: not open yet" }, + // a in b's close (b closes first), then in a's own close. + "a", + { code: "released", reason: "closed", message: "app.res(1) native #1: closed" }, + ]); + }); + + it("a partly failed attempt closes what it opened, with reason end", async () => { + const closes: string[] = []; + const { host, errors } = await loaded((h) => + h.serve(Res, (_, self) => { + self.open( + async () => "a", + (value, reason) => void closes.push(`${value} ${reason.kind}`), + ); + self.open( + async () => { + throw new Error("offline"); + }, + () => void closes.push("b"), + ); + return { n: constant(1) }; + }), + ); + expect(await rejection(host.get(Res, { id: 1 }, context, { readyTimeout: 10 }))).toEqual({ + code: "unavailable", + message: "app.res(1): offline", + }); + expect(closes).toEqual(["a end"]); + expect(failures(errors)).toEqual(["app.res(1): transient"]); + await host.close(context); + }); + + it.each(["end", "close"])( + "aborts the opener's context when the object ends while opening; a late value is closed once (%s)", + async (reason) => { + const opener = Promise.withResolvers(); + const late = Promise.withResolvers(); + const closes: unknown[] = []; + const { host } = await loaded((h) => + h.serve(Res, (_, self) => { + self.open( + async (ctx) => { + opener.resolve(ctx); + return late.promise; + }, + // The close runs under the object's context, not under the opener's aborted one. + (value, reason, ctx) => void closes.push([value, reason.kind, ctx.abortSignal?.aborted]), + ); + return { n: constant(1) }; + }), + ); + const { context: cancellable, cancel } = withCancel(context); + const got = host.get(Res, { id: 1 }, cancellable); + const openContext = await opener.promise; + if (reason === "end") cancel(new Error("left")); + else await host.close(context); + await expect(got).rejects.toThrow(reason === "end" ? "left" : "app.res(1): ended"); + expect(openContext.abortSignal?.aborted).toBe(true); + late.resolve("value"); + await flushed(); + await host.close(context); + expect(closes).toEqual([["value", reason, false]]); + }, + ); +}); + +describe("closing", () => { + it("closes in reverse, with reason end when the object ends and close when the host closes", async () => { + const closes: string[] = []; + const { host } = await loaded((h) => + h.serve(Res, ({ id }, self) => { + for (const name of ["a", "b"]) { + self.open( + async () => `${id}${name}`, + (value, reason) => void closes.push(`${value} ${reason.kind}`), + ); + } + return { n: constant(id) }; + }), + ); + const one = await host.get(Res, { id: 1 }, context); + await host.get(Res, { id: 2 }, context); + one.release(); + await flushed(); + await host.close(context); + expect(closes).toEqual(["1b end", "1a end", "2b close", "2a close"]); + }); + + it("collects every throwing close in host.close; an object ending in the background reports each", async () => { + const thrown = (name: string) => new Error(`close ${name} threw`); + const { host, errors } = await loaded((h) => + h.serve(Res, ({ id }, self) => { + for (const name of ["a", "b"]) { + self.open( + async () => name, + () => { + throw thrown(`${id}${name}`); + }, + ); + } + return { n: constant(id) }; + }), + ); + const one = await host.get(Res, { id: 1 }, context); + await host.get(Res, { id: 2 }, context); + one.release(); + await flushed(); + expect(errors).toEqual([ + { kind: "close_failed", object: "app.res(1)", native: { kind: "native", position: 2 }, error: thrown("1b") }, + { kind: "close_failed", object: "app.res(1)", native: { kind: "native", position: 1 }, error: thrown("1a") }, + ]); + const closed = await host.close(context).catch((error: unknown) => error); + expect(closed).toBeInstanceOf(AggregateError); + expect((closed as AggregateError).errors).toEqual([thrown("2b"), thrown("2a")]); + }); + + describe("timeouts", () => { + fakeTimers(); + + it("report a close still running after its closeTimeout, abort its context then, and close the others", async () => { + const events: string[] = []; + let slow: Context | undefined; + const { host, errors } = await loaded( + (h) => + h.serve(Res, (_, self) => { + self.open( + async () => "a", + () => void events.push("close a"), + ); + self.open( + async () => "b", + (_value, _reason, ctx) => { + slow = ctx; + return new Promise(() => {}); + }, + { closeTimeout: 500 }, + ); + return { n: constant(1) }; + }), + { closeTimeout: 10_000 }, + ); + (await host.get(Res, { id: 1 }, context)).release(); + await vi.advanceTimersByTimeAsync(499); + expect([events, slow?.abortSignal?.aborted]).toEqual([[], false]); + await vi.advanceTimersByTimeAsync(1); + expect([events, slow?.abortSignal?.aborted]).toEqual([["close a"], true]); + expect(errors).toEqual([ + { kind: "close_timeout", object: "app.res(1)", native: { kind: "native", position: 2 } }, + ]); + }); + }); +}); + +describe("facets", () => { + it("cannot open natives", () => { + expectTypeOf().not.toHaveProperty("open"); + }); +}); diff --git a/packages/chord/test/next/host/object.test.ts b/packages/chord/test/next/host/object.test.ts new file mode 100644 index 000000000..2df9253e8 --- /dev/null +++ b/packages/chord/test/next/host/object.test.ts @@ -0,0 +1,1047 @@ +import { describe, expect, it, vi } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { broken, ChordError } from "../../../src/next/data/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import type { Self } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart, type Message } from "../../../src/next/link/internal.ts"; +import { constant, type Watchable } from "../../../src/next/state/index.ts"; +import { + changesOf, + delivered, + expectCollected, + failLive, + failures, + fakeTimers, + flushed, + holdNextCommit, + instancesOf, + loaded, + memoryStore, + metered, + RawPeer, + rawWorkers, + rejection, + relayed, + restartable, + Session, + StoreRestartable, + storeRestartable, + testCodec, + transported, + valuesOf, +} from "../support.ts"; + +const List = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); +const COUNTER = { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) } as const; +const Store = local("app.store", { + version: 1, + params: {}, + members: { count: int(), add: method(obj({ text: str() })), clear: method() }, +}); +const Page = local("app.page", { version: 1, params: { id: int() }, members: { label: str(), slow: method() } }); + +/** The store: a list document, its count, and methods to add and clear. */ +function serveStore(self: Self) { + const list = self.state(List); + return { + count: self.derive([list], ([l]) => l.items.length), + add: async ({ text }: { text: string }, ctx: Context) => { + await list.change((draft) => draft.items.push(text), ctx); + return null; + }, + clear: async (ctx: Context) => { + await list.retire(ctx); + return null; + }, + }; +} + +const Svc = local("app.svc", { version: 1, params: { id: int() }, members: { n: int() } }); +const Core = local("app.core", { version: 1, params: {}, members: { n: int() } }); + +/** + * A factory with one native whose open throws "offline" on the attempts `fails` picks (counted from 1), and the times + * (fake, from 0) of its attempts. `gate`, when given, holds every open that does not fail until it resolves. + */ +function flaky(fails: (attempt: number) => boolean, gate?: Promise) { + const times: number[] = []; + const factory = (_: unknown, self: Self) => { + times.push(Date.now()); + const attempt = times.length; + self.open( + async () => { + if (fails(attempt)) throw new Error("offline"); + await gate; + return null; + }, + () => {}, + ); + return { n: constant(attempt) }; + }; + return { times, factory }; +} + +describe("identity and lifetime", () => { + it("one object per params key; a params type lives from its first get to its last release", async () => { + const factory = vi.fn(({ id }: { id: number }, self: Self) => { + const store = self.get(Store, {}); + return { + label: self.derive([store.count], ([count]) => `${id}: ${count}`), + slow: async () => null, + }; + }); + const { host } = await loaded((h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Page, factory); + }); + const [a, b, c] = await Promise.all([ + host.get(Page, { id: 1 }, context), + host.get(Page, { id: 1 }, context), + host.get(Page, { id: 2 }, context), + ]); + expect(factory).toHaveBeenCalledTimes(2); + const watch = await a.label.watch(context); + a.release(); + expect(b.label.value).toBe("1: 0"); + b.release(); + expect(await watch.closed).toEqual({ kind: "ended" }); + const again = await host.get(Page, { id: 1 }, context); + expect(factory).toHaveBeenCalledTimes(3); + expect([again.label.value, c.label.value]).toEqual(["1: 0", "2: 0"]); + }); + + it("keeps a type with empty params across releases", async () => { + const factory = vi.fn((_: object, self: Self) => serveStore(self)); + const { host } = await loaded((h) => h.serve(Store, factory)); + const first = await host.get(Store, {}, context); + await first.add({ text: "milk" }, context); + first.release(); + const second = await host.get(Store, {}, context); + expect(second.count.value).toBe(1); + expect(factory).toHaveBeenCalledTimes(1); + }); + + it("drops an ended object's documents and detaches its derives from other objects' members", async () => { + const telemetry = await metered(); + const { host } = await loaded( + (h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Page, ({ id }, self) => ({ + label: self.derive([self.get(Store, {}).count], ([count]) => `${id}: ${count}`), + slow: async () => null, + })); + }, + { telemetry: telemetry.telemetry }, + ); + const store = await host.get(Store, {}, context); + const page = await host.get(Page, { id: 1 }, context); + const watch = await page.label.watch(context); + page.release(); + expect(await watch.closed).toEqual({ kind: "ended" }); + const before = telemetry.counts()["chord.derive.computes"]; + await store.add({ text: "x" }, context); + expect(telemetry.counts()["chord.derive.computes"]).toBe(before); + }); +}); + +describe("retries", () => { + fakeTimers(); + + it("retry a transient failure after 1 s, doubling up to 60 s, and start over after a success", async () => { + const { times, factory } = flaky((attempt) => attempt <= 8 || attempt === 10); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const got = host.get(Svc, { id: 1 }, context, { readyTimeout: 1_000_000 }); + await vi.advanceTimersByTimeAsync(183_000); + (await got).release(); + expect(times).toEqual([0, 1000, 3000, 7000, 15000, 31000, 63000, 123000, 183000]); + // The success deleted the backoff: a new object tries at once and waits 1 s after its failure. + const again = host.get(Svc, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(1000); + expect((await again).n.value).toBe(11); + expect(times.slice(9)).toEqual([183000, 184000]); + }); + + it("share one attempt among concurrent gets", async () => { + const { times, factory } = flaky(() => false); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const refs = await Promise.all([1, 2, 3].map(() => host.get(Svc, { id: 1 }, context))); + expect([times, refs.map(({ n }) => n.value)]).toEqual([[0], [1, 1, 1]]); + }); + + it("never retry a broken failure; a new object after its end tries at once", async () => { + const bug = vi.fn(() => { + throw new Error("bug"); + }); + const { host } = await loaded((h) => { + h.serve(Svc, bug); + h.serve(Core, bug); + }); + expect(await rejection(host.get(Svc, { id: 1 }, context))).toEqual({ + code: "unavailable", + message: "app.svc(1): bug", + }); + await vi.advanceTimersByTimeAsync(120_000); + expect(bug).toHaveBeenCalledTimes(2); + expect(await rejection(host.get(Core, {}, context))).toEqual({ code: "unavailable", message: "app.core: bug" }); + await expect(host.get(Svc, { id: 1 }, context)).rejects.toThrow("bug"); + expect(bug).toHaveBeenCalledTimes(3); + }); + + it("end a failed object with its last reference; its backoff entry expires on its own", async () => { + const { times, factory } = flaky(() => true); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const first = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 500 })); + await vi.advanceTimersByTimeAsync(500); + expect(await first).toEqual({ code: "unavailable", message: "app.svc(1): offline" }); + await vi.advanceTimersByTimeAsync(4500); + expect([times, vi.getTimerCount()]).toEqual([[0], 0]); + // The entry expired: a new object tries at once and its window starts over at 1 s. + const second = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 1500 })); + await vi.advanceTimersByTimeAsync(1500); + await second; + expect(times).toEqual([0, 5000, 6000]); + }); + + it("make a get right after a failure wait for the scheduled retry", async () => { + const { times, factory } = flaky((attempt) => attempt === 1); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const first = rejection(host.get(Svc, { id: 1 }, context, { readyTimeout: 500 })); + await vi.advanceTimersByTimeAsync(500); + await first; + const second = host.get(Svc, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(499); + expect(times).toEqual([0]); + await vi.advanceTimersByTimeAsync(1); + expect((await second).n.value).toBe(2); + expect(times).toEqual([0, 1000]); + }); + + it("start a new instance only after the previous one closed", async () => { + const closing = Promise.withResolvers(); + const factory = vi.fn((_: { id: number }, self: Self) => { + self.open( + async () => null, + () => closing.promise, + ); + return { n: constant(factory.mock.calls.length) }; + }); + const { host } = await loaded((h) => h.serve(Svc, factory)); + (await host.get(Svc, { id: 1 }, context)).release(); + const again = host.get(Svc, { id: 1 }, context); + await flushed(); + expect(factory).toHaveBeenCalledTimes(1); + closing.resolve(); + expect((await again).n.value).toBe(2); + }); +}); + +describe("dependency recovery (R18)", () => { + fakeTimers(); + const View = local("app.view", { version: 1, params: { id: int() }, members: { n: int() } }); + + /** A view over `Type`'s `n`, and the times of its attempts. */ + function view(Type: typeof Svc | typeof Core) { + const times: number[] = []; + const factory = ({ id }: { id: number }, self: Self) => { + times.push(Date.now()); + return { n: Type === Core ? self.get(Core, {}).n : self.get(Svc, { id }).n }; + }; + return { times, factory }; + } + + it("fails a dependent transient with the dependency's cause and retries it at once when it goes live", async () => { + const core = flaky((attempt) => attempt === 1); + const viewer = view(Core); + const { host, errors } = await loaded((h) => { + h.serve(Core, core.factory); + h.serve(View, viewer.factory); + }); + await vi.advanceTimersByTimeAsync(500); + const got = host.get(View, { id: 1 }, context); + await flushed(); + const failed = errors.find((error) => error.kind === "activation_failed" && error.object === "app.view(1)"); + expect(failed).toMatchObject({ + failure: { + kind: "transient", + error: { code: "unavailable", message: "app.view(1): dependency app.core failed" }, + }, + }); + expect(failed?.kind === "activation_failed" && failed.failure.error.cause).toMatchObject({ + message: "app.core: offline", + }); + // The core's retry at 1 s succeeds; the view is retried then, not at the end of its own window (1.5 s). + await vi.advanceTimersByTimeAsync(500); + expect((await got).n.value).toBe(2); + await vi.advanceTimersByTimeAsync(5000); + expect(viewer.times).toEqual([500, 1000]); + }); + + it("keeps the dependency while failed; releasing the dependent releases it", async () => { + const svc = flaky(() => true); + const viewer = view(Svc); + const { host } = await loaded((h) => { + h.serve(Svc, svc.factory); + h.serve(View, viewer.factory); + }); + const got = rejection(host.get(View, { id: 1 }, context, { readyTimeout: 1500 })); + await vi.advanceTimersByTimeAsync(1500); + expect(await got).toEqual({ code: "unavailable", message: "app.view(1): dependency app.svc(1) failed" }); + await vi.advanceTimersByTimeAsync(10_000); + // Held by the failed view, the dependency was retried at 1 s; released with it at 1.5 s, it is not any more. + expect(svc.times).toEqual([0, 1000]); + }); + + it("binds a retry's references before releasing the previous attempt's", async () => { + const svc = flaky((attempt) => attempt === 1); + const viewer = view(Svc); + const { host } = await loaded((h) => { + h.serve(Svc, svc.factory); + h.serve(View, viewer.factory); + }); + const got = host.get(View, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(1000); + // Held only by the view: released first, the live dependency would have ended and started anew. + expect((await got).n.value).toBe(2); + expect([svc.times, viewer.times]).toEqual([ + [0, 1000], + [0, 1000], + ]); + }); + + it("ignores a trigger while an attempt runs: simultaneous triggers give one attempt", async () => { + const gate = Promise.withResolvers(); + const svc = flaky((attempt) => attempt === 1, gate.promise); + const viewer = view(Svc); + const { host } = await loaded((h) => { + h.serve(Svc, svc.factory); + h.serve(View, viewer.factory); + }); + const got = host.get(View, { id: 1 }, context); + // At 1 s both windows end: the dependency's retry waits at its gate, the view's waits for it. + await vi.advanceTimersByTimeAsync(1000); + gate.resolve(); + expect((await got).n.value).toBe(2); + expect(viewer.times).toEqual([0, 1000]); + }); + + it("retries a dependent of a broken dependency with its backoff, for as long as it is held", async () => { + const viewer = view(Core); + const { host } = await loaded((h) => { + h.serve(Core, () => { + throw new Error("bug"); + }); + h.serve(View, viewer.factory); + }); + const got = rejection(host.get(View, { id: 1 }, context, { readyTimeout: 16_000 })); + await vi.advanceTimersByTimeAsync(16_000); + expect(await got).toEqual({ code: "unavailable", message: "app.view(1): dependency app.core failed" }); + expect(viewer.times).toEqual([0, 1000, 3000, 7000, 15000]); + }); + + it("fails every object of a cycle broken; none is retried", async () => { + const Ping = local("app.ping", { version: 1, params: {}, members: { n: int() } }); + const Pong = local("app.pong", { version: 1, params: {}, members: { n: int() } }); + const factories = { ping: vi.fn(), pong: vi.fn() }; + const { host, errors } = await loaded((h) => { + h.serve(Ping, (_, self) => { + factories.ping(); + return { n: self.get(Pong, {}).n }; + }); + h.serve(Pong, (_, self) => { + factories.pong(); + return { n: self.get(Ping, {}).n }; + }); + }); + expect(failures(errors).sort()).toEqual(["app.ping: broken", "app.pong: broken"]); + await vi.advanceTimersByTimeAsync(120_000); + expect([factories.ping.mock.calls.length, factories.pong.mock.calls.length]).toEqual([1, 1]); + expect(await rejection(host.get(Pong, {}, context))).toEqual({ + code: "invalid", + reason: "cycle", + message: "dependency cycle: app.ping → app.pong → app.ping", + }); + }); +}); + +describe("cycle checks", () => { + it("stay linear on shared dependencies (a layered graph, every object depending on the whole next layer)", async () => { + const Layer = local("app.layer", { + version: 1, + params: { layer: int(), index: int() }, + members: { depth: int() }, + }); + const layers = 28; + const { host } = await loaded((h) => + h.serve(Layer, ({ layer }, self) => { + if (layer === layers) return { depth: constant(0) }; + const next = [0, 1].map((index) => self.get(Layer, { layer: layer + 1, index }).depth); + return { depth: self.derive(next, ([a]) => (a as number) + 1) }; + }), + ); + const telemetry = await metered(); + const top = await host.get(Layer, { layer: 0, index: 0 }, telemetry.context); + expect(top.depth.value).toBe(layers); + // 55 objects, 2 waits each: a search per wait visits at most every object once (an exponential one 2^28 chains). + expect(telemetry.counts()["chord.cycle.visits"]).toBeLessThanOrEqual(2 * 55 * 55); + }); +}); + +describe("GC", () => { + it("collects an ended object with its documents, derives and watches", async () => { + const refs: WeakRef[] = []; + const { host } = await loaded((h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Page, (_, self) => { + const list = self.state(List); + const label = self.derive([list, self.get(Store, {}).count], ([l, c]) => `${l.items.length}/${c}`); + refs.push(new WeakRef(self), new WeakRef(list), new WeakRef(label)); + return { label, slow: async () => null }; + }); + }); + const store = await host.get(Store, {}, context); + await (async () => { + const page = await host.get(Page, { id: 1 }, context); + const watch = await page.label.watch(context); + refs.push(new WeakRef(watch), new WeakRef(page)); + page.release(); + })(); + await store.add({ text: "x" }, context); + await delivered(); + await expectCollected(...refs); + }); + + it("keeps nothing in a released reference or a stopped watch", async () => { + const { host } = await loaded((h) => h.serve(Store, (_, self) => serveStore(self))); + const store = await host.get(Store, {}, context); + const held = await host.get(Store, {}, context); + const refs = await (async () => { + const watch = await store.count.watch(context); + await watch.stop(); + const reference = await host.get(Store, {}, context); + const watched: Watchable = reference.count; + await watched.watch(context); + reference.release(); + return [new WeakRef(watch), new WeakRef(reference)]; + })(); + await held.add({ text: "x" }, context); + await expectCollected(...refs); + }); +}); + +describe("GC of released references", () => { + it("keeps neither the declaring object nor the host in a released self.get reference", async () => { + const { reference, refs } = await (async () => { + const refs: WeakRef[] = []; + let declared: unknown; + const { host } = await loaded((h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Page, (_, self) => { + declared = self.get(Store, {}); + refs.push(new WeakRef(self)); + return { label: constant(""), slow: async () => null }; + }); + }); + (await host.get(Page, { id: 1 }, context)).release(); + await flushed(); + refs.push(new WeakRef(host)); + return { reference: declared, refs }; + })(); + await expectCollected(...refs); + expect(reference).toBeDefined(); + }); +}); + +describe("GC of failed objects", () => { + fakeTimers(); + + it("collects one after its last reference while its backoff entry exists; the entry expires on its own", async () => { + const refs: WeakRef[] = []; + const { host, errors } = await loaded((h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Page, (_, self) => { + const list = self.state(List); + const label = self.derive([list, self.get(Store, {}).count], ([l, c]) => `${l.items.length}/${c}`); + const native = self.open( + async () => ({ buffer: new Uint8Array(16) }), + () => {}, + ); + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + const slow = async () => null; + refs.push(new WeakRef(self), new WeakRef(list), new WeakRef(label), new WeakRef(native), new WeakRef(slow)); + return { label, slow }; + }); + }); + const got = rejection(host.get(Page, { id: 1 }, context, { readyTimeout: 100 })); + await vi.advanceTimersByTimeAsync(100); + await got; + expect(vi.getTimerCount()).toBe(1); + // The reports' errors keep their stack frames' receivers until formatted (V8): an app logs and drops them. + errors.length = 0; + await expectCollected(...refs); + await vi.advanceTimersByTimeAsync(900); + expect(vi.getTimerCount()).toBe(0); + }); +}); + +describe("ref.retry()", () => { + fakeTimers(); + + it("skips a transient failure's backoff window: an attempt now", async () => { + const { times, factory } = flaky((attempt) => attempt === 1); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const svc = host.hold(Svc, { id: 1 }, context); + await flushed(); + expect(svc.state.value.kind).toBe("failed"); + vi.advanceTimersByTime(200); + svc.retry(); + await flushed(); + expect([times, svc.state.value, svc.n.value]).toEqual([[0, 200], { kind: "live" }, 2]); + }); + + it("starts an object waiting for its identity's backoff window at once", async () => { + const { times, factory } = flaky((attempt) => attempt === 1); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const first = host.hold(Svc, { id: 1 }, context); + await flushed(); + // The failed object ends with its reference; its identity's window (1 s) still runs. + first.release(); + vi.advanceTimersByTime(100); + const second = host.hold(Svc, { id: 1 }, context); + await flushed(); + expect([times, second.state.value]).toEqual([[0], { kind: "loading" }]); + second.retry(); + await flushed(); + expect([times, second.state.value]).toEqual([[0, 100], { kind: "live" }]); + }); + + it("is ignored while an attempt runs and once live (race 11)", async () => { + const gate = Promise.withResolvers(); + const { times, factory } = flaky((attempt) => attempt === 1, gate.promise); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const svc = host.hold(Svc, { id: 1 }, context); + await flushed(); + svc.retry(); + svc.retry(); + await flushed(); + expect(times).toEqual([0, 0]); + gate.resolve(); + await flushed(); + svc.retry(); + await vi.advanceTimersByTimeAsync(5_000); + expect([times, svc.state.value]).toEqual([[0, 0], { kind: "live" }]); + }); + + it("renews a broken object for every reference (race 13)", async () => { + let fixed = false; + const factory = vi.fn((_: { id: number }, self: Self) => { + self.open( + async () => { + if (!fixed) throw broken(new Error("no configuration")); + }, + () => {}, + ); + return { n: constant(1) }; + }); + const { host } = await loaded((h) => h.serve(Svc, factory)); + const one = host.hold(Svc, { id: 1 }, context); + const two = host.hold(Svc, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(120_000); + expect([one.state.value.kind, two.state.value.kind, factory.mock.calls.length]).toEqual(["failed", "failed", 1]); + fixed = true; + one.retry(); + expect(two.state.value).toEqual({ kind: "loading" }); + await flushed(); + expect([one.state.value, two.state.value]).toEqual([{ kind: "live" }, { kind: "live" }]); + }); +}); + +describe("linger", () => { + fakeTimers(); + const Tally = local("app.tally", { version: 1, params: { id: int() }, members: { n: int(), bump: method() } }); + const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + + /** Serves `Tally` with `linger`: a document, a derive that fails at 2, a native whose closes `closes` records. */ + async function tallies(linger: number, open: () => Promise = async () => {}) { + const closes: string[] = []; + const factory = vi.fn(({ id }: { id: number }, self: Self) => { + const doc = self.state(N); + self.open(open, (_, reason) => void closes.push(`${id} ${reason.kind}`)); + return { + n: self.derive([doc], ([{ n }]) => { + if (n === 2) throw new Error("two"); + return n; + }), + bump: async (ctx: Context) => { + await doc.change((draft) => { + draft.n++; + }, ctx); + return null; + }, + }; + }); + const { host } = await loaded((h) => h.serve(Tally, factory, { linger })); + return { host, factory, closes }; + } + + it("keeps a live object live for linger after its last release: a get within it reuses it, with its documents", async () => { + const { host, factory, closes } = await tallies(500); + const first = await host.get(Tally, { id: 1 }, context); + await first.bump(context); + first.release(); + await vi.advanceTimersByTimeAsync(499); + const second = await host.get(Tally, { id: 1 }, context); + expect([second.n.value, factory.mock.calls.length, closes]).toEqual([1, 1, []]); + second.release(); + await vi.advanceTimersByTimeAsync(500); + expect(closes).toEqual(["1 end"]); + expect((await host.get(Tally, { id: 1 }, context)).n.value).toBe(0); + expect(factory).toHaveBeenCalledTimes(2); + }); + + it("a reuse cancels the expiry: the holder count decides, a later release starts a new window (race 8)", async () => { + const { host, closes } = await tallies(500); + (await host.get(Tally, { id: 1 }, context)).release(); + await vi.advanceTimersByTimeAsync(499); + const held = await host.get(Tally, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(5_000); + held.release(); + await vi.advanceTimersByTimeAsync(499); + expect(closes).toEqual([]); + await vi.advanceTimersByTimeAsync(1); + expect(closes).toEqual(["1 end"]); + }); + + it("only a live object lingers: an activating or failed one ends with its last reference (race 9)", async () => { + const gate = Promise.withResolvers(); + const activating = await tallies(500, () => gate.promise); + const opening = activating.host.hold(Tally, { id: 1 }, context); + await flushed(); + opening.release(); + gate.resolve(); + await flushed(); + // Its open's late value was closed at once. + expect(activating.closes).toEqual(["1 end"]); + const failing = await tallies(500, async () => { + throw broken(new Error("no configuration")); + }); + const failed = failing.host.hold(Tally, { id: 1 }, context); + await flushed(); + failed.release(); + failing.host.hold(Tally, { id: 1 }, context); + await flushed(); + expect(failing.factory).toHaveBeenCalledTimes(2); + }); + + it("a failed state member does not end a lingering object", async () => { + const { host, factory, closes } = await tallies(500); + const tally = await host.get(Tally, { id: 1 }, context); + await tally.bump(context); + await tally.bump(context); + expect(() => tally.n.value).toThrow("two"); + tally.release(); + await vi.advanceTimersByTimeAsync(400); + await host.get(Tally, { id: 1 }, context); + expect([factory.mock.calls.length, closes]).toEqual([1, []]); + }); + + it("host.close ends a lingering object at once", async () => { + const { host, closes } = await tallies(60_000); + (await host.get(Tally, { id: 1 }, context)).release(); + await host.close(context); + expect([closes, vi.getTimerCount()]).toEqual([["1 close"], 0]); + }); + + it("a lingering object is collected after its expiry", async () => { + const refs: WeakRef[] = []; + const { host } = await loaded((h) => + h.serve( + Svc, + ({ id }, self) => { + refs.push(new WeakRef(self)); + return { n: constant(id) }; + }, + { linger: 100 }, + ), + ); + (await host.get(Svc, { id: 1 }, context)).release(); + await vi.advanceTimersByTimeAsync(100); + vi.useRealTimers(); + await expectCollected(...refs); + }); +}); + +/** Every object attempted in this file, weakly: the objects of the cycles among them. */ +const attempted = instancesOf(ChordObject, "attempt"); + +/** On the server (no placement): its activation holds `Far` on the session's worker. */ +const Near = remote("app.near", { version: 1, params: { session: str() }, variants: { v: { state: { n: int() } } } }); + +/** On the session's worker: its activation holds `Near` back on the server, or, with `local`, `Middle`. */ +const Far = remote("app.far", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { v: { state: { n: int() } } }, +}); + +/** On the server, held by `Far`: its activation holds `Hub` on its own host (a cycle closed on one host). */ +const Middle = remote("app.middle", { + version: 1, + params: { session: str() }, + variants: { v: { state: { n: int() } } }, +}); + +/** On the server, a local type (no principal in its identity): its activation holds `Far` as the host. */ +const Hub = local("app.hub", { version: 1, params: { session: str() }, members: { n: int() } }); + +/** `Loop({ here, there })` lives on `here`'s worker and holds `Loop({ here: there, there: here })`. */ +const Loop = remote("app.loop", { + version: 1, + params: { here: str(), there: str() }, + placement: ({ here }) => Session({ id: here }), + variants: { v: { state: { n: int() } } }, +}); + +/** A tab, a server and its workers over trusted route links, serving the cycle types (`Far` holds `back`). */ +async function cyclic(back: "near" | "middle" = "near", trusted = true) { + return relayed({ + trusted, + server: (h) => { + h.serve(Near, ({ session }, _principal, self) => ({ + variant: "v", + members: { n: self.get(Far, { session }).n }, + })); + h.serve(Middle, ({ session }, _principal, self) => ({ + variant: "v", + members: { n: self.get(Hub, { session }).n }, + })); + h.serve(Hub, ({ session }, self) => ({ n: self.get(Far, { session }).n })); + }, + worker: (h) => { + h.serve(Far, ({ session }, _principal, self) => ({ + variant: "v", + members: { n: back === "near" ? self.get(Near, { session }).n : self.get(Middle, { session }).n }, + })); + h.serve(Loop, ({ here, there }, _principal, self) => ({ + variant: "v", + members: { n: self.get(Loop, { here: there, there: here }).n }, + })); + }, + }); +} + +describe("cycles across hosts (Q5)", () => { + it("refuse a hold whose chain names the activation it would bind: server → worker → server", async () => { + const { tab, errors } = await cyclic(); + const error = await rejection(tab.get(Near, { session: "S1" }, context)); + // The worker's hold back acts as the worker (upward boundary): it meets the cycle one turn later, at that + // principal's `Near`. + const message = "dependency cycle: app.near(S1) → … → app.near(S1)"; + expect(error).toEqual({ code: "invalid", reason: "cycle", message }); + // Proposal: every object on the cycle fails broken with the same error, none transient (nothing retries it). + const failed = new Set(failures([...errors.server, ...errors.worker])); + expect([...failed].sort()).toEqual(["app.far(S1): broken", "app.near(S1): broken"]); + }); + + it("find a cycle closed on one host through another: server → worker → server, then a local get", async () => { + const { server } = await cyclic("middle"); + const error = await rejection(server.get(Hub, { session: "S1" }, context)); + expect(error).toEqual({ + code: "invalid", + reason: "cycle", + message: "dependency cycle: app.hub(S1) → … → app.middle(S1) → app.hub(S1)", + }); + }); + + it("carry the chain through a relay (race 7): worker → server → worker → server → worker", async () => { + const { tab, errors } = await cyclic(); + const error = await rejection(tab.get(Loop, { here: "S1", there: "S2" }, context)); + expect(error).toMatchObject({ code: "invalid", reason: "cycle" }); + expect([...new Set(failures(errors.worker))].sort()).toEqual([ + "app.loop(S1|S2): broken", + "app.loop(S2|S1): broken", + ]); + }); + + it("drop the chain from holds arriving on untrusted links, and send none for holds made outside an activation", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers) }); + const [a, b] = memoryLinkPair(); + relay.server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const tab = new RawPeer(b); + const forged = "0190f5a2-7c3e-7d4a-9b1c-2d3e4f5a6b7c"; + tab.send({ ...farHold(0), chain: [forged] }); + await transported(); + await transported(); + const held = await workers[0]!.next("hold"); + expect(held.chain).toBeUndefined(); + relay.tab.hold(Far, { session: "S1" }, context); + expect((await workers[0]!.next("hold")).chain).toBeUndefined(); + }); + + it("never match a restarted object: its new instance has a new id (race 8)", async () => { + // The raw worker holds `Near` on the server itself (as the worker), whose activation holds `Far` back with its + // chain. The raw worker never binds `Far`. + const { worker } = await nearBehindRawWorker(); + const chainOf = async () => (await worker.next("hold")).chain!; + worker.send(nearHold(0)); + const [old] = await chainOf(); + // Released: that instance ends. The next hold makes a new instance with a new id. + worker.send({ kind: "release", hold: 0 }); + worker.send(nearHold(1)); + const [renewed] = await chainOf(); + expect(renewed).not.toBe(old); + // A hold naming the old instance binds to the new one (it is pending there); one naming the new one is a cycle. + worker.send({ ...nearHold(2), chain: [old!] }); + worker.send({ ...nearHold(3), chain: [renewed!] }); + const failed = await worker.next("failed"); + expect(failed).toMatchObject({ + hold: 3, + cause: { kind: "broken", error: { kind: "chord", error: { reason: "cycle" } } }, + }); + await transported(); + expect(worker.received.filter((message) => message.kind === "failed")).toEqual([]); + }); + + it("keep the last 32 entries of a longer chain: the oldest is dropped (a longer cycle goes undetected)", async () => { + const { worker } = await nearBehindRawWorker(); + const forged = Array.from( + { length: 32 }, + (_, index) => `0190f5a2-7c3e-7d4a-9b1c-${String(index).padStart(12, "0")}`, + ); + worker.send({ ...nearHold(0), chain: forged }); + const { chain } = await worker.next("hold"); + expect(chain).toHaveLength(32); + expect(chain!.slice(0, 31)).toEqual(forged.slice(1)); + expect(forged).not.toContain(chain![31]); + }); + + it("miss a cycle entering an activation that started already (documented limitation): both wait, transient", async () => { + const { relay, worker } = await nearBehindRawWorker(); + // The server's `Near` activates for the tab and holds `Far` with its chain. + const getting = rejection(relay.tab.get(Near, { session: "S1" }, context, { readyTimeout: 100 })); + const { chain } = await worker.next("hold"); + // The worker's own activation, started at the same moment, holds that `Near` back with a chain of its own. + worker.send({ ...nearHold(0), chain: ["0190f5a2-7c3e-7d4a-9b1c-000000000001"] }); + await transported(); + expect(chain).toHaveLength(1); + expect(worker.received.filter((message) => message.kind === "failed")).toEqual([]); + expect(await getting).toMatchObject({ code: "unavailable" }); + }); + + it("keep no object, and so no chain, once released (GC)", async () => { + const refs = await (async () => { + const made = attempted.length; + const { tab, server } = await cyclic(); + await rejection(tab.get(Near, { session: "S1" }, context)); + await transported(); + const refs = attempted.slice(made); + expect(refs.length).toBeGreaterThanOrEqual(4); + await tab.close(context); + await server.close(context); + return refs; + })(); + await expectCollected(...refs); + }); +}); + +/** + * A server serving `Near` (its activation holds `Far`) and a raw worker for S1 over a trusted route link, made by a + * first hold of `Far` and with that hold taken. + */ +async function nearBehindRawWorker() { + const workers: RawPeer[] = []; + const relay = await relayed({ + resolve: rawWorkers(workers, { trusted: true }), + server: (h) => + h.serve(Near, ({ session }, _principal, self) => ({ + variant: "v", + members: { n: self.get(Far, { session }).n }, + })), + }); + relay.tab.hold(Far, { session: "S1" }, context); + const worker = await (async () => { + for (let tries = 0; workers[0] === undefined && tries < 10; tries++) await transported(); + return workers[0]!; + })(); + await worker.next("hold"); + return { relay, worker }; +} + +/** A hold of `Far` for S1, as a tab sends it. */ +function farHold(hold: number): Extract { + const { name: type, version, fingerprint } = Far; + const address = { kind: "session", params: { id: "S1" } }; + return { kind: "hold", hold, type, version, fingerprint, params: jsonPart({ session: "S1" }), address }; +} + +/** A hold of `Near` for S1, as a worker sends it to the server. */ +function nearHold(hold: number): Extract { + const { name: type, version, fingerprint } = Near; + return { kind: "hold", hold, type, version, fingerprint, params: jsonPart({ session: "S1" }) }; +} + +describe("a live object failing (slice 6a §1.4)", () => { + it("fails broken: calls reject with the cause, one object_failed report, natives close, then references go", async () => { + const events: string[] = []; + let list: Watchable | undefined; + const { reference, fail, errors } = await restartable( + attempted, + (_, self) => { + list = self.state(List); + const core = self.get(Core, {}); + self.open( + async () => null, + () => { + events.push(`close sees core ${core.n.value}`); + }, + ); + }, + { serve: (h) => h.serve(Core, () => ({ n: constant(1) })) }, + ); + const states: string[] = []; + (await reference.state.watch(context)).start(async (call) => { + if (call.kind === "value") states.push(call.value.kind); + }); + fail(); + const touched = rejection(reference.touch(context)); + expect(await touched).toMatchObject({ code: "unavailable", message: "boom" }); + expect(errors).toEqual([expect.objectContaining({ kind: "object_failed", object: "app.restartable(1)" })]); + // A failure of the sealed attempt is ignored. + fail(); + await flushed(); + expect(events).toEqual(["close sees core 1"]); + expect(errors.length).toBe(1); + expect(list!.value).toEqual({ items: [] }); + await delivered(); + expect(states).toEqual(["failed"]); + }); + + it("keeps its state members' last values for same-host reads; a live dependent's derive over them stays live", async () => { + const Viewer = local("app.count-viewer", { version: 1, params: {}, members: { tens: int() } }); + let store: Self | undefined; + const { host, errors } = await loaded((h) => { + h.serve(Store, (_, self) => { + store = self; + return serveStore(self); + }); + h.serve(Viewer, (_, self) => ({ tens: self.derive([self.get(Store, {}).count], ([n]) => n * 10) })); + }); + const ref = await host.get(Store, {}, context); + await ref.add({ text: "a" }, context); + const viewer = await host.get(Viewer, {}, context); + const tens = changesOf(await viewer.tens.watch(context)); + // A change committed in the turn of the failure: the frozen member computes it. + await ref.add({ text: "b" }, context); + failLive(attempted, store!, new ChordError({ code: "unavailable", message: "boom" })); + expect(ref.count.value).toBe(2); + expect(await rejection(ref.add({ text: "c" }, context))).toMatchObject({ message: "boom" }); + expect(viewer.state.value).toEqual({ kind: "live" }); + expect(viewer.tens.value).toBe(20); + await delivered(); + expect(valuesOf(tens)).toEqual([20]); + expect(errors.map(({ kind }) => kind)).toEqual(["object_failed"]); + // A retry takes the new attempt's members. + ref.retry(); + (await host.get(Store, {}, context)).release(); + expect(ref.count.value).toBe(2); + await host.close(context); + }); + + it("lets a commit admitted before the failure adopt and publish once, derives and dependents included", async () => { + const Counter = defineState("counter", { ...COUNTER, persist: true, checkpointWhen: () => false }); + const Doubled = local("app.doubled", { version: 1, params: {}, members: { double: int() } }); + const Viewer = local("app.double-viewer", { version: 1, params: {}, members: { tens: int() } }); + const store = await memoryStore(); + let self: Self | undefined; + let counter: SingletonHandle<{ n: number }> | undefined; + const { host } = await loaded( + (h) => { + h.serve(Doubled, (_, s) => { + self = s; + counter = s.state(Counter); + return { double: s.derive([counter], ([c]) => c.n * 2) }; + }); + h.serve(Viewer, (_, s) => ({ tens: s.derive([s.get(Doubled, {}).double], ([n]) => n * 10) })); + }, + { store }, + ); + const ref = await host.get(Doubled, {}, context); + const viewer = await host.get(Viewer, {}, context); + const doubles = changesOf(await ref.double.watch(context)); + const tens = changesOf(await viewer.tens.watch(context)); + const held = holdNextCommit(store); + const change = counter!.change((draft) => { + draft.n = 1; + }, context); + await held.entered; + failLive(attempted, self!, new ChordError({ code: "unavailable", message: "boom" })); + held.release(); + await change; + await delivered(); + expect(ref.state.value.kind).toBe("failed"); + expect([counter!.value, ref.double.value, viewer.tens.value]).toEqual([{ n: 1 }, 2, 20]); + expect(valuesOf(doubles)).toEqual([2]); + expect(valuesOf(tens)).toEqual([20]); + await host.close(context); + await store.close(context); + }); + + it("starts a retry's factory only once a commit admitted before the failure adopted", async () => { + const V1 = defineState("counter", { ...COUNTER, persist: true, checkpointWhen: () => false }); + const V2 = defineState("counter", { + schema: obj({ m: int() }), + version: 2, + initial: () => ({ m: 0 }), + migrate: (old) => ({ m: (old as { n: number }).n }), + persist: true, + checkpointWhen: () => false, + }); + const store = await memoryStore(); + const counters: SingletonHandle<{ n: number } | { m: number }>[] = []; + const { host, reference, attempts } = await storeRestartable( + attempted, + (attempt, self) => void counters.push(self.state(attempt === 1 ? V1 : V2)), + { store, closeTimeout: 5 }, + ); + const v1 = counters[0] as SingletonHandle<{ n: number }>; + await v1.change((draft) => { + draft.n = 1; + }, context); + const held = holdNextCommit(store); + const change = v1.change((draft) => { + draft.n = 2; + }, context); + await held.entered; + failLive(attempted, attempts[0]!, new ChordError({ code: "unavailable", message: "boom" })); + // Past the drain: only the admitted step holds the retry. + await new Promise((resolve) => setTimeout(resolve, 20)); + reference.retry(); + await new Promise((resolve) => setTimeout(resolve, 20)); + expect(counters).toHaveLength(1); + held.release(); + await change; + (await host.get(StoreRestartable, {}, context)).release(); + expect(counters[1]!.value).toEqual({ m: 2 }); + await host.close(context); + await store.close(context); + }); + + it("is recovered by ref.retry(): a new attempt, the documents as they were", async () => { + const lists: SingletonHandle<{ items: string[] }>[] = []; + const { restart } = await restartable(attempted, (_, self) => { + lists.push(self.state(List)); + }); + await lists[0]!.change((draft) => void draft.items.push("a"), context); + await restart(); + expect(lists[1]!.value).toEqual({ items: ["a"] }); + }); +}); diff --git a/packages/chord/test/next/host/parked.test.ts b/packages/chord/test/next/host/parked.test.ts new file mode 100644 index 000000000..207da3019 --- /dev/null +++ b/packages/chord/test/next/host/parked.test.ts @@ -0,0 +1,213 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local, method, str } from "../../../src/next/contracts/index.ts"; +import { createHost, type Held } from "../../../src/next/host/index.ts"; +import { + EDITOR, + fakeTimers, + flushed, + forEachCodec, + loaded, + Notes, + partValue, + rawParent, + rawProvider, + rejection, + type TestPrincipal, + transported, + wireNames, +} from "../support.ts"; + +const Gated = local("app.gated", { version: 1, params: { id: int() }, members: { log: method(str(), int()) } }); + +/** + * Serves `Gated`, whose open waits for `gate`; `log` records its argument and returns how many it recorded. In its + * synchronous part a `log` of "release" releases `self.ref`, one of "call" makes the call `log("from a handler")` on + * it, and one of "block" takes 30 ms (the clock moves while no timer can fire). + */ +async function gated() { + const gate = Promise.withResolvers(); + const log: string[] = []; + const self: { ref?: Held } = {}; + const { host } = await loaded((h) => + h.serve(Gated, (_, attempt) => { + attempt.open( + () => gate.promise, + () => {}, + ); + return { + log: async (text) => { + log.push(text); + if (text === "release") self.ref?.release(); + if (text === "call") void self.ref?.log("from a handler", context); + if (text === "block") vi.setSystemTime(Date.now() + 30); + return log.length; + }, + }; + }), + ); + return { host, gate, log, self }; +} + +describe("calls on a loading local reference", () => { + fakeTimers(); + + it("are queued and sent in the order they were made once it is live, before later calls", async () => { + const { host, gate, log } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context); + const first = ref.log("a", context); + const second = ref.log("b", context); + await flushed(); + expect(log).toEqual([]); + gate.resolve(); + await flushed(); + const third = ref.log("c", context); + expect(await Promise.all([first, second, third])).toEqual([1, 2, 3]); + expect(log).toEqual(["a", "b", "c"]); + }); + + it("keep their order when a handler calls the reference while they are sent: its call queues behind them", async () => { + const { host, gate, log, self } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context); + self.ref = ref; + const calls = [ref.log("call", context), ref.log("b", context)]; + gate.resolve(); + await Promise.all(calls); + await flushed(); + expect(log).toEqual(["call", "b", "from a handler"]); + }); + + it("are checked against their deadline when sent: one that passed while earlier ones ran is not sent", async () => { + const { host, gate, log } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context, { readyTimeout: 10 }); + const first = ref.log("block", context); + const second = rejection(ref.log("b", context)); + gate.resolve(); + expect(await first).toBe(1); + expect(await second).toMatchObject({ code: "unavailable", reason: "park_timeout" }); + expect(log).toEqual(["block"]); + }); + + it("reject unavailable/park_timeout at the reference's readyTimeout, measured from each call", async () => { + const { host, gate, log } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context, { readyTimeout: 100 }); + const first = rejection(ref.log("a", context)); + await vi.advanceTimersByTimeAsync(60); + const second = ref.log("b", context); + await vi.advanceTimersByTimeAsync(40); + expect(await first).toEqual({ + code: "unavailable", + reason: "park_timeout", + message: "app.gated(1): a call waited 100 ms for its reference to go live", + }); + gate.resolve(); + expect(await second).toBe(1); + expect(log).toEqual(["b"]); + }); + + it("reject at once when their caller cancels; the others stay queued", async () => { + const { host, gate, log } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context); + const { context: cancellable, cancel } = withCancel(context); + const cancelled = ref.log("a", cancellable); + const kept = ref.log("b", context); + cancel(new Error("user cancelled")); + await expect(cancelled).rejects.toThrow("user cancelled"); + gate.resolve(); + expect(await kept).toBe(1); + expect(log).toEqual(["b"]); + }); + + it("reject released when the reference is released; nothing runs", async () => { + const { host, gate, log } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context); + const call = ref.log("a", context); + ref.release(); + expect(await rejection(call)).toEqual({ code: "released", message: "app.gated: released" }); + gate.resolve(); + await flushed(); + expect(log).toEqual([]); + }); + + it("reject with the cause when the object fails", async () => { + const { host, gate } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context); + const call = ref.log("a", context); + gate.reject(new Error("offline")); + expect(await rejection(call)).toEqual({ code: "unavailable", message: "app.gated(1): offline" }); + }); + + it("stop being sent when one's handler releases the reference: the rest reject released (race 15)", async () => { + const { host, gate, log, self } = await gated(); + const ref = host.hold(Gated, { id: 1 }, context); + self.ref = ref; + const first = ref.log("release", context); + const second = ref.log("b", context); + gate.resolve(); + expect(await rejection(second)).toEqual({ code: "released", message: "app.gated: released" }); + await first.catch(() => {}); + expect(log).toEqual(["release"]); + }); +}); + +forEachCodec(() => { + describe("calls on a loading remote reference", () => { + fakeTimers(); + + it("keep one deadline across reconnects; nothing is sent for them on either link", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context, { readyTimeout: 1_000 }); + const call = rejection(notes.wait(context)); + await vi.advanceTimersByTimeAsync(600); + peer.close(); + await transported(); + const next = await rawParent(client); + await next.next("hold"); + await vi.advanceTimersByTimeAsync(400); + expect(await call).toMatchObject({ code: "unavailable", reason: "park_timeout" }); + expect([peer.kinds(), next.kinds()]).toEqual([["hold"], []]); + }); + + it("are resolved against the binding they are sent on: indexes from the new variant, missing methods invalid", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + // Typed for the editor: before its first binding the reference may become either variant. + const editor = notes as Extract, { add: unknown }>; + const add = rejection(editor.add({ text: "a" }, context)); + const wait = editor.wait(context); + const { hold } = await peer.next("hold"); + const READER = wireNames(Notes, "reader"); + const members = [ + { member: READER.state.notes!, value: "[]" }, + { member: READER.state.count!, value: "0" }, + ]; + peer.send({ kind: "bound", holds: [{ hold, variant: READER.variant, members, failed: [] }] }); + expect(await add).toEqual({ + code: "invalid", + reason: "member", + message: 'app.notes(S1) (variant reader) has no method "add"', + }); + const sent = await peer.next("call"); + expect(sent.method).toBe(READER.methods.wait); + expect(sent.method).not.toBe(EDITOR.methods.wait); + peer.send({ kind: "result", call: sent.call }); + expect(await wait).toBe(null); + }); + + it("on a reference released while loading without a link: nothing is sent, the next connect holds nothing (race 3)", async () => { + const client = createHost({ onError: () => {} }); + await client.load([], context); + const notes = client.hold(Notes, { session: "S1" }, context); + const other = client.hold(Notes, { session: "S2" }, context); + const call = notes.wait(context); + notes.release(); + expect(await rejection(call)).toEqual({ code: "released", message: "app.notes: released" }); + const peer = await rawParent(client); + const { params } = await peer.next("hold"); + expect(partValue(params)).toEqual({ session: "S2" }); + await transported(); + expect(peer.kinds()).toEqual([]); + expect(other.state.value).toEqual({ kind: "loading" }); + }); + }); +}); diff --git a/packages/chord/test/next/host/presence.test.ts b/packages/chord/test/next/host/presence.test.ts new file mode 100644 index 000000000..6260f9360 --- /dev/null +++ b/packages/chord/test/next/host/presence.test.ts @@ -0,0 +1,328 @@ +// Tests of presence (§19): \`host.references(Type)\`, its entries and incarnations, its batches (a link drop as a departure, +// one discovered while a commit sends lands in the next batch), derives over it, its cost and GC. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { Hold } from "../../../src/next/remote/hold.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + addNote, + delivered, + expectCollected, + framesOf, + incarnationOf, + instancesOf, + linked, + NoteStore, + Notes, + PlacedNotes, + RawPeer, + relayed, + serveNotes, + type TestPrincipal, + testCodec, + transported, + wireNames, +} from "../support.ts"; + +/** Every hold started in this file, weakly. */ +const holds = instancesOf(Hold, "start"); + +describe("host.references", () => { + it("lists every reference with the object's params, its principal and its incarnation", async () => { + const { server, client } = await linked((h) => void serveNotes(h), { child: { kind: "guest", session: "S1" } }); + const notes = server.references(Notes); + expect(notes.value).toEqual([]); + await client.get(Notes, { session: "S1" }, context); + const own = await server.get(Notes, { session: "S2" }, context); + expect(notes.value).toEqual([ + { params: { session: "S1" }, principal: { kind: "guest", session: "S1" }, incarnation: incarnationOf(client) }, + { params: { session: "S2" }, principal: { kind: "server" }, incarnation: incarnationOf(server) }, + ]); + // A local type's entries have no principal: the store is referenced by both views. + expect(server.references(NoteStore).value).toEqual([ + { params: { session: "S1" }, incarnation: incarnationOf(server) }, + { params: { session: "S2" }, incarnation: incarnationOf(server) }, + ]); + own.release(); + expect(notes.value).toHaveLength(1); + }); + + it("is made for the first ask with the references that exist then, and kept", async () => { + const { server, client } = await linked((h) => void serveNotes(h)); + await client.get(Notes, { session: "S1" }, context); + const notes = server.references(Notes); + expect(notes.value).toHaveLength(1); + expect(server.references(Notes)).toBe(notes); + }); + + it("changes in the batch where a reference binds or ends: a watch gets one splice per change", async () => { + const { server, client } = await linked((h) => void serveNotes(h)); + const watch = await server.references(Notes).watch(context); + const frames = framesOf(watch); + const one = await client.get(Notes, { session: "S1" }, context); + await client.get(Notes, { session: "S2" }, context); + one.release(); + await transported(); + await delivered(); + const incarnation = incarnationOf(client); + const entry = (session: string) => ({ params: { session }, principal: { kind: "owner" }, incarnation }); + expect(frames.map(([, ops]) => ops)).toEqual([ + [["p", [], 0, 0, [entry("S1")]]], + [["p", [], 1, 0, [entry("S2")]]], + [["p", [], 0, 1, []]], + ]); + expect(frames.at(-1)![0]).toEqual([entry("S2")]); + }); + + it("feeds derives in factories: viewers per session, a link drop a departure in its batch (race 9)", async () => { + const Viewers = local("app.viewers", { version: 1, params: { session: str() }, members: { count: int() } }); + const Shown = remote("app.shown", { + version: 1, + params: { session: str() }, + variants: { v: { state: { here: int() } } }, + }); + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + await server.load( + [ + { + name: "@test/app", + facet: defineFacet((host) => { + host.serve(Shown, () => ({ variant: "v", members: { here: constant(1) } })); + host.serve(Viewers, ({ session }, self) => ({ + count: self.derive( + [host.references(Shown)], + ([refs]) => refs.filter((ref) => ref.params.session === session).length, + ), + })); + }), + }, + ], + context, + ); + const viewers = await server.get(Viewers, { session: "S1" }, context); + const counts = framesOf(await viewers.count.watch(context)); + const tabs = await Promise.all( + [0, 1].map(async () => { + const tab = createHost({ onError: () => {} }); + await tab.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" } }, context); + tab.connect(b, { principal: { kind: "server" } }, context); + return { tab, link: a }; + }), + ); + await Promise.all(tabs.map(({ tab }) => tab.get(Shown, { session: "S1" }, context))); + await tabs[0]!.tab.get(Shown, { session: "S2" }, context); + expect(viewers.count.value).toBe(2); + tabs[1]!.link.close(); + // The departure is in the batch of the link's close: the derive has it at once. + expect(viewers.count.value).toBe(1); + await delivered(); + expect(counts.at(-1)![0]).toBe(1); + }); + + it("puts a departure discovered while a commit sends into the next batch: one state message per commit (race 9)", async () => { + const Viewed = remote("app.viewed", { + version: 1, + params: { session: str() }, + variants: { v: { state: { count: int(), viewers: int() } } }, + }); + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const facet = defineFacet((host) => { + serveNotes(host); + host.serve(Viewed, ({ session }, _principal, self) => { + const store = self.get(NoteStore, { session }); + const count = self.derive([store.list], ([list]) => list.length); + const viewers = self.derive([host.references(Viewed)], ([refs]) => refs.length); + return { variant: "v", members: { count, viewers } }; + }); + }); + await server.load([{ name: "@test/app", facet }], context); + const names = wireNames(Viewed, "v").state as { count: number; viewers: number }; + /** A raw tab; its server end closes while it sends, once `drop.armed` (a transport failing on send). */ + const tab = (drop?: { armed: boolean }) => { + const [a, b] = memoryLinkPair(); + const end = drop === undefined ? a : closingOnSend(a, drop); + server.accept(end, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const { name: type, version, fingerprint } = Viewed; + peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: jsonPart({ session: "S1" }) }); + return peer; + }; + const drop = { armed: false }; + // The dropped tab binds first, so its link sends (and fails) before the kept one's in the commit's flushes. + const dropped = tab(drop); + await dropped.next("bound"); + const kept = tab(); + await kept.next("bound"); + await transported(); + kept.received.length = 0; + drop.armed = true; + await addNote(server, "S1", "x"); + await transported(); + const states = kept.received.filter((message) => message.kind === "state"); + const changed = states.map(({ holds }) => holds.flatMap(({ members }) => members.map(({ member }) => member))); + // The commit's message carries the commit (the count); the departure is the next batch's (the viewers). + expect(changed).toEqual([[names.count], [names.viewers]]); + expect(dropped.closed).toBe(true); + }); + + it("completes a hold's bookkeeping before its presence entry publishes: a link closing then leaks nothing", async () => { + const Viewed = remote("app.viewed", { + version: 1, + params: {}, + variants: { v: { state: { viewers: int() } } }, + }); + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const facet = defineFacet((host) => { + host.serve( + Viewed, + (_params, _principal, self) => ({ + variant: "v", + members: { viewers: self.derive([host.references(Viewed)], ([refs]) => refs.length) }, + }), + { linger: 60_000 }, + ); + }); + await server.load([{ name: "@test/app", facet }], context); + const [a, b] = memoryLinkPair(); + const drop = { armed: false }; + server.accept(closingOnSend(a, drop), { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const { name: type, version, fingerprint } = Viewed; + peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: "{}" }); + await peer.next("bound"); + peer.send({ kind: "ack" }); + await transported(); + // The second hold's presence entry changes the first's state: sending it closes the link. + drop.armed = true; + peer.send({ kind: "hold", hold: 1, type, version, fingerprint, params: "{}" }); + await transported(); + expect(peer.closed).toBe(true); + expect(server.references(Viewed).value).toEqual([]); + expect(server.activity.value).toMatchObject({ references: 0, lingering: 1 }); + await server.close(context); + }); + + it("completes a release's bookkeeping before its departure publishes: a link closing then leaves no lingering", async () => { + const Viewed = remote("app.viewed", { version: 1, params: {}, variants: { v: { state: { viewers: int() } } } }); + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const facet = defineFacet((host) => { + host.serve( + Viewed, + (_params, _principal, self) => ({ + variant: "v", + members: { viewers: self.derive([host.references(Viewed)], ([refs]) => refs.length) }, + }), + { linger: 50 }, + ); + }); + await server.load([{ name: "@test/app", facet }], context); + const [a, b] = memoryLinkPair(); + const drop = { armed: false }; + server.accept(closingOnSend(a, drop), { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const { name: type, version, fingerprint } = Viewed; + for (const hold of [0, 1]) { + peer.send({ kind: "hold", hold, type, version, fingerprint, params: "{}" }); + await peer.next("bound"); + peer.send({ kind: "ack" }); + } + await transported(); + // The first release changes the second hold's state: sending it closes the link, which releases the second. + drop.armed = true; + peer.send({ kind: "release", hold: 0 }); + await transported(); + expect(peer.closed).toBe(true); + expect(server.activity.value).toMatchObject({ references: 0, lingering: 1 }); + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(server.references(Viewed).value).toEqual([]); + expect(server.activity.value).toMatchObject({ references: 0, lingering: 0 }); + await server.close(context); + }); + + it("removes a bulk departure with one list and one splice op per entry, in one frame (a revoke of many holds)", async () => { + const server = createHost({ + principal: { kind: "server" }, + revokedBy: (principal) => (principal.kind === "guest" ? [{ session: principal.session }] : []), + onError: () => {}, + }); + await server.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "guest", session: "S1" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const { name: type, version, fingerprint } = Notes; + for (let hold = 0; hold < 4; hold++) { + peer.send({ kind: "hold", hold, type, version, fingerprint, params: jsonPart({ session: "S1" }) }); + } + await transported(); + const references = server.references(Notes); + const frames = framesOf(await references.watch(context)); + expect(references.value).toHaveLength(4); + server.revoke({ session: "S1" }, context); + await delivered(); + expect(frames.map(([, ops]) => ops)).toEqual([Array(4).fill(["p", [], 0, 1, []])]); + expect(frames[0]![0]).toEqual([]); + }); + + it("counts two tabs through a relay as two incarnations on the worker", async () => { + const relay = await relayed(); + const second = createHost({ onError: () => {} }); + await second.load([], context); + const [a, b] = memoryLinkPair(); + relay.server.accept(a, { principal: { kind: "owner" } }, context); + second.connect(b, { principal: { kind: "server" } }, context); + await relay.tab.get(PlacedNotes, { session: "S1" }, context); + await second.get(PlacedNotes, { session: "S1" }, context); + const worker = relay.workers.get("session|S1")!; + const incarnations = worker.references(PlacedNotes).value.map(({ incarnation }) => incarnation); + expect(incarnations).toEqual([incarnationOf(relay.tab), incarnationOf(second)]); + }); + + it("has no entries for lingering objects", async () => { + const Kept = remote("app.kept", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const { server, client } = await linked((h) => + h.serve(Kept, () => ({ variant: "v", members: { n: constant(1) } }), { linger: 60_000 }), + ); + const kept = await client.get(Kept, {}, context); + const references = server.references(Kept); + expect(references.value).toHaveLength(1); + kept.release(); + await transported(); + expect(references.value).toEqual([]); + expect(server.activity.value.lingering).toBe(1); + await server.close(context); + }); + + it("keeps no released reference (GC)", async () => { + const { server, refs } = await (async () => { + const { server, client } = await linked((h) => void serveNotes(h)); + const references = server.references(Notes); + const held = holds.length; + const notes = await client.get(Notes, { session: "S1" }, context); + expect(references.value).toHaveLength(1); + const refs = holds.slice(held); + notes.release(); + await transported(); + expect(references.value).toEqual([]); + return { server, refs }; + })(); + await expectCollected(...refs); + expect(server.references(Notes).value).toEqual([]); + }); +}); + +/** `link`, closing itself on a send once `drop.armed`, as a transport whose send fails. */ +function closingOnSend(link: Link, drop: { readonly armed: boolean }): Link { + return { + send: (message) => (drop.armed ? link.close() : link.send(message)), + onMessage: (listener) => link.onMessage(listener), + close: () => link.close(), + onClose: (listener) => link.onClose(listener), + }; +} diff --git a/packages/chord/test/next/host/ref-state.test.ts b/packages/chord/test/next/host/ref-state.test.ts new file mode 100644 index 000000000..d7ab08c47 --- /dev/null +++ b/packages/chord/test/next/host/ref-state.test.ts @@ -0,0 +1,189 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local, method } from "../../../src/next/contracts/index.ts"; +import { broken } from "../../../src/next/data/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + bindHold, + bindNext, + fakeTimers, + flushed, + forEachCodec, + loaded, + Notes, + rawParent, + rawProvider, + rejection, + statesOf, + transported, +} from "../support.ts"; + +const Flaky = local("app.flaky", { + version: 1, + params: { id: int() }, + members: { n: int(), get: method(undefined, int()) }, +}); + +/** Serves `Flaky`, whose open throws while `failing.left` is above 0 (broken while `failing.broken`). */ +async function flaky(failing: { left: number; broken: boolean }) { + return loaded((h) => + h.serve(Flaky, ({ id }, self) => { + self.open( + async () => { + if (failing.left-- <= 0) return; + if (failing.broken) throw broken(new Error("no configuration")); + throw new Error("offline"); + }, + () => {}, + ); + return { n: constant(id), get: async () => id }; + }), + ); +} + +const offline = { code: "unavailable", message: "app.flaky(1): offline" }; + +describe("ref.state of a local reference", () => { + fakeTimers(); + + it("is watched from its current state: failed with retryAt, loading on the next attempt, live; release ends it", async () => { + const { host } = await flaky({ left: 1, broken: false }); + const ref = host.hold(Flaky, { id: 1 }, context); + expect(ref.state.value).toEqual({ kind: "loading" }); + const watch = await ref.state.watch(context); + expect(watch.value).toEqual({ kind: "loading" }); + const states = statesOf(watch); + await flushed(); + expect(ref.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "transient", error: offline }, + retryAt: 1_000, + }); + await vi.advanceTimersByTimeAsync(1_000); + await flushed(); + expect(ref.state.value).toEqual({ kind: "live" }); + ref.release(); + expect(await watch.closed).toEqual({ kind: "ended" }); + expect(states).toEqual(["failed: app.flaky(1): offline", "loading", "live"]); + expect(ref.state.value).toMatchObject({ kind: "ended", error: { code: "released" } }); + }); + + it("reads and calls follow the state: not_ready before live, the cause's error while failed, released after", async () => { + const { host } = await flaky({ left: 1, broken: true }); + const ref = host.hold(Flaky, { id: 1 }, context); + expect(() => ref.n.value).toThrow("app.flaky(1): not live yet"); + await flushed(); + const cause = { code: "unavailable", message: "app.flaky(1): no configuration" }; + expect(ref.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: cause }, + retryAt: undefined, + }); + expect(() => ref.n.value).toThrow("no configuration"); + expect(await rejection(ref.get(context))).toEqual(cause); + ref.release(); + expect(() => ref.n.value).toThrow("app.flaky: released"); + expect(await rejection(ref.get(context))).toEqual({ code: "released", message: "app.flaky: released" }); + }); + + it("retry renews a broken object: loading, then live", async () => { + const { host } = await flaky({ left: 1, broken: true }); + const ref = host.hold(Flaky, { id: 1 }, context); + await flushed(); + expect(ref.state.value.kind).toBe("failed"); + ref.retry(); + expect(ref.state.value).toEqual({ kind: "loading" }); + await flushed(); + expect(ref.state.value).toEqual({ kind: "live" }); + expect(await ref.get(context)).toBe(1); + }); + + it("is made on first access, once; a watch taken after release has ended", async () => { + const { host } = await flaky({ left: 0, broken: false }); + const ref = await host.get(Flaky, { id: 1 }, context); + expect(ref.state).toBe(ref.state); + expect(ref.state.value).toEqual({ kind: "live" }); + ref.release(); + const late = await ref.state.watch(context); + expect(late.value).toMatchObject({ kind: "ended", error: { code: "released" } }); + expect(await late.closed).toEqual({ kind: "ended" }); + }); + + it("a watch ends cancelled with its context", async () => { + const { host } = await flaky({ left: 0, broken: false }); + const ref = host.hold(Flaky, { id: 1 }, context); + const { context: cancellable, cancel } = withCancel(context); + const watch = await ref.state.watch(cancellable); + cancel(); + expect(await watch.closed).toEqual({ kind: "aborted" }); + }); +}); + +forEachCodec(() => { + describe("ref.state of a remote reference", () => { + fakeTimers(); + + it("loading until bound, loading while its link is down, failed with retryAt from retryIn, live again", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + const states = statesOf(await notes.state.watch(context)); + await bindNext(peer); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + peer.close(); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + const next = await rawParent(client); + const { hold } = await next.next("hold"); + const error = { code: "unavailable", message: "offline" } as const; + next.send({ + kind: "failed", + hold, + cause: { kind: "transient", error: { kind: "chord", error } }, + retryIn: 500, + }); + await transported(); + expect(notes.state.value).toMatchObject({ kind: "failed", cause: { kind: "transient", error }, retryAt: 500 }); + next.send({ kind: "loading", hold }); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + bindHold(next, hold); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + expect(states).toEqual(["live", "loading", "failed: offline", "loading", "live"]); + }); + + it("a failed with retryIn 0 retries now: retryAt is the current time", async () => { + const { client, peer } = await rawProvider(); + vi.setSystemTime(2_000); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + const error = { code: "unavailable", message: "offline" } as const; + peer.send({ kind: "failed", hold, cause: { kind: "transient", error: { kind: "chord", error } }, retryIn: 0 }); + await transported(); + expect(notes.state.value).toMatchObject({ kind: "failed", retryAt: 2_000 }); + }); + + it("variant: undefined before the first binding, the last bound one while loading or failed", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + expect(notes.variant).toBeUndefined(); + await bindNext(peer, [{ text: "a" }]); + await transported(); + expect(notes.variant).toBe("editor"); + peer.close(); + await transported(); + expect([notes.variant, notes.count.value]).toEqual(["editor", 1]); + const next = await rawParent(client); + const { hold } = await next.next("hold"); + const error = { code: "unavailable", message: "offline" } as const; + next.send({ kind: "failed", hold, cause: { kind: "broken", error: { kind: "chord", error } } }); + await transported(); + expect(notes.state.value).toMatchObject({ kind: "failed", retryAt: undefined }); + expect(notes.variant).toBe("editor"); + expect(() => notes.count.value).toThrow("offline"); + notes.release(); + expect(() => notes.variant).toThrow("app.notes: released"); + }); + }); +}); diff --git a/packages/chord/test/next/host/reference.test.ts b/packages/chord/test/next/host/reference.test.ts new file mode 100644 index 000000000..adeeb7288 --- /dev/null +++ b/packages/chord/test/next/host/reference.test.ts @@ -0,0 +1,656 @@ +import { describe, expect, expectTypeOf, it, vi } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context, withAbortSignal } from "../../../src/context/index.ts"; +import { int, keyed, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { broken, type ChordError } from "../../../src/next/data/index.ts"; +import { defineState, type KeyedHandle } from "../../../src/next/documents/index.ts"; +import { + createHost, + defineFacet, + type Held, + type Reference, + type RefIn, + type RefState, + type Self, +} from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { RemoteObject } from "../../../src/next/remote/remote-object.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import type { Store } from "../../../src/next/store/index.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import type { NativeWatchable, Watchable } from "../../../src/watch.ts"; +import { + changesOf, + delivered, + expectCollected, + flushed, + framesOf, + instancesOf, + loaded, + memoryStore, + Notes, + rejection, + settledHeap, +} from "../support.ts"; + +/** Every remote object bound while this file runs, weakly. */ +const remoteObjects = instancesOf(RemoteObject, "hold"); +const chordObjects = instancesOf(ChordObject, "attempt"); + +const Doc = defineState("doc", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const Counter = local("app.counter", { version: 1, params: {}, members: { doc: obj({ n: int() }), bump: method() } }); +const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: { n: int() } }); + +/** A counter: one document, exposed, and a method that increments it. */ +function serveCounter(self: Self) { + const doc = self.state(Doc); + return { + doc, + bump: async (ctx: typeof context) => { + await doc.change((draft) => { + draft.n++; + }, ctx); + return null; + }, + }; +} + +describe("references", () => { + it("are hydrated when got; their state members are watchables", async () => { + const { host } = await loaded((h) => h.serve(Counter, (_, self) => serveCounter(self))); + const counter = await host.get(Counter, {}, context); + expect(counter.doc.value).toEqual({ n: 0 }); + const frames = framesOf(await counter.doc.watch(context)); + await counter.bump(context); + await delivered(); + expect(frames).toEqual([[{ n: 1 }, [["s", ["n"], 1]]]]); + expect(counter.doc.value).toEqual({ n: 1 }); + }); + + it("throw released on every use after release, which is idempotent", async () => { + const { host } = await loaded((h) => h.serve(Counter, (_, self) => serveCounter(self))); + const counter = await host.get(Counter, {}, context); + counter.release(); + counter.release(); + const released = { code: "released", message: "app.counter: released" }; + expect(() => counter.doc.value).toThrow("app.counter: released"); + expect(await rejection(counter.doc.watch(context))).toEqual(released); + expect(await rejection(counter.bump(context))).toEqual(released); + }); + + it("end their watches when released, while other references keep the object", async () => { + const { host } = await loaded((h) => h.serve(Counter, (_, self) => serveCounter(self))); + const first = await host.get(Counter, {}, context); + const second = await host.get(Counter, {}, context); + const one = await first.doc.watch(context); + const two = await second.doc.watch(context); + first.release(); + expect(await one.closed).toEqual({ kind: "ended" }); + const frames = framesOf(two); + await second.bump(context); + await delivered(); + expect(frames.map(([value]) => value)).toEqual([{ n: 1 }]); + }); + + it("declared with self.get keep their object alive and are released when the declaring object ends", async () => { + const Inner = local("app.inner", { version: 1, params: { id: int() }, members: { n: int() } }); + const inner = vi.fn(() => ({ n: constant(1) })); + const { host } = await loaded((h) => { + h.serve(Inner, inner); + h.serve(Viewer, ({ id }, self) => ({ n: self.get(Inner, { id }).n })); + }); + const viewer = await host.get(Viewer, { id: 1 }, context); + const direct = await host.get(Inner, { id: 1 }, context); + direct.release(); + expect(viewer.n.value).toBe(1); + expect(inner).toHaveBeenCalledTimes(1); + viewer.release(); + // The viewer's teardown releases its reference in the background. + await flushed(); + await host.get(Inner, { id: 1 }, context); + expect(inner).toHaveBeenCalledTimes(2); + }); +}); + +describe("cycles (Q5)", () => { + const A = local("a", { version: 1, params: { n: int() }, members: { v: str() } }); + const B = local("b", { version: 1, params: { n: int() }, members: { v: str() } }); + const C = local("c", { version: 1, params: { n: int() }, members: { v: str() } }); + + it("fail a two-object cycle with invalid/cycle and keep nothing", async () => { + const factories = { a: vi.fn(), b: vi.fn() }; + const { host } = await loaded((h) => { + h.serve(A, ({ n }, self) => { + factories.a(); + return { v: self.get(B, { n }).v }; + }); + h.serve(B, ({ n }, self) => { + factories.b(); + return { v: self.get(A, { n }).v }; + }); + }); + expect(await rejection(host.get(A, { n: 1 }, context))).toEqual({ + code: "invalid", + reason: "cycle", + message: "dependency cycle: a(1) → b(1) → a(1)", + }); + await expect(host.get(A, { n: 1 }, context)).rejects.toThrow("dependency cycle"); + expect([factories.a.mock.calls.length, factories.b.mock.calls.length]).toEqual([2, 2]); + }); + + it("fail a three-object cycle", async () => { + const { host } = await loaded((h) => { + h.serve(A, ({ n }, self) => ({ v: self.get(B, { n }).v })); + h.serve(B, ({ n }, self) => ({ v: self.get(C, { n }).v })); + h.serve(C, ({ n }, self) => ({ v: self.get(A, { n }).v })); + }); + expect(await rejection(host.get(A, { n: 1 }, context))).toMatchObject({ + message: "dependency cycle: a(1) → b(1) → c(1) → a(1)", + }); + }); + + it("fail an object that gets itself", async () => { + const { host } = await loaded((h) => h.serve(A, ({ n }, self) => ({ v: self.get(A, { n }).v }))); + expect(await rejection(host.get(A, { n: 1 }, context))).toMatchObject({ + message: "dependency cycle: a(1) → a(1)", + }); + }); +}); + +describe("GC", () => { + // AbortSignal.any keeps cleanup entries per call while its sources live before Node 24.16.0 (nodejs/node#62367): a + // call's signal is linked to its reference and its attempt only until it settles. + it.each([ + ["without a caller's signal", () => context], + ["with a long-lived caller's signal", () => withAbortSignal(new AbortController().signal, context)], + ])("10,000 calls on one reference retain flat memory, %s", async (_case, caller) => { + const { host } = await loaded((h) => h.serve(Counter, (_, self) => serveCounter(self))); + const counter = await host.get(Counter, {}, context); + const calls = caller(); + const run = async () => { + for (let i = 0; i < 10_000; i++) await counter.bump(calls); + }; + await run(); + const before = await settledHeap(); + await run(); + // The leak retained about 350 B per call; a document commit itself retains nothing. + expect((await settledHeap()) - before).toBeLessThan(500_000); + }); + + it("a released reference keeps neither its contract nor its object", async () => { + const objects: WeakRef[] = []; + const { host } = await loaded((h) => + h.serve(Viewer, ({ id }, self) => { + objects.push(new WeakRef(self)); + return { n: constant(id) }; + }), + ); + const { reference, contract } = await (async () => { + // The caller's own declaration, equal to the served one. + const Same = local("app.viewer", { version: 1, params: { id: int() }, members: { n: int() } }); + const reference = await host.get(Same, { id: 1 }, context); + await reference.n.watch(context); + reference.release(); + return { reference, contract: new WeakRef(Same) }; + })(); + await expectCollected(contract, ...objects); + expect(() => reference.n.value).toThrow("app.viewer: released"); + }); + + it("a released reference whose state was read keeps no host", async () => { + const { reference, host } = await (async () => { + const { host } = await loaded((h) => h.serve(Viewer, ({ id }) => ({ n: constant(id) }))); + const reference = await host.get(Viewer, { id: 1 }, context); + expect(reference.state.value).toEqual({ kind: "live" }); + await (await reference.state.watch(context)).stop(); + reference.release(); + return { reference, host: new WeakRef(host) }; + })(); + await expectCollected(host); + expect(reference.state.value).toMatchObject({ kind: "ended", error: { code: "released" } }); + }); + + it("a released reference whose state was read after a failure keeps neither the failure's error nor the host", async () => { + const { reference, refs } = await (async () => { + const { host } = await loaded((h) => + h.serve(Viewer, (_, self) => { + self.open( + async () => { + throw broken(new Error("no configuration")); + }, + () => {}, + ); + return { n: constant(1) }; + }), + ); + const reference = host.hold(Viewer, { id: 1 }, context); + await flushed(); + const state = reference.state.value; + if (state.kind !== "failed") throw new Error("expected a failure"); + const refs = [new WeakRef(state.cause.error), new WeakRef(host)]; + reference.release(); + return { reference, refs }; + })(); + await expectCollected(...refs); + expect(reference.state.value).toMatchObject({ kind: "ended", error: { code: "released" } }); + }); + + it("a hold released before load keeps nothing: the reference and its contract go while the host waits", async () => { + const host = createHost({ onError: () => {} }); + const refs = (() => { + const Same = local("app.viewer", { version: 1, params: { id: int() }, members: { n: int() } }); + const reference = host.hold(Same, { id: 1 }, context); + reference.release(); + return [new WeakRef(reference), new WeakRef(Same)]; + })(); + await expectCollected(...refs); + expect(host).toBeDefined(); + }); + + it("a released hold keeps neither its object, nor its state's watches, nor a remote object without a link", async () => { + const objects: WeakRef[] = []; + const { host } = await loaded((h) => + h.serve(Viewer, ({ id }, self) => { + objects.push(new WeakRef(self)); + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { n: constant(id) }; + }), + ); + const client = createHost({ onError: () => {} }); + await client.load([], context); + const { refs, held } = await (async () => { + const made = remoteObjects.length; + const local = host.hold(Viewer, { id: 1 }, context); + const remote = client.hold(Notes, { session: "S1" }, context); + const watches = [await local.state.watch(context), await remote.state.watch(context)]; + // A member watch waiting for the first value, and a parked call: both rejected by the release. + const waits = [local.n.watch(context).catch(() => {}), remote.wait(context).catch(() => {})]; + local.release(); + remote.release(); + await Promise.all(waits); + const refs = [...watches.map((watch) => new WeakRef(watch)), ...remoteObjects.slice(made)]; + return { refs, held: [local, remote] }; + })(); + await expectCollected(...objects, ...refs); + expect(held).toHaveLength(2); + }); +}); + +describe("release", () => { + it("rejects the calls made through the reference with released and aborts their handlers' contexts", async () => { + const Slow = local("app.slow", { version: 1, params: {}, members: { wait: method() } }); + const aborted: unknown[] = []; + const { host } = await loaded((h) => + h.serve(Slow, () => ({ + wait: (ctx) => + new Promise((resolve) => + ctx.abortSignal?.addEventListener("abort", () => { + aborted.push(ctx.abortSignal?.reason); + resolve(null); + }), + ), + })), + ); + const first = await host.get(Slow, {}, context); + const second = await host.get(Slow, {}, context); + const pending = first.wait(context); + const other = second.wait(context); + first.release(); + expect(await rejection(pending)).toEqual({ code: "released", message: "app.slow: released" }); + expect(aborted).toHaveLength(1); + second.release(); + await other.catch(() => {}); + expect(aborted).toHaveLength(2); + }); + + it("reads, watches and calls after the object ended carry the end's reason; the holder's own release has none", async () => { + const Ended = local("app.ended", { version: 1, params: {}, members: { n: int(), m: method() } }); + const { host } = await loaded((h) => h.serve(Ended, () => ({ n: constant(1), m: async () => null }))); + const ended = await host.get(Ended, {}, context); + const released = await host.get(Ended, {}, context); + released.release(); + await host.close(context); + const reasonOf = (error: unknown) => { + const { code, reason } = error as ChordError; + return `${code}/${reason}`; + }; + const thrown = (read: () => unknown) => { + try { + read(); + return "no throw"; + } catch (error) { + return reasonOf(error); + } + }; + expect([ + thrown(() => ended.n.value), + reasonOf(await ended.n.watch(context).catch((error: unknown) => error)), + reasonOf(await ended.m(context).catch((error: unknown) => error)), + ]).toEqual(["released/ended", "released/ended", "released/ended"]); + expect(thrown(() => released.n.value)).toBe("released/undefined"); + }); +}); + +describe("remote references", () => { + it("have every member of every variant; variant is undefined before live and released after release", async () => { + const Two = remote("app.two", { + version: 1, + params: {}, + variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { m: int() } } }, + }); + const Holder = local("app.holder", { version: 1, params: {}, members: { n: int() } }); + const seen: unknown[] = []; + const owner = { kind: "owner" } as const; + const host = createHost({ principal: owner, onError: () => {} }); + const facet = defineFacet((h) => { + h.serve(Two, () => ({ variant: "view", members: { m: constant(1) } })); + h.serve(Holder, (_, self) => { + const two = self.get(Two, {}); + seen.push(Object.keys(two).sort()); + seen.push(two.variant); + return { n: constant(1) }; + }); + }); + await host.load([{ name: "@test/app", facet }], context); + expect(seen).toEqual([["bump", "m", "n", "release", "retry", "state", "variant"], undefined]); + const two = await host.get(Two, {}, context); + expect(() => (two as unknown as { n: { value: number } }).n.value).toThrow( + 'app.two (variant view) has no state member "n"', + ); + two.release(); + expect(() => two.variant).toThrow("app.two: released"); + }); +}); + +describe("remote reference types", () => { + it("narrow on variant; RefIn picks variants", () => { + const Two = remote("app.two", { + version: 1, + params: {}, + variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { m: int() } } }, + }); + type Ref = Reference; + expectTypeOf().toEqualTypeOf<"full" | "view">(); + expectTypeOf["m"]["value"]>().toEqualTypeOf(); + expectTypeOf>().toEqualTypeOf< + "variant" | "n" | "bump" | "release" | "retry" | "state" + >(); + expectTypeOf>().not.toHaveProperty("variant"); + }); + + it("a held reference's variant may be undefined (before its first binding) and still narrows", () => { + const Two = remote("app.two", { + version: 1, + params: {}, + variants: { full: { state: { n: int() }, methods: { bump: method() } }, view: { state: { m: int() } } }, + }); + expectTypeOf["variant"]>().toEqualTypeOf<"full" | "view" | undefined>(); + const narrow = (held: Held) => (held.variant === "full" ? held.bump : undefined); + expectTypeOf(narrow).returns.toEqualTypeOf<((ctx: Context) => Promise) | undefined>(); + expectTypeOf>().toEqualTypeOf>(); + expectTypeOf["state"]>().toEqualTypeOf>(); + expectTypeOf["retry"]>().toEqualTypeOf<() => void>(); + }); +}); + +describe("keyed state members (slice 6a §12)", () => { + const NoteSchema = obj({ title: str() }); + const Memo = defineState("memo", { + schema: NoteSchema, + version: 1, + initial: () => ({ title: "" }), + keyed: true, + }); + const NoteStore = local("app.note-store", { + version: 1, + params: {}, + members: { notes: keyed(NoteSchema), write: method(obj({ id: str(), title: str() })), remove: method(str()) }, + }); + const Relay = local("app.relay", { version: 1, params: {}, members: { notes: keyed(NoteSchema) } }); + /** A view of note `id`; views of one note with another `copy` are other objects. */ + const NoteView = local("app.note-view", { + version: 1, + params: { id: str(), copy: int() }, + members: { note: NoteSchema }, + }); + + /** Serves the store, a relay re-exposing its keyed member, and views pinning one key each; `memos()` is the store's. */ + async function notes(options: { readonly viewOf?: "store" | "relay" } = {}) { + let memos: KeyedHandle<{ title: string }> | undefined; + const pinned: Watchable<{ title: string }>[] = []; + const refs: object[] = []; + const { host, errors } = await loaded((h) => { + h.serve(NoteStore, (_, self) => { + memos = self.state(Memo); + return { + notes: memos, + write: async ({ id, title }, ctx) => { + await memos!.change( + id, + (draft) => { + draft.title = title; + }, + ctx, + ); + return null; + }, + remove: async (id, ctx) => { + await memos!.retire(id, ctx); + return null; + }, + }; + }); + h.serve(Relay, (_, self) => ({ notes: self.get(NoteStore, {}).notes })); + h.serve(NoteView, ({ id }, self) => { + const source = options.viewOf === "relay" ? self.get(Relay, {}) : self.get(NoteStore, {}); + refs.push(source); + const note = source.notes.at(id); + pinned.push(note); + return { note }; + }); + }); + const store = await host.get(NoteStore, {}, context); + return { host, errors, store, memos: () => memos!, pinned, refs }; + } + + it("a consumer's at(key) is the provider's per-key source, loaded before it goes live; retirement carries the flag", async () => { + const { host, store } = await notes(); + await store.write({ id: "a", title: "x" }, context); + const view = await host.get(NoteView, { id: "a", copy: 1 }, context); + expect(view.note.value).toEqual({ title: "x" }); + const changes = changesOf(await view.note.watch(context)); + await store.write({ id: "a", title: "y" }, context); + await delivered(); + await store.remove("a", context); + await delivered(); + expect(changes).toEqual([ + { kind: "value", value: { title: "y" }, ops: [["s", ["title"], "y"]], retired: false }, + { kind: "value", value: { title: "" }, ops: [["r", { title: "" }]], retired: true }, + ]); + view.release(); + await host.close(context); + }); + + it("two consumers of one key share one source and one pin; each hold is released once, with its reference", async () => { + const { host, store, memos, pinned } = await notes(); + await store.write({ id: "a", title: "x" }, context); + const first = await host.get(NoteView, { id: "a", copy: 1 }, context); + const second = await host.get(NoteView, { id: "a", copy: 2 }, context); + const slot = (memos() as unknown as { slot: { source(key: string): { pins: number } | undefined } }).slot; + const [one, two] = pinned.map((at) => (at as unknown as { source: object }).source); + // The store object's source of the key, which follows its attempts. + expect(one).toBe(two); + const shared = slot.source("a")!; + expect(shared.pins).toBe(1); + first.release(); + await flushed(); + expect(slot.source("a")).toBe(shared); + second.release(); + await flushed(); + expect(slot.source("a")).toBeUndefined(); + await host.close(context); + }); + + it("a re-exposed keyed member resolves to the ultimate provider's source (no second tracker)", async () => { + const { host, store, memos } = await notes({ viewOf: "relay" }); + await store.write({ id: "a", title: "x" }, context); + const view = await host.get(NoteView, { id: "a", copy: 1 }, context); + expect(view.note.value).toEqual({ title: "x" }); + const slot = (memos() as unknown as { slot: { source(key: string): { pins: number } | undefined } }).slot; + // One pin on the ultimate slot, through the relay's key and the store's. + expect(slot.source("a")!.pins).toBe(1); + await store.write({ id: "a", title: "y" }, context); + expect(view.note.value).toEqual({ title: "y" }); + await host.close(context); + }); + + it("ref.member.get and watch work anywhere; at(key) only in a factory", async () => { + const { host, store } = await notes(); + expect(await store.notes.get("a", context)).toBeUndefined(); + expect(await store.notes.watch("a", context)).toBeUndefined(); + await store.write({ id: "a", title: "x" }, context); + expect(await store.notes.get("a", context)).toEqual({ title: "x" }); + const watch = (await store.notes.watch("a", context))!; + expect(watch.value).toEqual({ title: "x" }); + expect(() => store.notes.at("a")).toThrow("valid only in the factory"); + store.release(); + expect(await watch.closed).toEqual({ kind: "ended" }); + await host.close(context); + }); + + it("a watch(key) settling after its reference was released is ended at once with its pin; it rejects released", async () => { + const { host, store, memos } = await notes(); + await store.write({ id: "a", title: "x" }, context); + const slot = (memos() as unknown as { slot: { source(key: string): object | undefined } }).slot; + const watching = store.notes.watch("a", context); + // The slot's watch is underway: the release lands before the reference follows it. + await Promise.resolve(); + await Promise.resolve(); + store.release(); + expect(await rejection(watching)).toMatchObject({ code: "released" }); + await flushed(); + expect(slot.source("a")).toBeUndefined(); + await host.close(context); + }); + + it("a retained released factory reference keeps no provider: its at(key) drops the source", async () => { + const from = chordObjects.length; + const { host, store, refs, pinned } = await notes(); + await store.write({ id: "a", title: "x" }, context); + const view = await host.get(NoteView, { id: "a", copy: 1 }, context); + view.release(); + store.release(); + await host.close(context); + // The factory's reference and its at(key) are still held here (and by the host's factory). + expect(refs).toHaveLength(1); + await expectCollected(...chordObjects.slice(from)); + expect(() => (pinned[0] as unknown as { value: unknown }).value).toThrow("released"); + }); + + /** Serves `NoteStore` over `definition` (persisted) on `store`, and views pinning one key each. */ + async function persistedNotes(store: Store, definition: ReturnType) { + return loaded( + (h) => { + h.serve(NoteStore, (_, self) => { + const memos = self.state(definition); + return { + notes: memos, + write: async ({ id, title }, ctx) => { + await memos.change( + id, + (draft) => { + draft.title = title; + }, + ctx, + ); + return null; + }, + remove: async () => null, + }; + }); + h.serve(NoteView, ({ id }, self) => ({ note: self.get(NoteStore, {}).notes.at(id) })); + }, + { store }, + ); + } + + /** A persisted keyed `memo` at `version`, whose `migrate` (from older versions) throws. */ + function persistedMemo(version: number) { + return defineState("memo", { + schema: NoteSchema, + version, + initial: () => ({ title: "" }), + migrate: () => { + throw new Error("cannot migrate"); + }, + keyed: true, + persist: true, + checkpointWhen: () => false, + }); + } + + /** A store holding memo `a` = "x" at version 1, written by a host of its own. */ + async function seeded(): Promise { + const store = await memoryStore(); + const { host } = await persistedNotes(store, persistedMemo(1)); + await (await host.get(NoteStore, {}, context)).write({ id: "a", title: "x" }, context); + await host.close(context); + return store; + } + + it("a persisted key a consumer pins is loaded from the store before the consumer goes live", async () => { + const store = await seeded(); + const { host } = await persistedNotes(store, persistedMemo(1)); + const view = await host.get(NoteView, { id: "a", copy: 1 }, context); + expect(view.note.value).toEqual({ title: "x" }); + await host.close(context); + await store.close(context); + }); + + it("a pin load that fails its provider fails the consumer as a dependency failure (transient)", async () => { + const store = await seeded(); + const { host, errors } = await persistedNotes(store, persistedMemo(2)); + const provider = await host.get(NoteStore, {}, context); + const view = host.hold(NoteView, { id: "a", copy: 1 }, context); + await vi.waitFor(() => expect(view.state.value.kind).toBe("failed")); + expect(view.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "transient", error: { message: expect.stringContaining("cannot migrate") } }, + }); + expect(provider.state.value).toMatchObject({ kind: "failed", cause: { kind: "broken" } }); + expect(errors.map(({ kind }) => kind)).toContain("object_failed"); + view.release(); + await host.close(context); + await store.close(context); + }); + + it("a failed provider's ref.member.get rejects with the failure's cause", async () => { + const store = await seeded(); + const { host } = await persistedNotes(store, persistedMemo(2)); + const provider = await host.get(NoteStore, {}, context); + expect(await rejection(provider.notes.get("a", context))).toMatchObject({ + message: expect.stringContaining("cannot migrate"), + }); + expect(await rejection(provider.notes.get("b", context))).toMatchObject({ + message: expect.stringContaining("cannot migrate"), + }); + await host.close(context); + await store.close(context); + }); + + it("a provider returning something else for a keyed member fails broken", async () => { + const { host } = await loaded((h) => { + // Plugin code returning a wrong value for the keyed member. + h.serve( + NoteStore, + () => ({ notes: constant({}), write: async () => null, remove: async () => null }) as never, + ); + }); + expect(await rejection(host.get(NoteStore, {}, context))).toMatchObject({ + message: expect.stringContaining("notes: a keyed member is a keyed document handle"), + }); + await host.close(context); + }); +}); diff --git a/packages/chord/test/next/host/revocation.test.ts b/packages/chord/test/next/host/revocation.test.ts new file mode 100644 index 000000000..a80fba200 --- /dev/null +++ b/packages/chord/test/next/host/revocation.test.ts @@ -0,0 +1,430 @@ +// Tests of revocation (§4, R10, R11): revocation keys, \`host.revoke\` on one host (objects, running calls, activations, +// holds still waiting for \`load\`), child links and \`revoke\` messages, GC, and revocation across a relay. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, remote } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet, type FacetHost, type Host } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + expectCollected, + forEachCodec, + instancesOf, + Notes, + PlacedNotes, + RawPeer, + rawParent, + rawWorkers, + relayed, + Session, + serveNotes, + type TestPrincipal, + testCodec, + transported, + wireNames, +} from "../support.ts"; + +/** Every object attempted in this file, weakly. */ +const objects = instancesOf(ChordObject, "attempt"); + +/** The test app's revocations: a guest's session, or every owner. */ +type Revocation = { session: string } | { owners: true }; + +const revokedBy = (principal: TestPrincipal): Revocation[] => + principal.kind === "guest" + ? [{ session: principal.session }] + : principal.kind === "owner" + ? [{ owners: true }] + : principal.kind === "worker" + ? [{ session: principal.session }] + : []; + +const S1: TestPrincipal = { kind: "guest", session: "S1" }; + +/** A host-lifetime type whose activation waits for an open: `load` waits for it (the start barrier). */ +const Barrier = local("app.barrier", { version: 1, params: {}, members: { n: int() } }); + +/** A guest's variant: its `wait` method's index. */ +const { wait } = wireNames(Notes, "reader").methods as { wait: number }; + +/** A worker of session S1 serving `Notes` (as `serveNotes`), its parent a raw peer whose holds name their principal. */ +async function worker(setup: (host: FacetHost) => void = (h) => void serveNotes(h)) { + const host = createHost({ + principal: { kind: "worker", session: "S1" }, + revokedBy, + onError: () => {}, + }); + await host.load([{ name: "@test/worker", facet: defineFacet(setup) }], context); + const parent = await rawParent(host); + return { host, parent }; +} + +/** Sends a hold of `Notes` for S1 with id `hold`, acting as `principal` (honoured: the peer is the parent). */ +function holdAs(peer: RawPeer, hold: number, principal: TestPrincipal, Type: typeof Notes = Notes): void { + peer.send({ + kind: "hold", + hold, + type: Type.name, + version: Type.version, + fingerprint: Type.fingerprint, + params: jsonPart({ session: "S1" }), + principal, + }); +} + +describe("revocation keys", () => { + it("are computed once per principal object, compared as canonical JSON", async () => { + let calls = 0; + const host = createHost({ + principal: { kind: "server" }, + revokedBy: () => { + calls++; + return [{ b: 2, a: 1 }]; + }, + onError: () => {}, + }); + await host.load([{ name: "@test/app", facet: defineFacet((h) => serveNotes(h)) }], context); + const peer = await rawParent(host); + holdAs(peer, 0, S1); + holdAs(peer, 1, { kind: "guest", session: "S2" }); + await peer.next("bound"); + await peer.next("bound"); + host.revoke({ a: 1, b: 2 }, context); + host.revoke({ a: 1, b: 3 }, context); + await transported(); + expect(calls).toBe(2); + expect((await peer.next("gone")).hold).toBe(0); + expect((await peer.next("gone")).hold).toBe(1); + }); + + it("keep no principal alive (a WeakMap keyed by the principal the host cached them for)", async () => { + const cached: WeakRef[] = []; + const host = createHost({ + principal: { kind: "worker", session: "S1" }, + revokedBy: (principal) => { + // The principal decoded on this host, the cache's key: not the sender's object. + cached.push(new WeakRef(principal)); + return revokedBy(principal); + }, + onError: () => {}, + }); + await host.load([{ name: "@test/worker", facet: defineFacet((h) => serveNotes(h)) }], context); + const parent = await rawParent(host); + holdAs(parent, 0, { kind: "guest", session: "S9" }); + await parent.next("bound"); + host.revoke({ session: "S1" }, context); + parent.send({ kind: "release", hold: 0 }); + await transported(); + expect(cached.length).toBeGreaterThan(0); + await expectCollected(...cached); + // The cache's owner is still alive and working. + holdAs(parent, 1, S1); + await parent.next("bound"); + host.revoke({ session: "S1" }, context); + expect((await parent.next("gone")).hold).toBe(1); + await host.close(context); + }); +}); + +forEachCodec(() => { + describe("revoke on one host", () => { + it("ends the principal's objects at once, whatever references remain: each hold gets gone forbidden", async () => { + const { host, parent } = await worker(); + holdAs(parent, 0, S1); + holdAs(parent, 1, S1); + holdAs(parent, 2, { kind: "owner" }); + for (let i = 0; i < 3; i++) await parent.next("bound"); + host.revoke({ session: "S1" }, context); + await transported(); + const gone = [await parent.next("gone"), await parent.next("gone")]; + expect(gone.map(({ hold }) => hold).sort()).toEqual([0, 1]); + expect(gone[0]!.error).toEqual({ + kind: "chord", + error: { code: "forbidden", reason: "revoked", message: "app.notes(S1): revoked" }, + }); + // The owner's hold, the parent link and the local store stay. + expect(parent.closed).toBe(false); + expect(parent.kinds()).toEqual([]); + holdAs(parent, 3, S1); + expect((await parent.next("bound")).holds[0]!.hold).toBe(3); + }); + + it("aborts running calls (race 2): the handler's context, no result; a call crossing the gone is ignored", async () => { + let probe: ReturnType | undefined; + const { host, parent } = await worker((h) => { + probe = serveNotes(h); + }); + holdAs(parent, 0, S1); + await parent.next("bound"); + parent.send({ kind: "call", call: 0, hold: 0, method: wait }); + await transported(); + host.revoke({ session: "S1" }, context); + expect(probe!.waits[0]!.abortSignal?.reason).toMatchObject({ code: "forbidden" }); + expect((await parent.next("gone")).hold).toBe(0); + await transported(); + expect(parent.kinds()).toEqual([]); + parent.send({ kind: "call", call: 1, hold: 0, method: wait }); + await transported(); + expect(parent.kinds()).toEqual([]); + }); + + it("abandons an activation in progress (race 1): its open's context is aborted, its holders get gone", async () => { + const Slow = remote("app.slow", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const aborted: unknown[] = []; + const { host, parent } = await worker((h) => + h.serve(Slow, (_, _principal, self) => { + self.open( + (opening) => + new Promise((_, reject) => + opening.abortSignal?.addEventListener("abort", () => { + aborted.push(opening.abortSignal?.reason); + reject(opening.abortSignal?.reason); + }), + ), + () => {}, + ); + return { variant: "v", members: { n: constant(1) } }; + }), + ); + parent.send({ + kind: "hold", + hold: 0, + type: Slow.name, + version: Slow.version, + fingerprint: Slow.fingerprint, + params: "{}", + principal: S1, + }); + await transported(); + host.revoke({ session: "S1" }, context); + expect((await parent.next("gone")).error).toMatchObject({ kind: "chord", error: { code: "forbidden" } }); + expect(aborted).toMatchObject([{ code: "forbidden" }]); + }); + + it("never revokes the host's own principal", async () => { + const { host } = await worker(); + const own = await host.get(Notes, { session: "S1" }, context); + host.revoke({ session: "S1" }, context); + expect(own.state.value.kind).toBe("live"); + await host.close(context); + }); + + it("ends a hold the parent sent before its revoke while it still waits for load (race 3): gone, never bound", async () => { + for (const delayed of [false, true]) { + // A worker connects before it loads (W6); a load can wait for an activation (the start barrier). + const opened = Promise.withResolvers(); + const host = createHost({ + principal: { kind: "worker", session: "S1" }, + revokedBy, + onError: () => {}, + }); + const parent = await rawParent(host); + const facet = defineFacet((h) => { + serveNotes(h); + if (delayed) { + h.serve(Barrier, (_params, self) => { + self.open( + () => opened.promise, + () => {}, + ); + return { n: constant(0) }; + }); + } + }); + const loading = host.load([{ name: "@test/worker", facet }], context); + holdAs(parent, 0, S1); + parent.send({ kind: "revoke", revocation: { session: "S1" } }); + await transported(); + opened.resolve(); + await loading; + await transported(); + expect(await parent.next("gone")).toMatchObject({ + hold: 0, + error: { kind: "chord", error: { code: "forbidden", reason: "revoked" } }, + }); + expect(parent.kinds()).toEqual([]); + expect(host.references(Notes).value).toEqual([]); + await host.close(context); + } + }); + + it("ends a relayed hold of the principal still waiting for its route's link: never held there", async () => { + const host = createHost({ + principal: { kind: "worker", session: "S0" }, + revokedBy, + onError: () => {}, + }); + const resolving = Promise.withResolvers(); + const routed: RawPeer[] = []; + host.route( + Session, + { + authorize: () => true, + resolve: async () => { + await resolving.promise; + const [a, b] = memoryLinkPair(); + routed.push(new RawPeer(b)); + return { link: a, principal: { kind: "worker", session: "S1" }, codec: testCodec() }; + }, + }, + context, + ); + await host.load([], context); + const parent = await rawParent(host); + const { name: type, version, fingerprint } = PlacedNotes; + const address = { kind: "session", params: { id: "S1" } }; + parent.send({ + kind: "hold", + hold: 0, + type, + version, + fingerprint, + params: jsonPart({ session: "S1" }), + address, + principal: S1, + }); + await transported(); + parent.send({ kind: "revoke", revocation: { session: "S1" } }); + expect(await parent.next("gone")).toMatchObject({ + hold: 0, + error: { kind: "chord", error: { code: "forbidden", reason: "revoked" } }, + }); + resolving.resolve(); + await transported(); + await transported(); + expect(routed.flatMap((peer) => peer.received.filter(({ kind }) => kind === "hold"))).toEqual([]); + await host.close(context); + }); + + it("binds a hold made for a revoked principal, then the parent's revoke after it ends it (race 3, R10's window)", async () => { + const { host, parent } = await worker(); + holdAs(parent, 0, S1); + await parent.next("bound"); + // The parent's revoke follows the hold on the same link, so it reaches the worker after it. + parent.send({ kind: "revoke", revocation: { session: "S1" } }); + expect((await parent.next("gone")).error).toMatchObject({ kind: "chord", error: { code: "forbidden" } }); + await host.close(context); + }); + }); + + describe("links and revoke messages", () => { + it("closes the child links whose principal it revokes, never the parent link", async () => { + const host = createHost({ + principal: { kind: "server" }, + revokedBy, + onError: () => {}, + }); + await host.load([], context); + const peers = [S1, { kind: "guest", session: "S2" } as const, S1].map((principal) => { + const [a, b] = memoryLinkPair(); + host.accept(a, { principal, codec: testCodec() }, context); + return new RawPeer(b); + }); + const parent = await rawParent(host); + host.revoke({ session: "S1" }, context); + await transported(); + // Two tabs of one grant: both links close (race 4). + expect(peers.map(({ closed }) => closed)).toEqual([true, false, true]); + expect(parent.closed).toBe(false); + // A revocation is no message for a child, nor for the parent. + expect(peers[1]!.kinds()).toEqual(["hello"]); + expect(parent.kinds()).toEqual([]); + }); + + it("ignores a revoke from a child link (race 6)", async () => { + const host = createHost({ + principal: { kind: "server" }, + revokedBy, + onError: () => {}, + }); + await host.load([{ name: "@test/app", facet: defineFacet((h) => serveNotes(h)) }], context); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: S1, codec: testCodec() }, context); + const child = new RawPeer(b); + await child.next("hello"); + holdAs(child, 0, S1); + await child.next("bound"); + child.send({ kind: "revoke", revocation: { session: "S1" } }); + await transported(); + expect(child.closed).toBe(false); + expect(child.kinds()).toEqual([]); + }); + + it("sends revoke on every route link, and on none made later (race 5: nothing is replayed)", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers) }); + const server = relay.server as unknown as Host; + relay.tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + server.revoke({ session: "S7" }, context); + await transported(); + expect(workers[0]!.received.filter(({ kind }) => kind === "revoke")).toEqual([ + { kind: "revoke", revocation: { session: "S7" } }, + ]); + relay.tab.hold(PlacedNotes, { session: "S2" }, context); + await transported(); + expect(workers[1]!.received.some(({ kind }) => kind === "revoke")).toBe(false); + }); + }); +}); + +describe("GC of revoked objects", () => { + it("keeps no revoked object, however many references its holders still have", async () => { + const { host, refs, parent } = await (async () => { + const { host, parent } = await worker(); + const made = objects.length; + holdAs(parent, 0, S1); + await parent.next("bound"); + const refs = objects.slice(made); + expect(refs.length).toBeGreaterThan(0); + host.revoke({ session: "S1" }, context); + await parent.next("gone"); + return { host, refs, parent }; + })(); + await expectCollected(...refs.filter((ref: WeakRef) => ref.deref()?.served.kind === "remote")); + expect(parent.closed).toBe(false); + await host.close(context); + }); +}); + +describe("revocation across a relay", () => { + it("closes a revoked tab's link at the server and ends its objects on the worker", async () => { + const relay = await relayed({ tab: S1, revokedBy }); + const notes = relay.tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + await transported(); + expect(notes.state.value.kind).toBe("live"); + const worker = relay.workers.get("session|S1")!; + const present = worker.references(PlacedNotes); + expect(present.value).toHaveLength(1); + const server = relay.server as unknown as Host; + server.revoke({ session: "S1" }, context); + await transported(); + await transported(); + // The tab's link closed: its reference waits for a new link, its holds on the worker are released. + expect(notes.state.value.kind).toBe("loading"); + expect(present.value).toEqual([]); + }); + + it("ends the worker's objects of the principal through the revoke message alone; the relay forwards their gone", async () => { + // The server's own revokedBy names no guest: its tab link stays, only the worker applies the revocation. + const relay = await relayed({ tab: S1, revokedBy, serverRevokedBy: () => [] }); + const notes = relay.tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + await transported(); + expect(notes.state.value.kind).toBe("live"); + const server = relay.server as unknown as Host; + server.revoke({ session: "S1" }, context); + await transported(); + await transported(); + expect(notes.state.value.kind).toBe("ended"); + expect(relay.workers.get("session|S1")!.references(PlacedNotes).value).toEqual([]); + expect(relay.errors.tab).toEqual([]); + // The tab's link stayed open: a new hold binds. + const again = await relay.tab.get(PlacedNotes, { session: "S1" }, context); + expect(again.state.value.kind).toBe("live"); + }); +}); diff --git a/packages/chord/test/next/host/route.test.ts b/packages/chord/test/next/host/route.test.ts new file mode 100644 index 000000000..dc4485504 --- /dev/null +++ b/packages/chord/test/next/host/route.test.ts @@ -0,0 +1,368 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, createContextKey, withContextValue } from "../../../src/context/index.ts"; +import { hostKind, int, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet, type Host } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { + addNote, + expectCollected, + fakeTimers, + flushed, + forEachCodec, + PlacedNotes, + rejection, + relayed, + Session, + serveNotes, + type TestPrincipal, + transported, +} from "../support.ts"; + +/** Advances fake time by `ms`, then lets links and promises settle. */ +async function elapse(ms: number): Promise { + await vi.advanceTimersByTimeAsync(ms); + await transported(); +} + +forEachCodec(() => { + describe("route admission", () => { + it("runs one resolve per address, shared by the holds that wait for it (race 1)", async () => { + const { tab, resolves, workers } = await relayed(); + const [one, two] = await Promise.all([ + tab.get(PlacedNotes, { session: "S1" }, context), + tab.get(PlacedNotes, { session: "S1" }, context), + ]); + expect(resolves).toEqual(["session|S1"]); + await addNote(workers.get("session|S1")!, "S1", "a"); + await transported(); + expect(one.notes.value).toEqual([{ text: "a" }]); + expect(two.notes.value).toEqual([{ text: "a" }]); + await tab.get(PlacedNotes, { session: "S2" }, context); + expect(resolves).toEqual(["session|S1", "session|S2"]); + }); + + it("authorizes every hold with its principal: a refusal is broken forbidden, other principals still resolve (race 4)", async () => { + const authorize = (principal: TestPrincipal, address: { readonly key: string }) => + principal.kind === "owner" || + (principal.kind === "guest" && address.key === `session|${principal.session}`); + const guest = await relayed({ tab: { kind: "guest", session: "S1" }, authorize }); + expect(await rejection(guest.tab.get(PlacedNotes, { session: "S2" }, context))).toEqual({ + code: "forbidden", + message: "session|S2: not authorized", + }); + expect(guest.resolves).toEqual([]); + expect((await guest.tab.get(PlacedNotes, { session: "S1" }, context)).variant).toBe("reader"); + const owner = await relayed({ authorize }); + expect((await owner.tab.get(PlacedNotes, { session: "S2" }, context)).variant).toBe("editor"); + }); + + it("fails a hold broken with what authorize throws", async () => { + const { tab } = await relayed({ + authorize: () => { + throw new Error("authorize threw"); + }, + }); + expect(await rejection(tab.get(PlacedNotes, { session: "S1" }, context))).toMatchObject({ + message: "app.placed-notes: authorize threw", + }); + }); + + it("fails a hold broken invalid/placement for an address whose params do not fit the kind", async () => { + const Misplaced = remote("app.misplaced", { + version: 1, + params: { n: int() }, + // An address of the right kind with params of the wrong type, as a buggy placement could compute. + placement: ({ n }) => ({ kind: "session", params: { id: n }, key: "" }) as never, + variants: { v: { state: { n: int() } } }, + }); + const { tab, resolves } = await relayed(); + expect(await rejection(tab.get(Misplaced, { n: 1 }, context))).toMatchObject({ + code: "invalid", + reason: "placement", + }); + expect(resolves).toEqual([]); + }); + }); + + describe("resolve failures and re-resolves", () => { + fakeTimers(); + + it("a rejected resolve leaves the holds loading, is reported, and resolves again after the address's backoff (race 2)", async () => { + let failures = 2; + const { tab, resolves, errors } = await relayed({ + resolve: async (_address, worker) => { + if (failures-- > 0) throw new Error("no worker yet"); + return worker(); + }, + }); + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + expect(errors.server).toMatchObject([{ kind: "resolve_failed", address: "session|S1" }]); + await elapse(999); + expect(errors.server).toHaveLength(1); + await elapse(1); + // The second failure doubles the window. + expect(errors.server).toHaveLength(2); + await elapse(1999); + expect(notes.state.value).toEqual({ kind: "loading" }); + await elapse(1); + expect(resolves).toEqual(["session|S1"]); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it("a returned link that is closed already starts the backoff", async () => { + let closed = true; + const { tab, resolves } = await relayed({ + resolve: async (address, worker) => { + if (!closed) return worker(); + closed = false; + const [a, b] = memoryLinkPair(); + b.close(); + await flushed(); + return { link: a, principal: { kind: "worker", session: address.params.id } }; + }, + }); + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + await elapse(1000); + expect(resolves).toEqual(["session|S1"]); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it("a dropped link resolves again while held: references go loading, then live on a fresh worker (race 6)", async () => { + const { tab, resolves, routeLinks, workers } = await relayed(); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + const first = workers.get("session|S1")!; + routeLinks[0]!.close(); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + await elapse(1000); + expect(resolves).toEqual(["session|S1", "session|S1"]); + expect(workers.get("session|S1")).not.toBe(first); + expect(notes.state.value).toEqual({ kind: "live" }); + await addNote(workers.get("session|S1")!, "S1", "fresh"); + await transported(); + expect(notes.notes.value).toEqual([{ text: "fresh" }]); + }); + + it("only a link that completed its handshake resets the backoff", async () => { + let silent = 0; + const { tab, resolves, routeLinks } = await relayed({ + resolve: async (address, worker) => { + if (silent-- <= 0) return worker(); + // A link whose other end never greets and closes. + const [a, b] = memoryLinkPair(); + setTimeout(() => b.close(), 10); + return { link: a, principal: { kind: "worker", session: address.params.id } }; + }, + }); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + silent = 1; + routeLinks[0]!.close(); + await elapse(1000); + // The silent link closed before its handshake: the window doubles. + await elapse(10); + await elapse(1999); + expect(resolves).toEqual(["session|S1"]); + await elapse(1); + expect(resolves).toEqual(["session|S1", "session|S1"]); + expect(notes.state.value).toEqual({ kind: "live" }); + // That link greeted: the next drop starts again at 1 s. + routeLinks.at(-1)!.close(); + await elapse(1000); + expect(resolves).toHaveLength(3); + }); + + it("a link whose peer never greets closes after 30 s of silence; the address backs off, then resolves again while held", async () => { + // The heartbeat's timer too, before the hosts start it. + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "setInterval", "clearInterval", "Date"], now: 0 }); + let silent = true; + const { tab, resolves } = await relayed({ + resolve: async (address, worker) => { + if (!silent) return worker(); + silent = false; + // Its other end is never read: no hello, no close. + const [a] = memoryLinkPair(); + return { link: a, principal: { kind: "worker", session: address.params.id } }; + }, + }); + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + await elapse(29_000); + expect(resolves).toEqual([]); + expect(notes.state.value).toEqual({ kind: "loading" }); + // The heartbeat checks every 1.5 s: the silent link closes within it, then its address waits 1 s. + await elapse(2_500); + await elapse(1_000); + expect(resolves).toEqual(["session|S1"]); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it("an address without holds is not resolved again after its link drops", async () => { + const { tab, resolves, routeLinks } = await relayed(); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + notes.release(); + await transported(); + routeLinks[0]!.close(); + await elapse(60_000); + expect(resolves).toEqual(["session|S1"]); + }); + + it("a resolve whose holds were all released meanwhile finishes and its link is kept (race 3)", async () => { + let finish: () => void = () => {}; + const { tab, resolves, routeLinks, server } = await relayed({ + resolve: async (_address, worker) => { + await new Promise((resolve) => { + finish = resolve; + }); + return worker(); + }, + }); + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + notes.release(); + await transported(); + finish(); + await transported(); + expect(resolves).toEqual(["session|S1"]); + let closed = false; + routeLinks[0]!.onClose(() => { + closed = true; + }); + await elapse(60_000); + expect(closed).toBe(false); + // The kept link serves the next hold without a resolve. + await tab.get(PlacedNotes, { session: "S1" }, context); + expect(resolves).toEqual(["session|S1"]); + await server.close(context); + expect(closed).toBe(true); + }); + }); + + describe("host.close during a resolve (race 7)", () => { + it("aborts the resolve's context and closes a link returned afterwards", async () => { + let routing: AbortSignal | undefined; + let finish: () => void = () => {}; + const { tab, server, routeLinks } = await relayed({ + resolve: async (_address, worker, context) => { + routing = context.abortSignal; + await new Promise((resolve) => { + finish = resolve; + }); + return worker(); + }, + }); + tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + expect(routing?.aborted).toBe(false); + await server.close(context); + expect(routing?.aborted).toBe(true); + finish(); + await transported(); + let closed = false; + routeLinks[0]!.onClose(() => { + closed = true; + }); + expect(closed).toBe(true); + }); + }); + + describe("declaring routes", () => { + it("is done before load, once per kind", async () => { + const host = createHost({ onError: () => {} }); + const options = { authorize: () => true, resolve: () => new Promise(() => {}) }; + host.route(Session, options, context); + expect(() => host.route(Session, options, context)).toThrow("route: session is routed already"); + await host.load([], context); + expect(() => host.route(hostKind("other"), options, context)).toThrow( + "route: routes are declared before load", + ); + }); + }); + + describe("a route's context", () => { + it("is the parent of its resolves and of the work arriving on its links", async () => { + const App = createContextKey("app"); + const Who = remote("app.who", { + version: 1, + params: {}, + variants: { v: { methods: { app: method(undefined, str()) } } }, + }); + const resolved: (string | undefined)[] = []; + let worker: Host | undefined; + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + server.route( + Session, + { + authorize: () => true, + resolve: async (address, routing) => { + resolved.push(routing.value(App)); + const [a, b] = memoryLinkPair(); + worker = createHost({ + address, + principal: { kind: "worker", session: "S1" }, + onError: () => {}, + }); + worker.connect(b, { principal: { kind: "server" } }, context); + await worker.load( + [ + { + name: "@test/worker", + facet: defineFacet((h) => void serveNotes(h, PlacedNotes)), + }, + ], + context, + ); + return { link: a, principal: { kind: "worker", session: "S1" } }; + }, + }, + withContextValue(App, "the app's", context), + ); + const facet = defineFacet((h) => + h.serve(Who, () => ({ variant: "v", members: { app: async (ctx) => ctx.value(App) ?? "none" } })), + ); + await server.load([{ name: "@test/server", facet }], context); + server.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + await transported(); + // The worker's call reaches the server over the route link: its handler runs under the route's context. + const who = await worker!.get(Who, {}, context); + expect(await who.app(context)).toBe("the app's"); + expect(resolved).toEqual(["the app's"]); + }); + }); + + describe("GC", () => { + it("a closed route link and its released references keep no route state", async () => { + const { refs, tab } = await (async () => { + const { tab, routeLinks, workers } = await relayed(); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + const refs = [new WeakRef(notes), new WeakRef(workers.get("session|S1")!), new WeakRef(routeLinks[0]!)]; + notes.release(); + await transported(); + routeLinks[0]!.close(); + workers.clear(); + routeLinks.length = 0; + await transported(); + return { refs, tab }; + })(); + await expectCollected(...refs); + expect(tab).toBeDefined(); + }); + + it("holds released while their resolve never settles are not kept", async () => { + const { refs, server } = await (async () => { + const { tab, server } = await relayed({ resolve: () => new Promise(() => {}) }); + const notes = tab.hold(PlacedNotes, { session: "S1" }, context); + await transported(); + notes.release(); + await transported(); + return { refs: [new WeakRef(notes)], server }; + })(); + await expectCollected(...refs); + expect(server).toBeDefined(); + }); + }); +}); diff --git a/packages/chord/test/next/host/routines.test.ts b/packages/chord/test/next/host/routines.test.ts new file mode 100644 index 000000000..e3bb69a6e --- /dev/null +++ b/packages/chord/test/next/host/routines.test.ts @@ -0,0 +1,1148 @@ +// Tests of routines (§14): `on` (the first call with the current value, folding, no overlap, retries after a throw +// with the whole value, retirements, the source ending or failing, native sources, facets), `onLive` (returning, +// retries, the backoff reset after a long run, aborts), the drain and `host.close`, and what routines leave behind. + +import { describe, expect, expectTypeOf, it, vi } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import type { Op } from "../../../src/delta/index.ts"; +import { int, local, method, obj, remote } from "../../../src/next/contracts/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { + type Ctx, + createHost, + defineFacet, + type HostError, + type RefState, + type Self, + sleep, + untilAborted, +} from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { + NativeWatch, + type NativeWatchable, + type NativeWatchCall, + Watch, + type Watchable, + type WatchCall, +} from "../../../src/watch.ts"; +import { + EDITOR, + expectCollected, + fakeTimers, + flushed, + linked, + loaded, + Notes, + rawParent, + serveNotes, + type TestPrincipal, + transported, +} from "../support.ts"; + +/** What an `on` handler gets: a watch's value call. */ +type WatchValueCall = Extract, { kind: "value" }>; +type NativeWatchValueCall = Extract, { kind: "value" }>; + +const Doc = defineState("doc", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + +/** A host-lifetime object with a counter document and `bump`; its factory declares routines with `routines`. */ +const Store = local("app.store", { version: 1, params: {}, members: { n: int(), bump: method() } }); + +/** The same with params: it lives while referenced. */ +const Item = local("app.item", { version: 1, params: { id: int() }, members: { n: int(), bump: method() } }); + +/** Loads a host serving `Store`; `declare` declares its routines over its document. */ +async function store( + declare: (doc: SingletonHandle<{ n: number }>, self: Self) => void, + options: { readonly closeTimeout?: number } = {}, +) { + const loadedHost = await loaded( + (h) => + h.serve(Store, (_, self) => { + const doc = self.state(Doc); + declare(doc, self); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + options, + ); + const ref = await loadedHost.host.get(Store, {}, context); + return { ...loadedHost, ref }; +} + +/** Counts one up in one transaction. */ +async function bump(doc: SingletonHandle<{ n: number }>, ctx: Context): Promise { + await doc.change((d) => { + d.n++; + }, ctx); + return null; +} + +/** Waits one macrotask and a little: a call after a busy one waits for one (`settling`). */ +function settled(): Promise { + return new Promise((resolve) => setTimeout(resolve, 5)); +} + +/** The routine reports among `errors`, as `kind #position: message`. */ +function reports(errors: readonly HostError[]): string[] { + return errors.flatMap((error) => + error.kind === "routine_failed" + ? [`${error.routine.kind} #${error.routine.position}: ${(error.error as Error).message}`] + : [], + ); +} + +/** A JSON watchable (Chord's shared `Watch`) the test advances by hand, as a Durable document would. */ +function jsonSource(initial: { n: number }) { + let value = initial; + const watches = new Set>(); + const source: Watchable<{ n: number }> = { + get value() { + return value; + }, + async watch(ctx) { + const watch: Watch<{ n: number }> = new Watch( + value, + () => watches.delete(watch), + () => {}, + ); + watches.add(watch); + if (ctx.abortSignal !== undefined) watch.observeAbort(ctx.abortSignal); + return watch; + }, + }; + const advance = (call: WatchCall<{ n: number }>): void => { + if (call.kind === "value") value = call.value; + for (const watch of watches) watch.advance(call, context); + }; + return { source, advance }; +} + +/** A native watchable the test sets and ends by hand. */ +function writable(initial: T) { + let value = initial; + const watches = new Set>(); + const source: NativeWatchable & { set(next: T): void; end(): void } = { + get value() { + return value; + }, + async watch(ctx) { + const watch: NativeWatch = new NativeWatch( + value, + () => watches.delete(watch), + () => {}, + ); + watches.add(watch); + if (ctx.abortSignal !== undefined) watch.observeAbort(ctx.abortSignal); + return watch; + }, + set(next) { + value = next; + for (const watch of watches) watch.advance(next, context); + }, + end() { + for (const watch of watches) watch.end({ kind: "ended" }); + }, + }; + return { source, watches }; +} + +describe("on", () => { + it("calls first with the current value as [['r', value]], then once per change with its ops", async () => { + const calls: [unknown, readonly Op[]][] = []; + const { ref } = await store((doc, self) => { + self.on(doc, async ({ value, ops }) => { + calls.push([value, ops]); + }); + }); + await flushed(); + await ref.bump(context); + await flushed(); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]]], + [{ n: 1 }, [["s", ["n"], 1]]], + ]); + }); + + it("never overlaps calls: changes during a call fold into the next, which is not aborted by them", async () => { + const gate = Promise.withResolvers(); + const calls: [unknown, readonly Op[]][] = []; + let running = 0; + let overlapped = false; + let aborted = false; + const { ref } = await store((doc, self) => { + self.on(doc, async ({ value, ops }, ctx) => { + overlapped ||= running > 0; + running++; + calls.push([value, ops]); + if (calls.length === 2) await gate.promise; + aborted ||= ctx.abortSignal?.aborted === true; + running--; + }); + }); + await flushed(); + await ref.bump(context); + await flushed(); + await ref.bump(context); + await ref.bump(context); + gate.resolve(); + await settled(); + expect(calls.map(([value]) => value)).toEqual([{ n: 0 }, { n: 1 }, { n: 3 }]); + expect(calls[2]![1]).toEqual([["s", ["n"], 3]]); + expect({ overlapped, aborted }).toEqual({ overlapped: false, aborted: false }); + }); + + describe("after a throw", () => { + fakeTimers(); + + it("reports, keeps the watch, and calls again with [['r', value]] after 1 s doubling; success resets", async () => { + const calls: [unknown, readonly Op[], number][] = []; + let failing = 3; + const { ref, errors } = await store((doc, self) => { + self.on(doc, async ({ value, ops }) => { + calls.push([value, ops, Date.now()]); + if (failing-- > 0) throw new Error(`fail ${calls.length}`); + }); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + await vi.advanceTimersByTimeAsync(2_000); + await vi.advanceTimersByTimeAsync(4_000); + expect(calls.map(([, , at]) => at)).toEqual([0, 1_000, 3_000, 7_000]); + expect(calls.every(([, ops]) => ops[0]![0] === "r")).toBe(true); + expect(reports(errors)).toEqual(["on #1: fail 1", "on #1: fail 2", "on #1: fail 3"]); + // Succeeded: the next change gets its ops, and a later throw backs off from 1 s again. + await ref.bump(context); + await vi.advanceTimersByTimeAsync(0); + expect(calls.at(-1)!.slice(0, 2)).toEqual([{ n: 1 }, [["s", ["n"], 1]]]); + failing = 1; + await ref.bump(context); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + expect(calls.map(([, , at]) => at).slice(-2)).toEqual([7_000, 8_000]); + expect(vi.getTimerCount()).toBe(0); + }); + + it("retries a change folded behind the throwing call after the backoff, once: no third call, no timer", async () => { + const gate = Promise.withResolvers(); + const calls: [unknown, readonly Op[], number][] = []; + const { ref } = await store((doc, self) => { + self.on(doc, async ({ value, ops }) => { + calls.push([value, ops, Date.now()]); + if (calls.length === 1) { + await gate.promise; + throw new Error("first"); + } + }); + }); + await vi.advanceTimersByTimeAsync(0); + await ref.bump(context); + gate.resolve(); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(5_000); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]], 0], + [{ n: 1 }, [["r", { n: 1 }]], 1_000], + ]); + expect(vi.getTimerCount()).toBe(0); + }); + + it("makes one retry per throw; a change during a failing retry waits for its backoff; no timer left", async () => { + const calls: [unknown, number][] = []; + const { host, errors } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(doc, async ({ value }) => { + calls.push([value, Date.now()]); + // The retry at 1 s changes the source, then throws too. + if (calls.length === 2) await bump(doc, context); + if (calls.length <= 2) throw new Error(`fail ${calls.length}`); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + await vi.advanceTimersByTimeAsync(10_000); + expect(calls).toEqual([ + [{ n: 0 }, 0], + [{ n: 0 }, 1_000], + [{ n: 1 }, 3_000], + ]); + expect(reports(errors)).toEqual(["on #1: fail 1", "on #1: fail 2"]); + expect(vi.getTimerCount()).toBe(0); + item.release(); + await vi.advanceTimersByTimeAsync(0); + expect(vi.getTimerCount()).toBe(0); + }); + + it("backs off after a throw even when the failed call changed its own source: no burst", async () => { + const calls: number[] = []; + const { errors } = await store((doc, self) => { + self.on(doc, async (_change, ctx) => { + calls.push(Date.now()); + if (calls.length > 100) return; + // A reconciler that records its attempt in its own source, then fails. + await bump(doc, ctx); + throw new Error("downstream unavailable"); + }); + }); + await vi.advanceTimersByTimeAsync(0); + expect(calls).toEqual([0]); + await vi.advanceTimersByTimeAsync(1_000); + await vi.advanceTimersByTimeAsync(2_000); + expect(calls).toEqual([0, 1_000, 3_000]); + expect(reports(errors)).toHaveLength(3); + }); + + it("retries with the value that arrived during the failed call, as a whole value", async () => { + const calls: [unknown, readonly Op[], number][] = []; + await store((doc, self) => { + self.on(doc, async ({ value, ops }, ctx) => { + calls.push([value, ops, Date.now()]); + if (calls.length > 1) return; + await bump(doc, ctx); + throw new Error("once"); + }); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]], 0], + [{ n: 1 }, [["r", { n: 1 }]], 1_000], + ]); + }); + + it("folds changes made during a slow retry into one call with the newest value", async () => { + const gate = Promise.withResolvers(); + const calls: [unknown, readonly Op[]][] = []; + const { ref } = await store((doc, self) => { + self.on(doc, async ({ value, ops }) => { + calls.push([value, ops]); + if (calls.length === 1) throw new Error("first"); + if (calls.length === 2) await gate.promise; + }); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + for (let i = 0; i < 3; i++) await ref.bump(context); + gate.resolve(); + await vi.advanceTimersByTimeAsync(0); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]]], + [{ n: 0 }, [["r", { n: 0 }]]], + [{ n: 3 }, [["s", ["n"], 3]]], + ]); + }); + + it("waits for the next good value when its source fails during the backoff: the retry is dropped", async () => { + const calls: unknown[] = []; + let set: ((n: number) => Promise) | undefined; + const { errors } = await store((doc, self) => { + const failing = self.derive([doc], ([d]) => { + if (d.n < 0) throw new Error("compute"); + return d.n; + }); + set = (n) => + doc.change((d) => { + d.n = n; + }, context); + self.on(failing, async ({ value }) => { + calls.push(value); + if (value === 1) throw new Error("handler threw"); + }); + }); + await vi.advanceTimersByTimeAsync(0); + await set!(1); + await vi.advanceTimersByTimeAsync(0); + await set!(-1); + await vi.advanceTimersByTimeAsync(2_000); + expect(calls).toEqual([0, 1]); + expect(vi.getTimerCount()).toBe(0); + await set!(3); + await vi.advanceTimersByTimeAsync(0); + expect(calls).toEqual([0, 1, 3]); + expect(reports(errors)).toEqual(["on #1: handler threw"]); + }); + + it("clears its retry timer when its object ends during the backoff", async () => { + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(doc, async () => { + throw new Error("always"); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(0); + expect(vi.getTimerCount()).toBe(1); + item.release(); + await vi.advanceTimersByTimeAsync(0); + expect(vi.getTimerCount()).toBe(0); + }); + + it("reports a retirement again on the retry after the call that got it threw", async () => { + const { source, advance } = jsonSource({ n: 0 }); + const calls: WatchValueCall<{ n: number }>[] = []; + await store((_, self) => { + self.on(source, async (call) => { + calls.push(call); + if (calls.length === 2) throw new Error("retirement not handled"); + }); + }); + await vi.advanceTimersByTimeAsync(0); + advance({ kind: "value", value: { n: 0 }, ops: [], retired: true }); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + await vi.advanceTimersByTimeAsync(10_000); + expect(calls.slice(1)).toEqual([ + { kind: "value", value: { n: 0 }, ops: [], retired: true }, + { kind: "value", value: { n: 0 }, ops: [["r", { n: 0 }]], retired: true }, + ]); + }); + + it("calls at once with the newest whole value when the source changes during the backoff", async () => { + const calls: [unknown, readonly Op[], number][] = []; + const { ref } = await store((doc, self) => { + self.on(doc, async ({ value, ops }) => { + calls.push([value, ops, Date.now()]); + if (calls.length === 1) throw new Error("first"); + }); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(500); + await ref.bump(context); + await vi.advanceTimersByTimeAsync(0); + await ref.bump(context); + await vi.advanceTimersByTimeAsync(0); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]], 0], + [{ n: 1 }, [["r", { n: 1 }]], 500], + [{ n: 2 }, [["s", ["n"], 2]], 500], + ]); + // The backoff timer is gone: nothing calls again. + await vi.advanceTimersByTimeAsync(10_000); + expect(calls).toHaveLength(3); + }); + }); + + it("folds every change made during the first call into one next call: [0, 3], not [0, 1, 3]", async () => { + const gate = Promise.withResolvers(); + const calls: [unknown, readonly Op[]][] = []; + const { ref } = await store((doc, self) => { + self.on(doc, async ({ value, ops }) => { + calls.push([value, ops]); + if (calls.length === 1) await gate.promise; + }); + }); + await flushed(); + for (let i = 0; i < 3; i++) await ref.bump(context); + await flushed(); + gate.resolve(); + await settled(); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]]], + [{ n: 3 }, [["s", ["n"], 3]]], + ]); + }); + + it("follows an app's JSON watchable whose watch is Chord's shared Watch with ops, as its type says", async () => { + const { source, advance } = jsonSource({ n: 0 }); + const calls: [unknown, readonly Op[]][] = []; + await store((_, self) => { + self.on(source, async ({ value, ops }) => void calls.push([value, ops])); + }); + await flushed(); + advance({ kind: "value", value: { n: 1 }, ops: [["s", ["n"], 1]], retired: false }); + await flushed(); + expect(calls).toEqual([ + [{ n: 0 }, [["r", { n: 0 }]]], + [{ n: 1 }, [["s", ["n"], 1]]], + ]); + }); + + it("reports a retirement as retired, folded with the changes around it into one call, and goes on", async () => { + const gate = Promise.withResolvers(); + const { source, advance } = jsonSource({ n: 0 }); + const calls: WatchValueCall<{ n: number }>[] = []; + await store((_, self) => { + self.on(source, async (call) => { + calls.push(call); + if (calls.length === 1) await gate.promise; + }); + }); + await flushed(); + // A Durable document retired (the last value, no ops) and created again, while the first call runs. + advance({ kind: "value", value: { n: 0 }, ops: [], retired: true }); + advance({ kind: "value", value: { n: 5 }, ops: [["r", { n: 5 }]], retired: false }); + gate.resolve(); + await settled(); + advance({ kind: "value", value: { n: 6 }, ops: [["s", ["n"], 6]], retired: false }); + await flushed(); + expect(calls).toEqual([ + { kind: "value", value: { n: 0 }, ops: [["r", { n: 0 }]], retired: false }, + { kind: "value", value: { n: 5 }, ops: [["r", { n: 5 }]], retired: true }, + { kind: "value", value: { n: 6 }, ops: [["s", ["n"], 6]], retired: false }, + ]); + }); + + it("folds a native source to its newest value after a busy call", async () => { + const gate = Promise.withResolvers(); + const { source } = writable("a"); + const calls: unknown[] = []; + await store((_, self) => { + self.on(source, async ({ value }: { value: string }) => { + calls.push(value); + if (calls.length === 1) await gate.promise; + }); + }); + await flushed(); + source.set("b"); + source.set("c"); + await flushed(); + gate.resolve(); + await settled(); + expect(calls).toEqual(["a", "c"]); + }); + + it("takes every value a busy call left buffered before calling again: [0, 20], also when it set them itself", async () => { + const { source } = writable(0); + const calls: unknown[] = []; + await store((_, self) => { + self.on(source, async ({ value }: { value: number }) => { + calls.push(value); + if (value === 0) for (let n = 1; n <= 20; n++) source.set(n); + }); + }); + await settled(); + expect(calls).toEqual([0, 20]); + const gate = Promise.withResolvers(); + const later = writable(0); + const seen: unknown[] = []; + await store((_, self) => { + self.on(later.source, async ({ value }: { value: number }) => { + seen.push(value); + if (value === 0) await gate.promise; + }); + }); + await flushed(); + for (let n = 1; n <= 20; n++) later.source.set(n); + // No flush in between: the watch still holds the values when the call ends. + gate.resolve(); + await settled(); + expect(seen).toEqual([0, 20]); + }); + + it("reports a source whose watch cannot be taken; a source that ended meanwhile ends it quietly", async () => { + const failing: NativeWatchable = { + value: 0, + watch: async () => { + throw new Error("no watch"); + }, + }; + const ended: NativeWatchable = { + value: 0, + watch: async () => { + const watch = new NativeWatch( + 0, + () => {}, + () => {}, + ); + watch.end({ kind: "ended" }); + return watch; + }, + }; + const calls: unknown[] = []; + const { errors } = await store((_, self) => { + self.on(failing, async ({ value }) => void calls.push(value)); + self.on(ended, async ({ value }) => void calls.push(value)); + }); + await flushed(); + expect(calls).toEqual([]); + expect(reports(errors)).toEqual(["on #1: no watch"]); + }); + + it("ends without a report when its source ends", async () => { + const Other = local("app.other", { version: 1, params: { id: int() }, members: { n: int() } }); + let otherDoc: SingletonHandle<{ n: number }> | undefined; + const calls: unknown[] = []; + const { host, errors } = await loaded((h) => { + h.serve(Other, (_, self) => { + otherDoc = self.state(Doc); + return { n: self.derive([otherDoc], ([d]) => d.n) }; + }); + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + // Another object's document: it ends with that object. + self.on(otherDoc!, async ({ value }) => void calls.push(value)); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }); + }); + const other = await host.get(Other, { id: 1 }, context); + await host.get(Item, { id: 1 }, context); + await flushed(); + other.release(); + await flushed(); + expect(calls).toEqual([{ n: 0 }]); + expect(errors).toEqual([]); + }); + + it("is not called for its source's failures: it waits for the next value (a derive healing)", async () => { + const calls: unknown[] = []; + const { ref, errors } = await store((doc, self) => { + const failing = self.derive([doc], ([d]) => { + if (d.n > 0 && d.n < 3) throw new Error("compute"); + return d.n; + }); + self.on(failing, async ({ value, ops }) => void calls.push([value, ops])); + }); + await flushed(); + await ref.bump(context); + await ref.bump(context); + await flushed(); + expect(calls).toEqual([[0, [["r", 0]]]]); + await ref.bump(context); + await flushed(); + expect(calls).toEqual([ + [0, [["r", 0]]], + [3, [["r", 3]]], + ]); + // The derive's owner reports the failure (once for the episode); the routine reports nothing. + expect(errors).toEqual([expect.objectContaining({ kind: "derive_failed", error: new Error("compute") })]); + }); + + it("drops a value queued behind a busy call when a failure follows it; the next call is the healed value", async () => { + const calls: unknown[] = []; + const gate = Promise.withResolvers(); + const entered = Promise.withResolvers(); + let set: ((n: number) => Promise) | undefined; + const { errors } = await store((doc, self) => { + const failing = self.derive([doc], ([d]) => { + if (d.n < 0) throw new Error("compute"); + return d.n; + }); + set = (n) => + doc.change((d) => { + d.n = n; + }, context); + self.on(failing, async ({ value }) => { + calls.push(value); + if (value === 1) { + entered.resolve(); + await gate.promise; + } + }); + }); + await flushed(); + await set!(1); + await entered.promise; + // Queued behind the busy call, then replaced by the failure. + await set!(2); + await set!(-1); + gate.resolve(); + await settled(); + expect(calls).toEqual([0, 1]); + await set!(3); + await settled(); + expect(calls).toEqual([0, 1, 3]); + expect(reports(errors)).toEqual([]); + }); + + it("starts on a source failed now: no call and no report until its first good value (a reference member)", async () => { + const Failing = local("app.failing", { version: 1, params: {}, members: { t: int(), set: method(int()) } }); + const Viewer = local("app.on-viewer", { version: 1, params: { id: int() }, members: { m: method() } }); + const calls: unknown[] = []; + const { host, errors } = await loaded((h) => { + h.serve(Failing, (_, self) => { + const doc = self.state(Doc); + return { + t: self.derive([doc], ([d]) => { + if (d.n === 1) throw new Error("t threw"); + return d.n; + }), + set: async (n, ctx) => { + await doc.change((d) => { + d.n = n; + }, ctx); + return null; + }, + }; + }); + h.serve(Viewer, (_, self) => { + self.on(self.get(Failing, {}).t, async ({ value, ops }) => void calls.push([value, ops])); + return { m: async () => null }; + }); + }); + const failing = await host.get(Failing, {}, context); + await failing.set(1, context); + const viewer = await host.get(Viewer, { id: 1 }, context); + await settled(); + expect(calls).toEqual([]); + await failing.set(2, context); + await settled(); + expect(calls).toEqual([[2, [["r", 2]]]]); + expect(errors.map(({ kind }) => kind)).toEqual(["derive_failed"]); + viewer.release(); + await host.close(context); + }); + + it("follows a reference member: the provider's value first, then its changes", async () => { + const calls: [unknown, readonly Op[]][] = []; + let bumper: (() => Promise) | undefined; + const { host } = await loaded((h) => { + h.serve(Store, (_, self) => { + const doc = self.state(Doc); + bumper = () => bump(doc, context); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }); + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + const store = self.get(Store, {}); + self.on(store.n, async ({ value, ops }) => void calls.push([value, ops])); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }); + }); + await host.get(Item, { id: 1 }, context); + await flushed(); + await bumper!(); + await flushed(); + expect(calls).toEqual([ + [0, [["r", 0]]], + [1, [["r", 1]]], + ]); + }); + + it("converges when it writes its own source: an unchanged draft commits nothing", async () => { + let calls = 0; + await store((doc, self) => { + self.on(doc, async ({ value }, ctx) => { + calls++; + await doc.change((d) => { + d.n = Math.min(value.n + 1, 3); + }, ctx); + }); + }); + await new Promise((resolve) => setTimeout(resolve, 20)); + // 0 → 1 → 2 → 3, then the write of 3 changes nothing and nothing calls again. + expect(calls).toBe(4); + }); + + it("follows a native source: one call per new value, without ops; ends with it", async () => { + const { source, watches } = writable("a"); + const calls: unknown[] = []; + const { errors } = await store((_, self) => { + self.on(source, async (call) => void calls.push(call)); + }); + await flushed(); + source.set("b"); + source.set("c"); + await settled(); + expect(calls).toEqual([ + { kind: "value", value: "a" }, + { kind: "value", value: "b" }, + { kind: "value", value: "c" }, + ]); + source.end(); + expect(watches.size).toBe(0); + expect(errors).toEqual([]); + }); + + it("on a remote member with no value yet, a first value followed at once by a failure calls nothing", async () => { + const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: {} }); + const calls: unknown[] = []; + const client = createHost({ onError: () => {} }); + const facet = defineFacet((h) => + h.serve(Viewer, (_, self) => { + self.on(self.get(Notes, { session: "S1" }).count, async ({ value }) => void calls.push(value)); + return {}; + }), + ); + await client.load([{ name: "@test/client", facet }], context); + const peer = await rawParent(client); + const getting = client.get(Viewer, { id: 1 }, context); + const { hold } = await peer.next("hold"); + const member = EDITOR.state.count!; + const failed = [{ member, error: { kind: "plugin", code: "internal", message: "count threw" } } as const]; + const notes = { member: EDITOR.state.notes!, value: jsonPart([]) }; + peer.send({ kind: "bound", holds: [{ hold, variant: EDITOR.variant, members: [notes], failed }] }); + await getting; + await transported(); + // Two state messages in one receive turn: the first value, then a failure again. + const value = (n: number) => [{ member, ops: jsonPart([["r", n]]) }]; + peer.send({ kind: "state", holds: [{ hold, members: value(5), failed: [], retired: [] }] }); + peer.send({ kind: "state", holds: [{ hold, members: [], failed, retired: [] }] }); + await transported(); + await flushed(); + expect(calls).toEqual([]); + peer.send({ kind: "state", holds: [{ hold, members: value(6), failed: [], retired: [] }] }); + await transported(); + await flushed(); + expect(calls).toEqual([6]); + await client.close(context); + }); + + it("on a remote member failed since its binding (no value yet), waits for its first value", async () => { + const Halves = remote("app.halves", { + version: 1, + params: {}, + variants: { v: { state: { half: int() }, methods: { set: method(int()) } } }, + }); + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const serve = defineFacet((h) => + h.serve(Halves, (_, _principal, self) => { + const doc = self.state(Doc); + return { + variant: "v", + members: { + half: self.derive([doc], ([{ n }]) => { + if (n === 2) throw new Error("half threw"); + return n / 2; + }), + set: async (n, ctx) => { + await doc.change((d) => { + d.n = n; + }, ctx); + return null; + }, + }, + }; + }), + ); + await server.load([{ name: "@test/app", facet: serve }], context); + const halves = await server.get(Halves, {}, context); + if (halves.variant !== "v") throw new Error("expected variant v"); + await halves.set(2, context); + const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: {} }); + const calls: unknown[] = []; + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const facet = defineFacet((h) => + h.serve(Viewer, (_, self) => { + self.on(self.get(Halves, {}).half, async ({ value, ops }) => void calls.push([value, ops])); + return {}; + }), + ); + await client.load([{ name: "@test/client", facet }], context); + const [a, b] = memoryLinkPair(); + // The server's own principal: the client holds the object `halves` refers to. + server.accept(a, { principal: { kind: "server" } }, context); + client.connect(b, { principal: { kind: "server" } }, context); + await client.get(Viewer, { id: 1 }, context); + await transported(); + await flushed(); + expect([calls, errors]).toEqual([[], []]); + await halves.set(4, context); + await transported(); + await flushed(); + expect(calls).toEqual([[2, [["r", 2]]]]); + await client.close(context); + await server.close(context); + }); + + it("follows ref.state: a reference over a link going loading and live again", async () => { + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + await server.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); + const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: {} }); + const states: string[] = []; + const client = createHost({ onError: () => {} }); + const facet = defineFacet((h) => + h.serve(Viewer, (_, self) => { + const notes = self.get(Notes, { session: "S1" }); + self.on(notes.state, async ({ value: state }: { value: RefState }) => void states.push(state.kind)); + return {}; + }), + ); + await client.load([{ name: "@test/client", facet }], context); + const connect = () => { + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" } }, context); + client.connect(b, { principal: { kind: "server" } }, context); + return a; + }; + const link = connect(); + await client.get(Viewer, { id: 1 }, context); + await flushed(); + link.close(); + await transported(); + connect(); + await transported(); + await flushed(); + expect(states).toEqual(["live", "loading", "live"]); + await client.close(context); + await server.close(context); + }); + + it("releases its watch when its object ends; a call that throws as it ends is not reported", async () => { + const { source, watches } = writable(0); + const { host, errors } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(source, async (_change, ctx) => { + await untilAborted(ctx); + throw new Error("stopping"); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await flushed(); + expect(watches.size).toBe(1); + item.release(); + await flushed(); + expect(watches.size).toBe(0); + expect(errors).toEqual([]); + }); +}); + +describe("onLive", () => { + fakeTimers(); + + it("is called once when its object is live; returning ends it", async () => { + let calls = 0; + await store((_, self) => { + self.onLive(async () => { + calls++; + }); + }); + await vi.advanceTimersByTimeAsync(10_000); + expect(calls).toBe(1); + }); + + it("restarts after a throw with the backoff, reset after a run of at least a minute", async () => { + const starts: number[] = []; + const { errors } = await store((_, self) => { + self.onLive(async (ctx) => { + starts.push(Date.now()); + if (starts.length === 3) await sleep(60_000, ctx); + throw new Error(`run ${starts.length}`); + }); + }); + await vi.advanceTimersByTimeAsync(0); + await vi.advanceTimersByTimeAsync(1_000); + await vi.advanceTimersByTimeAsync(2_000 + 60_000); + await vi.advanceTimersByTimeAsync(1_000); + expect(starts).toEqual([0, 1_000, 3_000, 64_000]); + expect(reports(errors)).toEqual(["onLive #1: run 1", "onLive #1: run 2", "onLive #1: run 3", "onLive #1: run 4"]); + }); + + it("is aborted by its object's end: no report, no restart", async () => { + let calls = 0; + const { host, errors } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.onLive(async (ctx) => { + calls++; + await sleep(1_000_000, ctx); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await vi.advanceTimersByTimeAsync(0); + item.release(); + await vi.advanceTimersByTimeAsync(10_000); + expect({ calls, errors, timers: vi.getTimerCount() }).toEqual({ calls: 1, errors: [], timers: 0 }); + }); +}); + +describe("routines and the end", () => { + it("start in the turn their object went live, next to a method call admitted in that turn", async () => { + const order: string[] = []; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(doc, async ({ value }) => void order.push(`on ${value.n}`)); + self.onLive(async () => void order.push("onLive")); + return { + n: self.derive([doc], ([d]) => d.n), + bump: async (ctx) => { + order.push("bump"); + return bump(doc, ctx); + }, + }; + }), + ); + const item = host.hold(Item, { id: 1 }, context); + const bumped = item.bump(context); + // The parked call runs as the object goes live, then `onLive` in that turn, the first `on` call after its watch; + // the call's commit may land while `on 0` runs, and a call after a busy one waits for the settling turn. + await item.state.watch(context).then((watch) => watch.stop()); + await bumped; + await settled(); + expect(order).toEqual(["bump", "onLive", "on 0", "on 1"]); + }); + + it("never start a routine call once the object ended in its first turn: no call, no report, no rejection", async () => { + const calls: string[] = []; + let held: { release(): void } | undefined; + const { host, errors } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(doc, async () => void calls.push("on")); + self.onLive(async () => { + calls.push("onLive #1"); + held!.release(); + }); + self.onLive(async () => void calls.push("onLive #2")); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + held = host.hold(Item, { id: 1 }, context); + await new Promise((resolve) => setTimeout(resolve, 10)); + expect({ calls, errors }).toEqual({ calls: ["onLive #1"], errors: [] }); + }); + + it("are in the drain before their handler runs: a handler releasing the last reference keeps natives open", async () => { + const gate = Promise.withResolvers(); + const events: string[] = []; + let held: { release(): void } | undefined; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.open( + async () => null, + () => void events.push("native closed"), + ); + self.on(doc, async ({ value }) => { + if (value.n === 0) return; + // Ends its own object before its first await. + held!.release(); + await gate.promise; + events.push("handler done"); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + held = item; + await flushed(); + // The call may see its object end: its own routine ends it. + await item.bump(context).catch(() => {}); + await flushed(); + expect(events).toEqual([]); + gate.resolve(); + await flushed(); + expect(events).toEqual(["handler done", "native closed"]); + }); + + it("make no call queued behind a busy one once the object ended", async () => { + const gate = Promise.withResolvers(); + const calls: unknown[] = []; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(doc, async ({ value }) => { + calls.push(value); + await gate.promise; + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await flushed(); + await item.bump(context); + await flushed(); + item.release(); + gate.resolve(); + await flushed(); + expect(calls).toEqual([{ n: 0 }]); + }); + + it("release their calls' ctx.get references when each call settles", async () => { + const Other = local("app.other", { version: 1, params: { id: int() }, members: { n: int() } }); + const { host } = await loaded((h) => { + h.serve(Other, (_, self) => ({ n: self.derive([self.state(Doc)], ([d]) => d.n) })); + h.serve(Store, (_, self) => { + const doc = self.state(Doc); + self.onLive(async (ctx) => { + await ctx.get(Other, { id: 1 }); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }); + }); + await flushed(); + expect(host.references(Other).value).toEqual([]); + }); + + it("act as a remote object's principal", async () => { + const Watcher = remote("app.watcher", { version: 1, params: {}, variants: { v: { state: {} } } }); + const seen: unknown[] = []; + const { client } = await linked( + (h) => + h.serve(Watcher, (_, _principal, self) => { + self.onLive(async (ctx) => void seen.push(ctx.principal)); + return { variant: "v", members: {} }; + }), + { child: { kind: "guest", session: "S1" } }, + ); + await client.get(Watcher, {}, context); + await flushed(); + expect(seen).toEqual([{ kind: "guest", session: "S1" }]); + }); + + it("type their handler by the source's kind: ops and retired for a JSON watchable, the value alone for a native one", () => { + const declare = (self: Self, doc: Watchable<{ n: number }>, state: NativeWatchable) => { + self.on(doc, async (call, ctx) => { + expectTypeOf(call).toEqualTypeOf>(); + expectTypeOf(call.ops).toEqualTypeOf(); + expectTypeOf(call.retired).toEqualTypeOf(); + expectTypeOf(ctx).toEqualTypeOf>(); + }); + self.on(state, async (call, ctx) => { + expectTypeOf(call).toEqualTypeOf>(); + expectTypeOf(ctx).toEqualTypeOf>(); + }); + }; + void declare; + }); + + it("are aborted by host.close, which drains their running calls (bounded by closeTimeout)", async () => { + const ended: string[] = []; + const { host, errors } = await loaded( + (h) => { + h.serve(Store, (_, self) => { + const doc = self.state(Doc); + self.onLive(async (ctx) => { + await new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", resolve)); + await new Promise((resolve) => setTimeout(resolve, 10)); + ended.push("object"); + }); + self.onLive(async () => { + // Never stops: the drain gives up at closeTimeout. + await new Promise(() => {}); + }); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }); + }, + { closeTimeout: 50 }, + ); + await flushed(); + await host.close(context); + expect(ended).toEqual(["object"]); + expect(errors).toEqual([{ kind: "drain_timeout", object: "app.store" }]); + }); + + it("leave nothing of an ended object: its routines' watches and contexts are collected", async () => { + const { source, watches } = writable(0); + let refs: WeakRef[] = []; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + const doc = self.state(Doc); + self.on(source, async (_change, ctx) => void refs.push(new WeakRef(ctx))); + self.on(doc, async (_change, ctx) => void refs.push(new WeakRef(ctx))); + return { n: self.derive([doc], ([d]) => d.n), bump: async (ctx) => bump(doc, ctx) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await flushed(); + item.release(); + await flushed(); + expect({ watches: watches.size, calls: refs.length }).toEqual({ watches: 0, calls: 2 }); + await expectCollected(...refs); + refs = []; + }); +}); diff --git a/packages/chord/test/next/host/sleep.test.ts b/packages/chord/test/next/host/sleep.test.ts new file mode 100644 index 000000000..93c5fb9ab --- /dev/null +++ b/packages/chord/test/next/host/sleep.test.ts @@ -0,0 +1,74 @@ +// Tests of the waiting helpers (§14): `sleep` resolving, rejecting with the abort reason, chaining delays above the +// timer limit and leaving no timer behind; `untilAborted`. + +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT, withCancel } from "../../../src/context/index.ts"; +import { sleep, untilAborted } from "../../../src/next/host/index.ts"; +import { fakeTimers } from "../support.ts"; + +describe("sleep", () => { + fakeTimers(); + + it("resolves after its delay", async () => { + let done = false; + void sleep(1_000, BACKGROUND_CONTEXT).then(() => { + done = true; + }); + await vi.advanceTimersByTimeAsync(999); + expect(done).toBe(false); + await vi.advanceTimersByTimeAsync(1); + expect(done).toBe(true); + expect(vi.getTimerCount()).toBe(0); + }); + + it("rejects with the abort reason and clears its timer", async () => { + const { context, cancel } = withCancel(BACKGROUND_CONTEXT); + const waiting = sleep(60_000, context); + const reason = new Error("stop"); + cancel(reason); + await expect(waiting).rejects.toBe(reason); + expect(vi.getTimerCount()).toBe(0); + }); + + it("rejects at once when its context is aborted already", async () => { + const { context, cancel } = withCancel(BACKGROUND_CONTEXT); + cancel(new Error("before")); + await expect(sleep(10, context)).rejects.toThrow("before"); + expect(vi.getTimerCount()).toBe(0); + }); + + it("chains timers for delays above 2^31-1 ms; an abort clears the chained timer", async () => { + const limit = 2 ** 31 - 1; + let done = false; + void sleep(limit + 5_000, BACKGROUND_CONTEXT).then(() => { + done = true; + }); + await vi.advanceTimersByTimeAsync(limit); + expect(done).toBe(false); + expect(vi.getTimerCount()).toBe(1); + await vi.advanceTimersByTimeAsync(5_000); + expect(done).toBe(true); + const { context, cancel } = withCancel(BACKGROUND_CONTEXT); + const long = sleep(3 * limit, context); + await vi.advanceTimersByTimeAsync(limit + 1); + cancel(new Error("late")); + await expect(long).rejects.toThrow("late"); + expect(vi.getTimerCount()).toBe(0); + }); +}); + +describe("untilAborted", () => { + it("resolves once its context is aborted, at once when it is already", async () => { + const { context, cancel } = withCancel(BACKGROUND_CONTEXT); + let done = false; + const waiting = untilAborted(context).then(() => { + done = true; + }); + await Promise.resolve(); + expect(done).toBe(false); + cancel(); + await waiting; + expect(done).toBe(true); + await untilAborted(context); + }); +}); diff --git a/packages/chord/test/next/jobs/definition.test.ts b/packages/chord/test/next/jobs/definition.test.ts new file mode 100644 index 000000000..4f16502ac --- /dev/null +++ b/packages/chord/test/next/jobs/definition.test.ts @@ -0,0 +1,106 @@ +// Tests of job definitions (§15): what `defineJob` checks at declaration (names, versions, closed JSON args, a +// singleton state, concurrency, the shape of `accepts`, `persist` refused before slice 6) and the types it gives. + +import { describe, expect, expectTypeOf, it } from "vitest"; +import { bytes, int, map, obj, str, unknown } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { defineJob, type JobDefinition } from "../../../src/next/jobs/index.ts"; + +const Progress = defineState("app.progress", { + schema: obj({ output: str() }), + version: 1, + initial: () => ({ output: "" }), +}); + +const Keyed = defineState("app.keyed", { + schema: obj({ output: str() }), + version: 1, + initial: () => ({ output: "" }), + keyed: true, +}); + +const args = obj({ command: str() }); + +describe("defineJob", () => { + it("keeps its name, version, Chord's copies of its schemas, its state and concurrency", () => { + const Shell = defineJob("app.shell", { version: 2, args, result: int(), state: Progress, concurrency: 1 }); + expect(Shell).toMatchObject({ name: "app.shell", version: 2, state: Progress, concurrency: 1, accepts: {} }); + expect(Shell.args.schema).not.toBe(args); + expectTypeOf(Shell).toEqualTypeOf>(); + const Plain = defineJob("app.plain", { version: 1, args, result: str() }); + expectTypeOf(Plain).toEqualTypeOf>(); + }); + + it('takes a restricted name only: lowercase [a-z0-9@/._~-] (decision "Restricted names")', () => { + for (const name of ["@app/jobs.run_1~x-y", "a"]) + expect(defineJob(name, { version: 1, args, result: int() }).name).toBe(name); + for (const name of ["", "App.run", "app:run", "app run", "a\0b", "a\uD800", "é"]) { + expect(() => defineJob(name, { version: 1, args, result: int() })).toThrow( + `A job name ${JSON.stringify(name)}: expected lowercase [a-z0-9@/._~-]`, + ); + } + // Plugin code, not type-checked. + expect(() => defineJob(42 as never, { version: 1, args, result: int() })).toThrow("A job name 42"); + }); + + it("refuses a bad name, version, unknown option or concurrency", () => { + expect(() => defineJob("", { version: 1, args, result: int() })).toThrow("A job name"); + expect(() => defineJob("j", { version: 0, args, result: int() })).toThrow( + "j.version: expected a positive integer", + ); + // Plugin code, not type-checked. + const unknownKey = { version: 1, args, result: int(), retries: 3 } as never; + expect(() => defineJob("j", unknownKey)).toThrow('j: unknown key "retries"'); + for (const concurrency of [0, 1.5, -1]) { + expect(() => defineJob("j", { version: 1, args, result: int(), concurrency })).toThrow( + "j.concurrency: expected a positive integer", + ); + } + }); + + it("requires closed JSON args, as a method's: open objects and bytes are refused; records are fine", () => { + const open = { type: "object", properties: { a: str() } } as never; + expect(() => defineJob("j", { version: 1, args: open, result: int() })).toThrow( + "argument objects must be closed", + ); + expect(() => defineJob("j", { version: 1, args: obj({ data: bytes() }), result: int() })).toThrow( + "j.args: job arguments are JSON", + ); + expect(() => defineJob("j", { version: 1, args: obj({ tags: map(str()) }), result: unknown() })).not.toThrow(); + }); + + it("takes a singleton document definition as its state", () => { + expect(() => defineJob("j", { version: 1, args, result: int(), state: Keyed as never })).toThrow( + "j.state: expected a singleton document definition", + ); + }); + + it("checks accepts at declaration: older versions only, each with an args and a result function", () => { + const adapter = { args: (old: { cmd: string }) => ({ command: old.cmd }), result: (old: number) => old }; + expect(() => + defineJob("j", { version: 3, args, result: int(), accepts: { 1: adapter, 2: adapter } }), + ).not.toThrow(); + expect(() => defineJob("j", { version: 2, args, result: int(), accepts: { 2: adapter } })).toThrow( + "j.accepts[2]: not an older version", + ); + expect(() => defineJob("j", { version: 2, args, result: int(), accepts: { 0: adapter } })).toThrow( + "j.accepts[0]: not an older version", + ); + const noResult = { 1: { args: adapter.args } } as never; + expect(() => defineJob("j", { version: 2, args, result: int(), accepts: noResult })).toThrow( + "j.accepts[1].result: expected a function", + ); + const extra = { 1: { ...adapter, state: () => null } } as never; + expect(() => defineJob("j", { version: 2, args, result: int(), accepts: extra })).toThrow( + 'j.accepts[1]: unknown key "state"', + ); + }); + + it("refuses persist before slice 6, in its final shape", () => { + expect(() => defineJob("j", { version: 1, args, result: int(), persist: { resume: "rerun" } })).toThrow( + "j.persist: persisted jobs are not supported before slice 6", + ); + // @ts-expect-error: resume is required + expect(() => defineJob("j", { version: 1, args, result: int(), persist: {} })).toThrow(); + }); +}); diff --git a/packages/chord/test/next/jobs/job.test.ts b/packages/chord/test/next/jobs/job.test.ts new file mode 100644 index 000000000..a868d0f04 --- /dev/null +++ b/packages/chord/test/next/jobs/job.test.ts @@ -0,0 +1,1378 @@ +// Tests of jobs in memory (§15): starts (args, dedupe, concurrency, `tx.start`, `outcome`, when `run` starts), runs +// (the ctx, the state document and its fence, throws and aborts), finishes (cleanup, retries, blocked, dropped), +// `running`, keep-alive, the object's end, cost guards and GC. + +import { describe, expect, expectTypeOf, it, vi } from "vitest"; +import { + type Context, + BACKGROUND_CONTEXT as context, + withAbortSignal, + withCancel, +} from "../../../src/context/index.ts"; +import type { Op } from "../../../src/delta/index.ts"; +import { arr, int, local, obj, remote, str, unknown } from "../../../src/next/contracts/index.ts"; +import { ChordError, PluginError, type Principal } from "../../../src/next/data/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type HostError, type Self, untilAborted } from "../../../src/next/host/index.ts"; +import { + defineJob, + type JobCtx, + type JobDefinition, + type JobHandle, + type JobImplementation, + type Outcome, + type Run, +} from "../../../src/next/jobs/index.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { Batch } from "../../../src/next/state/internal.ts"; +import { + delivered, + expectCollected, + fakeTimers, + flushed, + framesOf, + loaded, + metered, + rejection, + type TestPrincipal, + transported, +} from "../support.ts"; + +const Log = defineState("app.log", { schema: obj({ list: arr(str()) }), version: 1, initial: () => ({ list: [] }) }); + +const Progress = defineState("app.progress", { + schema: obj({ output: str() }), + version: 1, + initial: () => ({ output: "" }), +}); + +/** A job without a state document. */ +const Echo = defineJob("app.echo", { version: 1, args: obj({ text: str() }), result: str() }); + +/** A job with a state document and one run at a time. */ +const Shell = defineJob("app.shell", { + version: 1, + args: obj({ command: str() }), + result: obj({ exit: int() }), + state: Progress, + concurrency: 1, +}); + +/** The host-lifetime object of most tests: its log document, and the jobs the test declares. */ +const Store = local("app.store", { version: 1, params: {}, members: { log: obj({ list: arr(str()) }) } }); + +/** The same with params: it lives while referenced or running jobs. */ +const Item = local("app.item", { version: 1, params: { id: int() }, members: { log: obj({ list: arr(str()) }) } }); + +type Declared = { + readonly self: Self; + readonly log: SingletonHandle<{ list: string[] }>; + readonly job: JobHandle; + readonly errors: HostError[]; +}; + +/** Loads a host serving `Store` with the job `definition`; `implement` gets the object's `self` and log. */ +async function declared( + definition: JobDefinition, + implement: (self: Self, log: SingletonHandle<{ list: string[] }>) => JobImplementation, + options: Parameters[1] = {}, +): Promise & { readonly host: Awaited>["host"] }> { + let found: Omit, "errors"> | undefined; + const { host, errors } = await loaded( + (h) => + h.serve(Store, (_, self) => { + const log = self.state(Log); + found = { self, log, job: self.job(definition, implement(self, log)) }; + return { log }; + }), + options, + ); + return { ...found!, errors, host }; +} + +/** Appends `entry` to the log in `tx`. */ +async function append( + tx: Parameters[0]>[0], + log: SingletonHandle<{ list: string[] }>, + entry: string, +) { + (await tx.draft(log)).list.push(entry); +} + +/** A `run` that waits for `gate`, then returns `text`. */ +function gated(gate: Promise) { + return async ({ text }: { text: string }) => { + await gate; + return text; + }; +} + +/** The reported job errors, as `kind id: message`. */ +function jobReports(errors: readonly HostError[]): string[] { + return errors.flatMap((error) => { + if (error.kind === "job_failed" || error.kind === "job_finish_failed") { + return [`${error.kind}: ${(error.error as Error).message}`]; + } + return error.kind === "job_dropped" ? [`${error.kind} ${error.job}`] : []; + }); +} + +describe("start", () => { + it("checks the args against the definition (invalid/args) before anything is recorded", async () => { + const { job } = await declared(Echo, () => ({ run: async ({ text }) => text })); + // Plugin code passing wrong args (not type-checked). + const wrong = { text: 1 } as unknown as { text: string }; + expect(await rejection(job.start(wrong, context))).toMatchObject({ code: "invalid", reason: "args" }); + const extra = { text: "a", more: true } as unknown as { text: string }; + expect(await rejection(job.start(extra, context))).toMatchObject({ code: "invalid", reason: "args" }); + expect(job.running.value).toEqual([]); + }); + + it("compiles the args check once per definition (cost guard)", async () => { + const { telemetry, context: counting, counts } = await metered(); + const { job } = await declared(Echo, () => ({ run: async ({ text }) => text }), { telemetry }); + const before = counts()["chord.data.compilations"] ?? 0; + for (const text of ["a", "b", "c"]) await (await job.start({ text }, counting)).outcome(context); + expect((counts()["chord.data.compilations"] ?? 0) - before).toBeLessThanOrEqual(1); + expect(counts()["chord.data.validations"]).toBeGreaterThanOrEqual(3); + }); + + it("returns the unsettled run with the same key, across transactions, whatever its args", async () => { + const gate = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ run: gated(gate.promise) })); + // Two starts in one turn: the line serializes them, the second sees the first's record. + const [first, again] = await Promise.all([ + job.start({ text: "a" }, context, { key: "k" }), + job.start({ text: "other" }, context, { key: "k" }), + ]); + expect(again).toBe(first); + expect(await job.start({ text: "third" }, context, { key: "k" })).toBe(first); + expect(job.running.value).toHaveLength(1); + gate.resolve(); + expect(await first.outcome(context)).toMatchObject({ kind: "done", result: "a" }); + // Settled: the key names nothing any more (Q9). + const next = await job.start({ text: "b" }, context, { key: "k" }); + expect(next.id).not.toBe(first.id); + }); + + it("commits nothing for a start that only dedupes", async () => { + const gate = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ run: gated(gate.promise) })); + const run = await job.start({ text: "a" }, context, { key: "k" }); + const commits = vi.spyOn(Batch.prototype, "commit"); + try { + expect(await job.start({ text: "a" }, context, { key: "k" })).toBe(run); + expect(commits).not.toHaveBeenCalled(); + } finally { + commits.mockRestore(); + gate.resolve(); + } + }); + + it("returns the same run for a key started twice in one transaction", async () => { + const { job, self } = await declared(Echo, () => ({ run: async ({ text }) => text })); + const [one, two] = await self.tx(async (tx) => { + return [tx.start(job, { text: "a" }, { key: "k" }), tx.start(job, { text: "b" }, { key: "k" })]; + }, context); + expect(two).toBe(one); + expect((await one!.outcome(context)).args).toEqual({ text: "a" }); + }); + + describe("at the concurrency limit", () => { + fakeTimers(); + + it("rejects unavailable/busy: running and finishing runs count", async () => { + const runGate = Promise.withResolvers(); + let finishes = 0; + const { job } = await declared(Shell, () => ({ + run: async () => { + await runGate.promise; + return { exit: 0 }; + }, + finish: async () => { + // The first finish throws: the run stays finishing during the backoff, the line free. + if (finishes++ === 0) throw new Error("once"); + }, + })); + const run = await job.start({ command: "a" }, context); + const busy = { code: "unavailable", reason: "busy" }; + expect(await rejection(job.start({ command: "b" }, context))).toMatchObject(busy); + runGate.resolve(); + await vi.advanceTimersByTimeAsync(0); + expect(job.running.value.map((entry) => entry.phase)).toEqual(["finishing"]); + expect(await rejection(job.start({ command: "b" }, context))).toMatchObject(busy); + await vi.advanceTimersByTimeAsync(1_000); + await run.outcome(context); + await (await job.start({ command: "c" }, context)).outcome(context); + }); + }); + + it("counts runs started earlier in the same transaction", async () => { + const { job, self } = await declared(Shell, () => ({ run: async () => ({ exit: 0 }) })); + const error = await rejection( + self.tx(async (tx) => { + tx.start(job, { command: "a" }); + tx.start(job, { command: "b" }); + }, context), + ); + expect(error).toMatchObject({ code: "unavailable", reason: "busy" }); + expect(job.running.value).toEqual([]); + }); + + it("lets finish chain the same job at concurrency 1: its own run is not counted", async () => { + const done: string[] = []; + const { job } = await declared(Shell, (_, log) => ({ + run: async ({ command }) => ({ exit: command.length }), + finish: async (outcome, tx) => { + done.push(outcome.args.command); + await append(tx, log, outcome.args.command); + if (outcome.args.command.length < 3) tx.start(job, { command: `${outcome.args.command}x` }); + }, + })); + await (await job.start({ command: "a" }, context)).outcome(context); + await flushed(); + await flushed(); + expect(done).toEqual(["a", "ax", "axx"]); + }); + + it("lets finish start the same job again with the same key: a new run, not the finishing one", async () => { + const gate = Promise.withResolvers(); + let successor: Run | undefined; + const { job } = await declared(Echo, () => ({ + run: async ({ text }) => { + if (text === "again") await gate.promise; + return text; + }, + finish: async (outcome, tx) => { + if (outcome.args.text === "first") successor = tx.start(job, { text: "again" }, { key: "k" }); + }, + })); + const first = await job.start({ text: "first" }, context, { key: "k" }); + expect(await first.outcome(context)).toMatchObject({ kind: "done", result: "first" }); + expect(successor!.id).not.toBe(first.id); + expect(job.running.value).toMatchObject([{ id: successor!.id, key: "k", phase: "running" }]); + // The key names the successor now. + expect(await job.start({ text: "other" }, context, { key: "k" })).toBe(successor); + gate.resolve(); + expect(await successor!.outcome(context)).toMatchObject({ kind: "done", result: "again" }); + }); + + it("tx.start commits with the transaction's other writes, both or neither", async () => { + const { job, self, log } = await declared(Echo, () => ({ run: async ({ text }) => text })); + const run = await self.tx(async (tx) => { + await append(tx, log, "requested"); + return tx.start(job, { text: "a" }); + }, context); + expect(log.value.list).toEqual(["requested"]); + expect((await run.outcome(context)).kind).toBe("done"); + let discarded: Run | undefined; + const error = await rejection( + self.tx(async (tx) => { + await append(tx, log, "lost"); + discarded = tx.start(job, { text: "b" }); + throw new Error("changed my mind"); + }, context), + ); + expect(error).toMatchObject({ message: "changed my mind" }); + expect(log.value.list).toEqual(["requested"]); + expect(job.running.value).toEqual([]); + expect(await rejection(discarded!.outcome(context))).toMatchObject({ code: "released" }); + expect(await rejection(job.outcome(discarded!.id, context))).toMatchObject({ code: "gone" }); + }); + + it("gives a short run's handle its outcome however late the caller waits", async () => { + const { job } = await declared(Echo, () => ({ run: async ({ text }) => text })); + const run = await job.start({ text: "quick" }, context); + await flushed(); + expect(job.running.value).toEqual([]); + expect(await run.outcome(context)).toEqual({ + id: run.id, + args: { text: "quick" }, + kind: "done", + result: "quick", + }); + }); + + it("waits by id for unsettled runs only: settled and unknown ids reject gone", async () => { + const gate = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ run: gated(gate.promise) })); + const run = await job.start({ text: "a" }, context); + const waiting = job.outcome(run.id, context); + gate.resolve(); + expect((await waiting).kind).toBe("done"); + expect(await rejection(job.outcome(run.id, context))).toMatchObject({ code: "gone" }); + expect(await rejection(job.outcome("nope", context))).toMatchObject({ code: "gone" }); + }); + + it("starts run after the commit, outside the batch, once the transaction settled", async () => { + const seen: string[] = []; + const { job, self, log } = await declared(Echo, (s, l) => ({ + run: async ({ text }) => { + seen.push(`run sees ${l.value.list.join(",")}`); + // Outside the batch and the line: a transaction of its own works. + await s.tx(async (tx) => append(tx, l, `from run ${text}`), context); + return text; + }, + })); + const run = await self.tx(async (tx) => { + const started = tx.start(job, { text: "a" }); + await append(tx, log, "written with the start"); + seen.push("transaction done"); + return started; + }, context); + await run.outcome(context); + expect(seen).toEqual(["transaction done", "run sees written with the start"]); + expect(log.value.list).toEqual(["written with the start", "from run a"]); + }); + + it("starts run with its signal aborted after an abort between the commit and its start", async () => { + let signalled: boolean | undefined; + const { job } = await declared(Echo, () => ({ + run: async ({ text }, ctx) => { + signalled = ctx.abortSignal?.aborted; + return text; + }, + })); + const watch = await job.running.watch(context); + // The watch's frame of the start's commit arrives before `run` starts. + watch.start(async (call) => { + if (call.kind !== "value") return; + for (const entry of call.value) job.abort(entry.id); + }); + const run = await job.start({ text: "a" }, context); + expect(await run.outcome(context)).toMatchObject({ kind: "done", result: "a" }); + expect(signalled).toBe(true); + await watch.stop(); + }); + + it("creates nothing for an aborted caller; a caller aborted after the commit does not cancel the run", async () => { + const gate = Promise.withResolvers(); + let signalled: boolean | undefined; + const { job } = await declared(Echo, () => ({ + run: async ({ text }, ctx) => { + await gate.promise; + signalled = ctx.abortSignal?.aborted; + return text; + }, + })); + const early = withCancel(context); + early.cancel(new Error("gone already")); + expect(await rejection(job.start({ text: "a" }, early.context))).toMatchObject({ message: "gone already" }); + expect(job.running.value).toEqual([]); + const caller = withCancel(context); + const run = await job.start({ text: "b" }, caller.context); + caller.cancel(new Error("caller left")); + gate.resolve(); + expect(await run.outcome(context)).toMatchObject({ kind: "done", result: "b" }); + expect(signalled).toBe(false); + }); + + it("rolls back the transaction's other writes when the state's initial() fails, naming the definition", async () => { + const Bad = defineState("app.bad-state", { + schema: obj({ output: str() }), + version: 1, + // Plugin code: a wrong initial value. + initial: () => ({ output: 1 }) as unknown as { output: string }, + }); + const BadJob = defineJob("app.bad", { version: 1, args: obj({ text: str() }), result: str(), state: Bad }); + const { job, self, log } = await declared(BadJob, () => ({ run: async ({ text }) => text })); + const error = await rejection( + self.tx(async (tx) => { + await append(tx, log, "lost"); + tx.start(job, { text: "a" }); + }, context), + ); + expect(error).toMatchObject({ message: "app.bad-state: initial() does not match the schema" }); + expect({ log: log.value.list, running: job.running.value }).toEqual({ log: [], running: [] }); + }); + + it("is refused before its object is live and once it ended", async () => { + let early: Promise | undefined; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + const log = self.state(Log); + const job = self.job(Echo, { run: async ({ text }) => text }); + self.open( + async () => { + early = job.start({ text: "a" }, context); + return null; + }, + () => {}, + ); + return { log }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + expect(await rejection(early!)).toMatchObject({ code: "not_ready" }); + item.release(); + }); +}); + +describe("self.job", () => { + it("is declared once per job name, only while the factory runs", async () => { + let later: Self | undefined; + const { host, errors } = await loaded((h) => + h.serve(Item, (_, self) => { + later = self; + self.job(Echo, { run: async ({ text }) => text }); + self.job(Echo, { run: async ({ text }) => text }); + return { log: self.state(Log) }; + }), + ); + expect(await rejection(host.get(Item, { id: 1 }, context))).toMatchObject({ + message: expect.stringContaining('job "app.echo" is declared twice'), + }); + expect(errors.map((error) => error.kind)).toEqual(["activation_failed"]); + expect(() => later!.job(Shell, { run: async () => ({ exit: 0 }) })).toThrow( + "app.item(1): self.job is valid only in the factory", + ); + }); +}); + +describe("run", () => { + it("gets a ctx with the run's id, the host's principal for a local object, and no state without one", async () => { + let seen: Record | undefined; + const host = createHost({ principal: { kind: "server" }, onError: () => {} }); + let job: JobHandle<{ text: string }, string, undefined> | undefined; + const facet = defineFacet((h) => + h.serve(Store, (_, self) => { + job = self.job(Echo, { + run: async ({ text }, ctx) => { + seen = { jobId: ctx.jobId, principal: ctx.principal, hasState: "state" in ctx }; + expectTypeOf(ctx).toEqualTypeOf>(); + return text; + }, + }); + return { log: self.state(Log) }; + }), + ); + await host.load([{ name: "@test/app", facet }], context); + const run = await job!.start({ text: "a" }, context); + await run.outcome(context); + expect(seen).toEqual({ jobId: run.id, principal: { kind: "server" }, hasState: false }); + await host.close(context); + }); + + it("reads and changes its state document; `running` carries it", async () => { + const gate = Promise.withResolvers(); + const values: string[] = []; + const { job } = await declared(Shell, () => ({ + run: async (_, ctx) => { + values.push(ctx.state.value.output); + await ctx.state.change((d) => { + d.output = "half"; + }, ctx); + values.push(ctx.state.value.output); + await gate.promise; + return { exit: 0 }; + }, + })); + const run = await job.start({ command: "a" }, context); + await flushed(); + expect(job.running.value).toEqual([ + { id: run.id, version: 1, args: { command: "a" }, state: { output: "half" }, phase: "running" }, + ]); + gate.resolve(); + expect(await run.outcome(context)).toMatchObject({ kind: "done", state: { output: "half" } }); + expect(values).toEqual(["", "half"]); + }); + + it("keeps its state documents apart from the object's own documents of the same definition", async () => { + const gate = Promise.withResolvers(); + const Many = defineJob("app.many", { version: 1, args: obj({ n: int() }), result: int(), state: Progress }); + let mine: SingletonHandle<{ output: string }> | undefined; + let job: JobHandle<{ n: number }, number, { output: string }> | undefined; + await loaded((h) => + h.serve(Store, (_, self) => { + mine = self.state(Progress); + job = self.job(Many, { + run: async ({ n }, ctx) => { + await ctx.state.change((d) => { + d.output = `run ${n}`; + }, ctx); + await gate.promise; + return n; + }, + }); + return { log: self.state(Log) }; + }), + ); + const runs = await Promise.all([job!.start({ n: 1 }, context), job!.start({ n: 2 }, context)]); + await flushed(); + expect(job!.running.value.map((entry) => entry.phase !== "blocked" && entry.state.output)).toEqual([ + "run 1", + "run 2", + ]); + expect(mine!.value).toEqual({ output: "" }); + gate.resolve(); + expect((await Promise.all(runs.map((run) => run.outcome(context)))).map((o) => o.state.output)).toEqual([ + "run 1", + "run 2", + ]); + }); + + it("refuses state changes once run settled (released), and never recreates the retired document", async () => { + let captured: JobCtx | undefined; + let inFinish: unknown; + const { job } = await declared(Shell, () => ({ + run: async (_, ctx) => { + captured = ctx; + return { exit: 0 }; + }, + // After run returned, before finish committed: refused; the outcome's state is the state at run's settle. + finish: async (outcome) => { + inFinish = [outcome.state, await rejection(captured!.state.change(() => {}, context))]; + }, + })); + await (await job.start({ command: "a" }, context)).outcome(context); + expect(inFinish).toEqual([{ output: "" }, expect.objectContaining({ code: "released" })]); + const late = captured!.state.change((d) => { + d.output = "late"; + }, context); + expect(await rejection(late)).toMatchObject({ code: "released" }); + expect(() => captured!.state.value).toThrow(ChordError); + expect(job.running.value).toEqual([]); + }); + + it("commits a change still waiting for the line when run settles, before the outcome is recorded", async () => { + const go = Promise.withResolvers(); + const line = Promise.withResolvers(); + const entered = Promise.withResolvers(); + let returned = false; + const { job, self } = await declared(Shell, () => ({ + run: async (_, ctx) => { + await go.promise; + // Admitted while run runs; it waits behind the transaction holding the line. + void ctx.state.change((d) => { + d.output = "last words"; + }, ctx); + returned = true; + return { exit: 0 }; + }, + })); + const run = await job.start({ command: "a" }, context); + const holding = self.tx(async () => { + entered.resolve(); + await line.promise; + }, context); + await entered.promise; + go.resolve(); + await flushed(); + // run settled with its change still waiting: nothing committed, the outcome not recorded yet. + expect(returned).toBe(true); + expect(job.running.value).toMatchObject([{ phase: "running", state: { output: "" } }]); + line.resolve(); + await holding; + expect(await run.outcome(context)).toMatchObject({ kind: "done", state: { output: "last words" } }); + }); + + it("commits changes made after an abort while run stops; a throw then ends aborted, not threw", async () => { + const { job, errors } = await declared(Shell, () => ({ + run: async (_, ctx) => { + await untilAborted(ctx); + await ctx.state.change((d) => { + d.output = "stopped cleanly"; + }, ctx); + throw new Error("interrupted"); + }, + })); + const run = await job.start({ command: "a" }, context); + await flushed(); + job.abort(run.id); + expect(await run.outcome(context)).toEqual({ + id: run.id, + args: { command: "a" }, + state: { output: "stopped cleanly" }, + kind: "failed", + error: { kind: "aborted", reason: "abort" }, + }); + expect(jobReports(errors)).toEqual([]); + }); + + it("keeps a result returned after an abort", async () => { + const { job } = await declared(Echo, () => ({ + run: async ({ text }, ctx) => { + await untilAborted(ctx); + return text; + }, + })); + const run = await job.start({ text: "kept" }, context); + await flushed(); + job.abort(run.id); + expect(await run.outcome(context)).toMatchObject({ kind: "done", result: "kept" }); + }); + + it("ends threw when run throws, reported as job_failed; a plain error becomes PluginError internal", async () => { + const plugin = new PluginError("quota", "too many"); + const { job, errors } = await declared(Echo, () => ({ + run: async ({ text }) => { + if (text === "plugin") throw plugin; + throw new Error("plain"); + }, + })); + const first = await (await job.start({ text: "plugin" }, context)).outcome(context); + expect(first).toMatchObject({ kind: "failed", error: { kind: "threw", error: plugin } }); + const second = await (await job.start({ text: "plain" }, context)).outcome(context); + expect(second.kind === "failed" && second.error).toMatchObject({ kind: "threw" }); + const thrown = second.kind === "failed" && second.error.kind === "threw" ? second.error.error : undefined; + expect(thrown).toBeInstanceOf(PluginError); + expect(thrown).toMatchObject({ code: "internal", message: "plain" }); + expect(jobReports(errors)).toEqual(["job_failed: too many", "job_failed: plain"]); + expect(errors[0]).toMatchObject({ kind: "job_failed", object: "app.store", job: "app.echo", id: first.id }); + }); + + it("releases its ctx.get references when it settles", async () => { + const Other = local("app.other", { + version: 1, + params: { id: int() }, + members: { log: obj({ list: arr(str()) }) }, + }); + let job: JobHandle<{ text: string }, string, undefined> | undefined; + const { host } = await loaded((h) => { + h.serve(Other, (_, self) => ({ log: self.state(Log) })); + h.serve(Store, (_, self) => { + job = self.job(Echo, { + run: async ({ text }, ctx) => { + await ctx.get(Other, { id: 1 }); + expect(host.references(Other).value).toHaveLength(1); + return text; + }, + }); + return { log: self.state(Log) }; + }); + }); + await (await job!.start({ text: "a" }, context)).outcome(context); + expect(host.references(Other).value).toHaveLength(0); + }); + + it("that ignores its signal at the object's end is drained up to closeTimeout; its later changes are refused", async () => { + const gate = Promise.withResolvers(); + const late = Promise.withResolvers(); + const { job, host, errors } = await declared( + Shell, + () => ({ + run: async (_, ctx) => { + await gate.promise; + late.resolve( + await rejection( + ctx.state.change((d) => { + d.output = "too late"; + }, ctx), + ), + ); + return { exit: 0 }; + }, + }), + { closeTimeout: 20 }, + ); + await job.start({ command: "a" }, context); + await flushed(); + await host.close(context); + expect(errors).toEqual([{ kind: "drain_timeout", object: "app.store" }]); + gate.resolve(); + expect(await late.promise).toMatchObject({ code: "released" }); + }); + + it("may start the same job again from run (recursion): a new run", async () => { + const ids: string[] = []; + const { job } = await declared(Echo, () => ({ + run: async ({ text }, ctx) => { + ids.push(ctx.jobId); + if (text.length < 3) await job.start({ text: `${text}+` }, ctx); + return text; + }, + })); + await (await job.start({ text: "a" }, context)).outcome(context); + await flushed(); + await flushed(); + expect(new Set(ids).size).toBe(3); + }); +}); + +describe("finish", () => { + it("gets each outcome kind; the record, key and state document go in its commit, atomically with its writes", async () => { + const outcomes: string[] = []; + const { job, log, self } = await declared(Shell, (_, l) => ({ + run: async ({ command }, ctx) => { + await ctx.state.change((d) => { + d.output = `out ${command}`; + }, ctx); + if (command === "fail") throw new Error("no"); + return { exit: 0 }; + }, + finish: async (outcome, tx) => { + outcomes.push(`${outcome.kind} ${outcome.state.output}`); + await append(tx, l, outcome.args.command); + }, + })); + // The log entry and the removed entry arrive in one commit: whenever the log's watch sees an entry, the run is gone. + const seen: [number, number][] = []; + (await log.watch(context)).start(async (call) => { + if (call.kind === "value") seen.push([call.value.list.length, job.running.value.length]); + }); + void self; + await (await job.start({ command: "ok" }, context, { key: "k" })).outcome(context); + await (await job.start({ command: "fail" }, context, { key: "k" })).outcome(context); + await delivered(); + expect(outcomes).toEqual(["done out ok", "failed out fail"]); + expect(seen).toEqual([ + [1, 0], + [2, 0], + ]); + expect(job.running.value).toEqual([]); + }); + + it("only cleans up without a finish", async () => { + const { job } = await declared(Shell, () => ({ run: async () => ({ exit: 3 }) })); + const run = await job.start({ command: "a" }, context, { key: "k" }); + expect(await run.outcome(context)).toMatchObject({ kind: "done", result: { exit: 3 } }); + expect(job.running.value).toEqual([]); + }); + + describe("throwing", () => { + fakeTimers(); + + it("is retried after 1 s doubling, each failure reported; a later success finishes the run", async () => { + const at: number[] = []; + const { job, errors } = await declared(Echo, () => ({ + run: async ({ text }) => text, + finish: async () => { + at.push(Date.now()); + if (at.length < 4) throw new Error(`finish ${at.length}`); + }, + })); + const run = await job.start({ text: "a" }, context); + await vi.advanceTimersByTimeAsync(10_000); + expect(at).toEqual([0, 1_000, 3_000, 7_000]); + expect(jobReports(errors)).toEqual([ + "job_finish_failed: finish 1", + "job_finish_failed: finish 2", + "job_finish_failed: finish 3", + ]); + expect((await run.outcome(context)).kind).toBe("done"); + }); + + it("blocks the run after five failures in a row: outcome, state and key kept, not counted; abort drops it", async () => { + const { job, errors } = await declared(Shell, () => ({ + run: async (_, ctx) => { + await ctx.state.change((d) => { + d.output = "kept"; + }, ctx); + return { exit: 0 }; + }, + finish: async () => { + throw new Error("always"); + }, + })); + const run = await job.start({ command: "a" }, context, { key: "k" }); + const { context: waiting, cancel } = withCancel(context); + const waiter = rejection(run.outcome(waiting)); + const dropped = rejection(run.outcome(context)); + await vi.advanceTimersByTimeAsync(60_000); + expect(jobReports(errors)).toHaveLength(5); + expect(job.running.value).toEqual([{ id: run.id, version: 1, phase: "blocked", reason: "finish_failed" }]); + // Its waiters keep waiting; their own context ends a wait. + cancel(new Error("gave up")); + expect(await waiter).toMatchObject({ message: "gave up" }); + // Dedupe returns it; a different key starts a new run (blocked runs do not count for concurrency). + expect(await job.start({ command: "b" }, context, { key: "k" })).toBe(run); + const other = await job.start({ command: "c" }, context, { key: "other" }); + await vi.advanceTimersByTimeAsync(0); + expect(job.running.value.map((entry) => entry.phase)).toEqual(["blocked", "finishing"]); + job.abort(run.id); + await vi.advanceTimersByTimeAsync(0); + expect(await dropped).toMatchObject({ code: "aborted" }); + expect(job.running.value.map((entry) => entry.id)).toEqual([other.id]); + expect(errors.filter((error) => error.kind === "job_dropped")).toEqual([ + { kind: "job_dropped", object: "app.store", job: "app.shell", id: run.id }, + ]); + }); + }); + + describe("backing off", () => { + fakeTimers(); + + it("leaves no timer behind when its object ends during the backoff", async () => { + const { job, host, errors } = await declared(Echo, () => ({ + run: async ({ text }) => text, + finish: async () => { + throw new Error("stuck"); + }, + })); + const run = rejection((await job.start({ text: "a" }, context)).outcome(context)); + await vi.advanceTimersByTimeAsync(0); + expect(errors).toHaveLength(1); + expect(vi.getTimerCount()).toBe(1); + await host.close(context); + expect(vi.getTimerCount()).toBe(0); + await vi.advanceTimersByTimeAsync(60_000); + expect(errors).toHaveLength(1); + expect(await run).toMatchObject({ code: "released" }); + }); + }); + + it("ignores an abort while it runs", async () => { + const gate = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ + run: async ({ text }) => text, + finish: async () => gate.promise, + })); + const run = await job.start({ text: "a" }, context); + await flushed(); + job.abort(run.id); + gate.resolve(); + expect((await run.outcome(context)).kind).toBe("done"); + }); + + it("does not run when its object ends while the outcome and finish wait for the line: no call, no report", async () => { + const go = Promise.withResolvers(); + const line = Promise.withResolvers(); + const entered = Promise.withResolvers(); + const finished: string[] = []; + const { job, self, host, errors } = await declared(Echo, () => ({ + run: async ({ text }) => { + await go.promise; + return text; + }, + finish: async ({ args }) => void finished.push(args.text), + })); + const run = await job.start({ text: "a" }, context); + const waiting = rejection(run.outcome(context)); + const holding = self.tx(async () => { + entered.resolve(); + await line.promise; + }, context); + await entered.promise; + // run settles while the line is held: its outcome and finish queue behind it. + go.resolve(); + await flushed(); + const closing = host.close(context); + line.resolve(); + await holding.catch(() => {}); + await closing; + expect({ finished, errors }).toEqual({ finished: [], errors: [] }); + expect(await waiting).toMatchObject({ code: "released" }); + }); + + it("is drained at the object's end when it runs: its commit is refused, nothing is reported", async () => { + const gate = Promise.withResolvers(); + const entered = Promise.withResolvers(); + const events: string[] = []; + const { job, host, errors } = await declared(Echo, () => ({ + run: async ({ text }) => text, + finish: async () => { + entered.resolve(); + await gate.promise; + events.push("finish returned"); + }, + })); + await job.start({ text: "a" }, context); + await entered.promise; + const closing = host.close(context).then(() => void events.push("closed")); + await flushed(); + expect(events).toEqual([]); + gate.resolve(); + await closing; + expect(events).toEqual(["finish returned", "closed"]); + expect(errors).toEqual([]); + }); +}); + +describe("running", () => { + it("has one entry per unsettled run, in start order, tagged by phase; a key when given", async () => { + const gates = [Promise.withResolvers(), Promise.withResolvers()]; + const finishing = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ + run: async ({ text }) => { + await gates[Number(text)]!.promise; + return text; + }, + finish: async () => finishing.promise, + })); + const zero = await job.start({ text: "0" }, context, { key: "z" }); + const one = await job.start({ text: "1" }, context); + gates[0]!.resolve(); + await flushed(); + expect(job.running.value).toEqual([ + { id: zero.id, version: 1, key: "z", args: { text: "0" }, phase: "finishing" }, + { id: one.id, version: 1, args: { text: "1" }, phase: "running" }, + ]); + finishing.resolve(); + gates[1]!.resolve(); + await one.outcome(context); + expect(job.running.value).toEqual([]); + }); + + it("replaces only the entry whose state changed, with ops under its entry (cost guard: no walk of other runs)", async () => { + const gate = Promise.withResolvers(); + const contexts = new Map>(); + const Many = defineJob("app.many", { version: 1, args: obj({ n: int() }), result: int(), state: Progress }); + const { job } = await declared(Many, () => ({ + run: async ({ n }, ctx) => { + contexts.set(String(n), ctx); + await gate.promise; + return n; + }, + })); + for (let n = 0; n < 100; n++) await job.start({ n }, context); + await flushed(); + const before = job.running.value; + const frames = framesOf(await job.running.watch(context)); + await contexts.get("42")!.state.change((d) => { + d.output = "x"; + }, context); + await delivered(); + const after = job.running.value; + expect(after).not.toBe(before); + expect(after.filter((entry, index) => entry !== before[index])).toHaveLength(1); + expect(after[42]).toMatchObject({ state: { output: "x" } }); + expect(frames.map(([, ops]) => ops)).toEqual([[["a", [42, "state", "output"], "x"]] satisfies Op[]]); + gate.resolve(); + }); + + it("does no validation, walk or compilation for a state change (cost guard)", async () => { + const { telemetry, context: counting, counts } = await metered(); + const gate = Promise.withResolvers(); + const contexts: JobCtx[] = []; + const { job } = await declared( + Shell, + () => ({ + run: async (_, ctx) => { + contexts.push(ctx); + await gate.promise; + return { exit: 0 }; + }, + }), + { telemetry, context: counting }, + ); + await job.start({ command: "a" }, counting); + await flushed(); + const frames = framesOf(await job.running.watch(context)); + const data = () => Object.entries(counts()).filter(([name]) => name.startsWith("chord.data.")); + const before = data(); + await contexts[0]!.state.change((d) => { + d.output = "x".repeat(10_000); + }, context); + await delivered(); + expect(data()).toEqual(before); + expect(frames).toHaveLength(1); + gate.resolve(); + }); + + it("frames record changes as splices and sets of the list", async () => { + const gate = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ run: gated(gate.promise) })); + const frames = framesOf(await job.running.watch(context)); + const run = await job.start({ text: "a" }, context); + await delivered(); + gate.resolve(); + await run.outcome(context); + await delivered(); + const entry = { id: run.id, version: 1, args: { text: "a" }, phase: "running" }; + expect(frames.map(([, ops]) => ops)).toEqual([ + [["p", [], 0, 0, [entry]]], + [["s", [0], { ...entry, phase: "finishing" }]], + [["p", [], 0, 1, []]], + ]); + }); +}); + +describe("keep-alive", () => { + it("keeps an object without holders while runs run, and ends it after the last one and its linger", async () => { + const gate = Promise.withResolvers(); + let job: JobHandle<{ text: string }, string, undefined> | undefined; + let ended = false; + const { host } = await loaded((h) => + h.serve( + Item, + (_, self) => { + job = self.job(Echo, { run: gated(gate.promise) }); + self.onLive(async (ctx) => { + await untilAborted(ctx); + ended = true; + }); + return { log: self.state(Log) }; + }, + { linger: 20 }, + ), + ); + const item = await host.get(Item, { id: 1 }, context); + await job!.start({ text: "a" }, context); + expect(host.activity.value.jobs).toBe(1); + expect(host.references(Item).value).toHaveLength(1); + item.release(); + await new Promise((resolve) => setTimeout(resolve, 40)); + expect(ended).toBe(false); + expect(host.references(Item).value).toHaveLength(0); + expect(host.activity.value).toMatchObject({ jobs: 1, lingering: 0 }); + gate.resolve(); + await flushed(); + expect(host.activity.value).toMatchObject({ jobs: 0, lingering: 1 }); + await new Promise((resolve) => setTimeout(resolve, 40)); + expect(ended).toBe(true); + expect(host.activity.value).toMatchObject({ jobs: 0, lingering: 0 }); + }); + + it("keeps an object whose last run's finish chains another job: the chained run runs, then the object ends", async () => { + const A = defineJob("app.a", { version: 1, args: obj({ text: str() }), result: str() }); + const events: string[] = []; + let a: JobHandle<{ text: string }, string, undefined> | undefined; + let chained: Run | undefined; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + self.open( + async () => null, + () => void events.push("object ended"), + ); + const b = self.job(Echo, { + run: async ({ text }) => { + await new Promise((resolve) => setTimeout(resolve, 10)); + events.push(`b ran ${text}`); + return text; + }, + }); + a = self.job(A, { + run: async ({ text }) => text, + finish: async (_outcome, tx) => { + chained = tx.start(b, { text: "chained" }); + }, + }); + return { log: self.state(Log) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await a!.start({ text: "x" }, context); + item.release(); + await new Promise((resolve) => setTimeout(resolve, 40)); + expect(await chained!.outcome(context)).toMatchObject({ kind: "done", result: "chained" }); + expect(events).toEqual(["b ran chained", "object ended"]); + expect(host.activity.value.jobs).toBe(0); + }); + + it("keeps an object whose last reference goes while its start's commit publishes (a link closing as it sends)", async () => { + const View = remote("app.view", { + version: 1, + params: {}, + variants: { v: { state: { running: arr(unknown()) } } }, + }); + let job: JobHandle<{ text: string }, string, undefined> | undefined; + let ran = false; + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const facet = defineFacet((h) => + h.serve(View, (_, _principal, self) => { + job = self.job(Echo, { + run: async ({ text }) => { + ran = true; + return text; + }, + }); + // The running list as a state member, so the start's commit sends a state message. + return { variant: "v", members: { running: job.running as never } }; + }), + ); + await server.load([{ name: "@test/app", facet }], context); + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + let closeOnSend = false; + // The server's end of the link closes while it sends: the client's hold, the last reference, goes at once. + const failing: Link = { + send: (message) => { + if (!closeOnSend) return a.send(message); + closeOnSend = false; + a.close(); + }, + onMessage: (listener) => a.onMessage(listener), + close: () => a.close(), + onClose: (listener) => a.onClose(listener), + }; + server.accept(failing, { principal: { kind: "owner" } }, context); + client.connect(b, { principal: { kind: "server" } }, context); + await client.get(View, {}, context); + await transported(); + closeOnSend = true; + const run = await job!.start({ text: "a" }, context); + expect(await run.outcome(context)).toMatchObject({ kind: "done", result: "a" }); + expect(ran).toBe(true); + await client.close(context); + await server.close(context); + }); + + it("drops blocked runs, reported, when the object ends after its last run", async () => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"], now: 0 }); + try { + let job: JobHandle<{ text: string }, string, undefined> | undefined; + const errors: HostError[] = []; + const host = createHost({ onError: (error) => errors.push(error) }); + const facet = defineFacet((h) => + h.serve(Item, (_, self) => { + job = self.job(Echo, { + run: async ({ text }) => text, + finish: async () => { + throw new Error("stuck"); + }, + }); + return { log: self.state(Log) }; + }), + ); + await host.load([{ name: "@test/app", facet }], context); + const item = await host.get(Item, { id: 1 }, context); + const run = await job!.start({ text: "a" }, context); + item.release(); + await vi.advanceTimersByTimeAsync(60_000); + expect(errors.map((error) => error.kind)).toEqual([...Array(5).fill("job_finish_failed"), "job_dropped"]); + expect(await rejection(run.outcome(context))).toMatchObject({ code: "released" }); + expect(host.activity.value.jobs).toBe(0); + } finally { + vi.useRealTimers(); + } + }); + + it("keeps a lingering object when a run starts during its linger", async () => { + const gate = Promise.withResolvers(); + let job: JobHandle<{ text: string }, string, undefined> | undefined; + const { host } = await loaded((h) => + h.serve( + Item, + (_, self) => { + job = self.job(Echo, { run: gated(gate.promise) }); + return { log: self.state(Log) }; + }, + { linger: 20 }, + ), + ); + (await host.get(Item, { id: 1 }, context)).release(); + expect(host.activity.value.lingering).toBe(1); + const run = await job!.start({ text: "a" }, context); + expect(host.activity.value).toMatchObject({ jobs: 1, lingering: 0 }); + await new Promise((resolve) => setTimeout(resolve, 40)); + gate.resolve(); + expect((await run.outcome(context)).kind).toBe("done"); + }); +}); + +describe("object end", () => { + type Revocation = { session: string }; + const Jobs = remote("app.jobs", { version: 1, params: {}, variants: { v: { state: {} } } }); + + it("revocation (R11): runs' signals fire with the end's error, the drain waits for them, no finish", async () => { + let job: JobHandle<{ text: string }, string, undefined> | undefined; + const finished: string[] = []; + let reason: unknown; + const server = createHost({ + principal: { kind: "server" }, + revokedBy: (p) => (p.kind === "guest" ? [{ session: p.session }] : []), + onError: () => {}, + }); + const facet = defineFacet((h) => + h.serve(Jobs, (_, _principal, self) => { + job = self.job(Echo, { + run: async ({ text }, ctx) => { + await untilAborted(ctx); + reason = ctx.abortSignal?.reason; + await new Promise((resolve) => setTimeout(resolve, 10)); + finished.push("run stopped"); + return text; + }, + finish: async () => void finished.push("finish"), + }); + return { variant: "v", members: {} }; + }), + ); + await server.load([{ name: "@test/app", facet }], context); + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "guest", session: "S1" } }, context); + client.connect(b, { principal: { kind: "server" } }, context); + await client.get(Jobs, {}, context); + const run = await job!.start({ text: "a" }, context); + await flushed(); + server.revoke({ session: "S1" }, context); + await flushed(); + expect(reason).toMatchObject({ code: "forbidden", reason: "revoked" }); + expect(await rejection(run.outcome(context))).toMatchObject({ code: "forbidden", reason: "revoked" }); + await new Promise((resolve) => setTimeout(resolve, 30)); + await transported(); + expect(finished).toEqual(["run stopped"]); + expect(server.activity.value.jobs).toBe(0); + await client.close(context); + await server.close(context); + }); + + it("drains a run that ends its own object before its first await: its natives stay open until it stops", async () => { + let job: JobHandle<{ text: string }, string, undefined> | undefined; + const gate = Promise.withResolvers(); + const events: string[] = []; + const server = createHost({ + principal: { kind: "server" }, + revokedBy: (p) => (p.kind === "guest" ? [{ session: p.session }] : []), + onError: () => {}, + }); + const facet = defineFacet((h) => + h.serve(Jobs, (_, _principal, self) => { + self.open( + async () => null, + () => void events.push("native closed"), + ); + job = self.job(Echo, { + run: async ({ text }) => { + server.revoke({ session: "S1" }, context); + await gate.promise; + events.push("run stopped"); + return text; + }, + }); + return { variant: "v", members: {} }; + }), + ); + await server.load([{ name: "@test/app", facet }], context); + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "guest", session: "S1" } }, context); + client.connect(b, { principal: { kind: "server" } }, context); + await client.get(Jobs, {}, context); + const run = rejection((await job!.start({ text: "a" }, context)).outcome(context)); + await flushed(); + expect(events).toEqual([]); + gate.resolve(); + await flushed(); + expect(events).toEqual(["run stopped", "native closed"]); + expect(await run).toMatchObject({ code: "forbidden", reason: "revoked" }); + await client.close(context); + await server.close(context); + }); + + it("host.close: signals fire, close waits for the run (drain), no finish, waiters rejected released", async () => { + const finished: string[] = []; + const { job, host } = await declared(Echo, () => ({ + run: async ({ text }, ctx) => { + await untilAborted(ctx); + await new Promise((resolve) => setTimeout(resolve, 10)); + finished.push("run stopped"); + return text; + }, + finish: async () => void finished.push("finish"), + })); + const run = await job.start({ text: "a" }, context); + await flushed(); + const waiting = rejection(run.outcome(context)); + await host.close(context); + expect(finished).toEqual(["run stopped"]); + expect(await waiting).toMatchObject({ code: "released" }); + }); +}); + +describe("GC", () => { + it("finished and dropped runs leave nothing on their object", async () => { + const refs: WeakRef[] = []; + const { job } = await declared(Echo, () => ({ + run: async (args, ctx) => { + refs.push(new WeakRef(args), new WeakRef(ctx)); + return args.text; + }, + })); + await (await job.start({ text: "a" }, context)).outcome(context); + await flushed(); + await expectCollected(...refs); + }); + + describe("of dropped runs", () => { + fakeTimers(); + + it("a dropped blocked run leaves nothing on its object", async () => { + const refs: WeakRef[] = []; + const { job, errors } = await declared(Echo, () => ({ + run: async (args, ctx) => { + refs.push(new WeakRef(args), new WeakRef(ctx)); + return args.text; + }, + finish: async () => { + throw new Error("stuck"); + }, + })); + const run = await job.start({ text: "a" }, context); + await vi.advanceTimersByTimeAsync(60_000); + expect(job.running.value.map((entry) => entry.phase)).toEqual(["blocked"]); + const dropped = run.outcome(context).then( + () => new Error("expected a rejection"), + (error: Error) => error, + ); + job.abort(run.id); + await vi.advanceTimersByTimeAsync(0); + expect(await dropped).toMatchObject({ code: "aborted" }); + expect(job.running.value).toEqual([]); + // An error keeps its stack frames' receivers (the dropping transaction) until its stack is read (V8). + void (await dropped).stack; + for (const error of errors) if (error.kind === "job_finish_failed") void (error.error as Error).stack; + await expectCollected(...refs); + }); + }); + + it("keeps nothing of a cancelled wait while its run still runs", async () => { + const gate = Promise.withResolvers(); + const { job } = await declared(Echo, () => ({ run: gated(gate.promise) })); + const run = await job.start({ text: "a" }, context); + /** One wait, cancelled; returns its signal, weakly. */ + const cancelled = async (): Promise> => { + const controller = new AbortController(); + const waiting = rejection(run.outcome(withAbortSignal(controller.signal, context))); + controller.abort("gave up"); + expect(await waiting).toBe("gave up"); + return new WeakRef(controller.signal); + }; + const refs: WeakRef[] = []; + for (let i = 0; i < 5; i++) refs.push(await cancelled()); + await expectCollected(...refs); + gate.resolve(); + expect((await run.outcome(context)).kind).toBe("done"); + }); + + it("keeps no ended run's args or state in a job handle kept after its object ended", async () => { + const refs: WeakRef[] = []; + const { job, host } = await declared(Shell, () => ({ + run: async (args, ctx) => { + refs.push(new WeakRef(args), new WeakRef(ctx.state.value)); + await untilAborted(ctx); + return { exit: 1 }; + }, + })); + await job.start({ command: "a" }, context); + await flushed(); + await host.close(context); + await expectCollected(...refs); + expect(() => job.running.value).toThrow(ChordError); + }); + + it("an ended object releases its runs, their documents and waiters", async () => { + const refs: WeakRef[] = []; + let job: JobHandle<{ command: string }, { exit: number }, { output: string }> | undefined; + const { host } = await loaded((h) => + h.serve(Item, (_, self) => { + job = self.job(Shell, { + run: async (args, ctx) => { + refs.push(new WeakRef(args), new WeakRef(ctx), new WeakRef(ctx.state.value)); + await untilAborted(ctx); + return { exit: 1 }; + }, + }); + return { log: self.state(Log) }; + }), + ); + const item = await host.get(Item, { id: 1 }, context); + await job!.start({ command: "a" }, context); + await flushed(); + refs.push(new WeakRef(job!)); + job = undefined; + await host.close(context); + void item; + await expectCollected(...refs); + }); +}); + +// The type of `outcome` follows the definition. +expectTypeOf["args"]>().toEqualTypeOf<{ + command: string; +}>(); +void (null as unknown as Context); diff --git a/packages/chord/test/next/jobs/records.test.ts b/packages/chord/test/next/jobs/records.test.ts new file mode 100644 index 000000000..736791615 --- /dev/null +++ b/packages/chord/test/next/jobs/records.test.ts @@ -0,0 +1,205 @@ +// Tests of a job's records on its object (slice 6a §1.2, §15 rule 5): runs across attempts, cut off by an attempt's +// seal, finished on the next live attempt, and the keep-alive of runs waiting for one. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import { untilAborted } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { defineJob, type JobHandle, type Outcome, type RunningEntry } from "../../../src/next/jobs/index.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import { delivered, flushed, instancesOf, memoryStore, rejection, restartable, storeRestartable } from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +const Progress = defineState("app.progress", { + schema: obj({ output: str() }), + version: 1, + initial: () => ({ output: "" }), +}); + +const Work = defineJob("app.work", { version: 1, args: obj({ text: str() }), result: str(), state: Progress }); + +type WorkHandle = JobHandle<{ text: string }, string, { output: string }>; + +/** What a test's attempts do with a run: `run` waits for its abort (or `release`), `finish` records by attempt. */ +async function working( + finish: (attempt: number, outcome: Outcome<{ text: string }, string, { output: string }>) => void | Promise, +) { + const jobs: WorkHandle[] = []; + const object = await restartable( + objects, + (attempt, self) => { + jobs.push( + self.job(Work, { + run: async ({ text }, ctx) => { + await ctx.state.change((draft) => { + draft.output = `${text} started`; + }, ctx); + await untilAborted(ctx); + return text; + }, + finish: (outcome) => finish(attempt, outcome), + }), + ); + }, + { closeTimeout: 50 }, + ); + return { ...object, jobs }; +} + +describe("runs across attempts", () => { + it("a run cut off by its attempt's seal ends aborted (restart) and is finished on the next live attempt", async () => { + const finished: string[] = []; + const { jobs, fail, retry } = await working((attempt, outcome) => { + finished.push(`${attempt}: ${outcome.kind} ${JSON.stringify(outcome)}`); + }); + const run = await jobs[0]!.start({ text: "a" }, context); + await flushed(); + fail(); + await flushed(); + // No live attempt: the run waits, finishing. + expect(finished).toEqual([]); + expect((jobs[0]!.running.value as readonly RunningEntry[]).map(({ phase }) => phase)).toEqual([ + "finishing", + ]); + await retry(); + const outcome = await run.outcome(context); + expect(outcome).toEqual({ + id: run.id, + args: { text: "a" }, + state: { output: "a started" }, + kind: "failed", + error: { kind: "aborted", reason: "restart" }, + }); + expect(finished).toEqual([`2: failed ${JSON.stringify(outcome)}`]); + expect(jobs[1]!.running.value).toEqual([]); + }); + + it("refuses the cut run's changes made while it stops; its settling records nothing more", async () => { + const late = Promise.withResolvers(); + const jobs: WorkHandle[] = []; + const { restart } = await restartable(objects, (_, self) => { + const job = self.job(Work, { + run: async (_args, ctx) => { + await untilAborted(ctx); + // Accepted after an abort (§15), refused once the attempt's seal cut the run off. + await ctx.state + .change((draft) => { + draft.output = "late"; + }, context) + .then(late.resolve, late.resolve); + return "done"; + }, + }); + jobs.push(job); + }); + const run = await jobs[0]!.start({ text: "a" }, context); + await flushed(); + await restart(); + const outcome = await run.outcome(context); + expect(await late.promise).toMatchObject({ code: "released" }); + expect(outcome).toMatchObject({ kind: "failed", error: { reason: "restart" }, state: { output: "" } }); + await flushed(); + expect(jobs[1]!.running.value).toEqual([]); + }); + + it("a finish sealed mid-way is not counted as a failure and runs again on the next live attempt", async () => { + const gate = Promise.withResolvers(); + const attempts: number[] = []; + const { jobs, restart, errors } = await working(async (attempt) => { + attempts.push(attempt); + if (attempt === 1) await gate.promise; + }); + const run = await jobs[0]!.start({ text: "a" }, context); + await flushed(); + jobs[0]!.abort(run.id); + await flushed(); + expect(attempts).toEqual([1]); + await restart(); + gate.resolve(); + // `run` returns its result after the abort: a success. + expect(await run.outcome(context)).toMatchObject({ kind: "done", result: "a" }); + expect(attempts).toEqual([1, 2]); + // No job_finish_failed: the stuck finish only outlived the drain. + expect(errors.map(({ kind }) => kind)).toEqual(["object_failed", "drain_timeout"]); + }); +}); + +describe("keep-alive", () => { + it("a run waiting for a live attempt keeps a failed object no longer: it ends with its last reference", async () => { + const { jobs, fail, reference } = await working(() => {}); + const run = await jobs[0]!.start({ text: "a" }, context); + await flushed(); + fail(); + reference.release(); + await delivered(); + expect(await rejection(run.outcome(context))).toMatchObject({ code: "released" }); + }); +}); + +describe("the object's end during an admitted commit", () => { + const Saved = defineState("app.saved", { + schema: obj({ n: int() }), + version: 1, + initial: () => ({ n: 0 }), + persist: true, + checkpointWhen: () => false, + }); + + it("a run started in a commit held in the SQL while the host closes ends with the object; no run is left counted", async () => { + const store = await memoryStore(); + const declared: { saved: SingletonHandle<{ n: number }>; job: WorkHandle }[] = []; + const { host, attempts } = await storeRestartable( + objects, + (_, self) => { + const job = self.job(Work, { + run: async (_args, ctx) => { + await untilAborted(ctx); + return ""; + }, + finish: () => {}, + }); + declared.push({ saved: self.state(Saved), job }); + }, + { store }, + ); + const { saved, job } = declared[0]!; + // The next commit's step waits inside the backend transaction, after its admission. + const gate = Promise.withResolvers(); + const inStep = Promise.withResolvers(); + const commit = store.commit.bind(store); + store.commit = (writes, fence, ctx) => { + const held: StagedWrite = { + order: "write", + apply: async () => { + inStep.resolve(); + await gate.promise; + return { kind: "dropIndex", table: "x", name: "y" }; + }, + }; + return commit([...writes, held], fence, ctx); + }; + const started = attempts[0]!.tx(async (tx) => { + (await tx.draft(saved)).n = 1; + return tx.start(job, { text: "a" }); + }, context); + await inStep.promise; + let closed = false; + const closing = host.close(context).then(() => { + closed = true; + }); + await delivered(); + // The object's teardown waits for the admitted step, so the run's record is adopted before the end. + expect(closed).toBe(false); + gate.resolve(); + const run = await started; + await closing; + store.commit = commit; + expect(await rejection(run.outcome(context))).toMatchObject({ code: "released" }); + expect(host.activity.value.jobs).toBe(0); + expect(store.lastSeq).toBe(1); + await store.close(context); + }); +}); diff --git a/packages/chord/test/next/jobs/run.test.ts b/packages/chord/test/next/jobs/run.test.ts new file mode 100644 index 000000000..0899a33f7 --- /dev/null +++ b/packages/chord/test/next/jobs/run.test.ts @@ -0,0 +1,49 @@ +// Tests of one run's record (§15): its `running` entry per phase, and its handle's wait for the outcome. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { ChordError } from "../../../src/next/data/index.ts"; +import { JobRun } from "../../../src/next/jobs/run.ts"; +import { rejection } from "../support.ts"; + +describe("a run", () => { + it("has an entry per phase: args and state while running or finishing, neither when blocked", () => { + const run = new JobRun("r1", 2, "k", { command: "ls" }); + expect(run.entryFor({ output: "" })).toEqual({ + id: "r1", + version: 2, + key: "k", + args: { command: "ls" }, + state: { output: "" }, + phase: "running", + }); + run.phase = "finishing"; + expect(new JobRun("r2", 1, undefined, { a: 1 }).entryFor(undefined)).toEqual({ + id: "r2", + version: 1, + args: { a: 1 }, + phase: "running", + }); + run.phase = "blocked"; + expect(run.entryFor({ output: "x" })).toEqual({ + id: "r1", + version: 2, + phase: "blocked", + reason: "finish_failed", + }); + }); + + it("gives every waiter the outcome, late ones too; a waiter's context ends only its wait", async () => { + const run = new JobRun("r1", 1, undefined, {}); + const { context: waiting, cancel } = withCancel(context); + const cancelled = rejection(run.handle.outcome(waiting)); + cancel(new Error("left")); + expect(await cancelled).toMatchObject({ message: "left" }); + const outcome = { id: "r1", args: {}, state: undefined, kind: "done", result: 1 } as const; + run.resolve(outcome); + expect(await run.handle.outcome(context)).toBe(outcome); + const rejected = new JobRun("r2", 1, undefined, {}); + rejected.reject(new ChordError({ code: "aborted", message: "dropped" })); + expect(await rejection(rejected.handle.outcome(context))).toMatchObject({ code: "aborted" }); + }); +}); diff --git a/packages/chord/test/next/jobs/running.test.ts b/packages/chord/test/next/jobs/running.test.ts new file mode 100644 index 000000000..6f153ec1f --- /dev/null +++ b/packages/chord/test/next/jobs/running.test.ts @@ -0,0 +1,124 @@ +// Tests of `running`'s list (§15): each change copies the list, keeps the other entries identical and gives the ops +// of exactly that change (a state change re-pathed under its entry); one frame per batch; derives over it are marked. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { applyImmutable, type Op } from "../../../src/delta/index.ts"; +import type { JsonValue } from "../../../src/json.ts"; +import { Running } from "../../../src/next/jobs/running.ts"; +import { Batch, Derive } from "../../../src/next/state/internal.ts"; +import { delivered, framesOf } from "../support.ts"; + +function running() { + const batch = new Batch(() => {}, undefined); + return { batch, list: new Running(batch, "app.store job app.echo") }; +} + +describe("running's list", () => { + it("copies the list per change, keeps other entries identical, and frames apply to the previous value", async () => { + const { batch, list } = running(); + const a = { id: "a", phase: "running", state: { output: "" } }; + const b = { id: "b", phase: "running", state: { output: "" } }; + const watch = await list.watch(context); + const frames = framesOf(watch); + let previous = list.value; + batch.commit(() => { + list.add(a); + list.add(b); + }, context); + await delivered(); + const added = list.value as JsonValue[]; + expect(added).toEqual([a, b]); + expect(added).not.toBe(previous); + const b2 = { ...b, state: { output: "x" } }; + batch.commit(() => list.stateChanged(b, b2, [["a", ["output"], "x"]]), context); + await delivered(); + const a2 = { ...a, phase: "finishing" }; + batch.commit(() => list.replace(a, a2), context); + await delivered(); + batch.commit(() => list.remove(a2), context); + await delivered(); + expect(list.value).toEqual([b2]); + expect((list.value as JsonValue[])[0]).toBe(b2); + expect(frames.map(([, ops]) => ops)).toEqual([ + [ + ["p", [], 0, 0, [a]], + ["p", [], 1, 0, [b]], + ], + [["a", [1, "state", "output"], "x"]], + [["s", [0], a2]], + [["p", [], 0, 1, []]], + ] satisfies Op[][]); + // Every frame turns the previous value into the next. + previous = []; + for (const [value, ops] of frames) { + expect(applyImmutable(previous, ops)).toEqual(value); + previous = value; + } + }); + + it("re-paths every kind of state op under its entry; a replacement becomes a set", async () => { + const { batch, list } = running(); + const entry = { id: "a", state: { list: [1], text: "", n: 0 } }; + batch.commit(() => list.add(entry), context); + const frames = framesOf(await list.watch(context)); + const after = { id: "a", state: { list: [2], text: "t", n: 1 } }; + const ops: Op[] = [ + ["r", { list: [], text: "", n: 0 }], + ["s", ["n"], 1], + ["p", ["list"], 0, 0, [2]], + ["a", ["text"], "t"], + ["d", ["gone"]], + ["t", ["text"], 1], + ["m", ["list"], [0]], + ]; + batch.commit(() => list.stateChanged(entry, after, ops), context); + await delivered(); + expect(frames[0]![1]).toEqual([ + ["s", [0, "state"], { list: [], text: "", n: 0 }], + ["s", [0, "state", "n"], 1], + ["p", [0, "state", "list"], 0, 0, [2]], + ["a", [0, "state", "text"], "t"], + ["d", [0, "state", "gone"]], + ["t", [0, "state", "text"], 1], + ["m", [0, "state", "list"], [0]], + ]); + }); + + it("builds no op for a state change while unwatched (cost guard: the ops are not even walked)", () => { + const { batch, list } = running(); + const entry = { id: "a", state: { output: "" } }; + batch.commit(() => list.add(entry), context); + const unwalked = { + [Symbol.iterator]: () => { + throw new Error("walked"); + }, + } as unknown as Op[]; + const after = { id: "a", state: { output: "x" } }; + batch.commit(() => list.stateChanged(entry, after, unwalked), context); + expect(list.value).toEqual([after]); + }); + + it("keeps nothing once ended: reads throw released", async () => { + const { batch, list } = running(); + batch.commit(() => list.add({ id: "a" }), context); + const watch = await list.watch(context); + list.end(); + expect(await watch.closed).toEqual({ kind: "ended" }); + expect(() => list.value).toThrow("app.store job app.echo: ended"); + }); + + it("records no ops while unwatched, and marks derives over it at every change", () => { + const { batch, list } = running(); + const count = new Derive([list], ([value]) => (value as JsonValue[]).length, batch); + count.attach( + (source) => source as Running, + () => {}, + ); + expect(count.value).toBe(0); + batch.commit(() => list.add({ id: "a" }), context); + expect(count.value).toBe(1); + batch.commit(() => list.add({ id: "b" }), context); + expect(count.value).toBe(2); + }); +}); diff --git a/packages/chord/test/next/link/channel.test.ts b/packages/chord/test/next/link/channel.test.ts new file mode 100644 index 000000000..f5abbe8eb --- /dev/null +++ b/packages/chord/test/next/link/channel.test.ts @@ -0,0 +1,342 @@ +// Tests of a host's end of a link: the handshake, checks, the heartbeat, trusted links, and in-order handling while a +// message waits for an asynchronous inflate. + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, remote } from "../../../src/next/contracts/index.ts"; +import { PROTOCOL_VERSION } from "../../../src/next/data/internal.ts"; +import { createHost, type HostError } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { binaryCodec, Channel, type Message, textCodec } from "../../../src/next/link/internal.ts"; +import { + bindNext, + expectCollected, + flushed, + forEachCodec, + linked, + metered, + Notes, + RAW_PEER, + RawPeer, + rawLinked, + rawProvider, + testCodec, + transported, +} from "../support.ts"; + +const Thing = remote("app.thing", { version: 1, params: {}, variants: { only: { state: { n: int() } } } }); + +const hold = { + kind: "hold", + hold: 0, + type: "app.thing", + version: 1, + fingerprint: "0000000000000000", + params: "{}", +} as const; + +const problems = (errors: readonly HostError[]) => + errors.flatMap((error) => (error.kind === "protocol_error" ? [(error.error as Error).message] : [])); + +forEachCodec(() => + describe("the handshake", () => { + it("sends hello with the protocol version and the host's incarnation first", async () => { + const { peer } = await rawLinked(() => {}); + const hello = await peer.next("hello"); + expect(hello.protocol).toBe(PROTOCOL_VERSION); + expect(hello.incarnation).toMatch(/^[0-9a-f]{8}-[0-9a-f]{4}-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/); + }); + + it("closes the link on traffic before hello", async () => { + const { peer, errors } = await rawLinked(() => {}, { hello: false }); + peer.send(hold); + await transported(); + expect(peer.closed).toBe(true); + expect(problems(errors)).toEqual(["handshake: hold before hello"]); + }); + + it("closes the link on a second hello", async () => { + const { peer, errors } = await rawLinked(() => {}); + peer.send({ kind: "hello", protocol: PROTOCOL_VERSION, incarnation: RAW_PEER, calls: 256 }); + await transported(); + expect(peer.closed).toBe(true); + expect(problems(errors)).toEqual(["handshake: a second hello"]); + }); + + it("closes the link on another protocol version", async () => { + const { peer, errors } = await rawLinked(() => {}, { hello: false }); + peer.send({ kind: "hello", protocol: PROTOCOL_VERSION + 1, incarnation: RAW_PEER, calls: 256 }); + await transported(); + expect(peer.closed).toBe(true); + expect(problems(errors)).toEqual([ + `handshake: protocol ${PROTOCOL_VERSION + 1}, this host speaks ${PROTOCOL_VERSION}`, + ]); + }); + + it("closes the link on a message failing its checks", async () => { + const { peer, errors } = await rawLinked(() => {}); + peer.sendBytes(testCodec().encode({ kind: "error", call: 0, error: { kind: "nope" } } as never)); + await transported(); + expect(peer.closed).toBe(true); + expect(problems(errors)).toEqual(["message: fails the message schema"]); + }); + + it("queues a consumer's messages until the peer's hello; a get over a link that never greets waits for its deadline", async () => { + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const peer = new RawPeer(b, { hello: false }); + const got = client.get(Thing, {}, context, { readyTimeout: 20 }); + await flushed(); + expect(peer.kinds()).toEqual(["hello"]); + await expect(got).rejects.toThrow("app.thing: did not become live within 20 ms"); + peer.send({ kind: "hello", protocol: PROTOCOL_VERSION, incarnation: RAW_PEER, calls: 256 }); + await transported(); + expect(peer.kinds()).toEqual(["hello", "hold", "release"]); + }); + }), +); + +describe("a codec mismatch", () => { + it("fails the first hello as a protocol error naming the expected codec", async () => { + const errors: HostError[] = []; + const server = createHost({ onError: (error) => errors.push(error) }); + await server.load([], context); + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: binaryCodec }, context); + client.connect(b, { principal: { kind: "server" }, codec: textCodec }, context); + await transported(); + expect(problems(errors)).toEqual(["handshake: the first message is no binary message"]); + }); +}); + +describe("the heartbeat", () => { + beforeEach(() => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "setInterval", "clearInterval", "Date"], now: 0 }); + }); + afterEach(() => { + vi.useRealTimers(); + }); + + /** Advances fake time by `ms` in heartbeat ticks, letting links deliver between them. */ + async function elapse(ms: number): Promise { + for (let step = 0; step < ms; step += 1500) { + await vi.advanceTimersByTimeAsync(Math.min(1500, ms - step)); + await transported(); + } + } + + it("closes a link whose peer never sends hello after 30 s (race 30)", async () => { + const { peer } = await rawLinked(() => {}, { hello: false }); + await elapse(29_000); + expect(peer.closed).toBe(false); + await elapse(1_500); + expect(peer.closed).toBe(true); + }); + + it("pings a silent link after 15 s and closes it after 30 s without an answer: references go loading", async () => { + const { client, peer } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + await bindNext(peer); + const notes = await got; + await elapse(13_500); + expect(peer.kinds()).not.toContain("ping"); + await elapse(3_000); + expect(peer.kinds().filter((kind) => kind === "ping")).toEqual(["ping"]); + await elapse(15_000); + expect(peer.closed).toBe(true); + expect(notes.state.value).toEqual({ kind: "loading" }); + }); + + it("sends no pings on a busy link", async () => { + const { peer } = await rawLinked(() => {}); + for (let second = 0; second < 60; second += 5) { + peer.send({ kind: "release", hold: 99 }); + await elapse(5_000); + } + expect(peer.kinds()).toEqual(["hello"]); + expect(peer.closed).toBe(false); + }); + + it("answers each ping with a pong at once; two sides pinging at once both answer", async () => { + const { client, server } = await linked(() => {}); + await elapse(60_000); + // Both sides pinged at 15 s and were answered: neither closed the link. + const notes = client.hold(Notes, { session: "S1" }, context); + await transported(); + expect(notes.state.value).toMatchObject({ kind: "failed" }); + expect(server).toBeDefined(); + const { peer } = await rawLinked(() => {}); + peer.send({ kind: "ping" }); + expect(await peer.next("pong")).toEqual({ kind: "pong" }); + }); + + it("runs one timer per host, only while it has links: none per message", async () => { + const host = createHost({ onError: () => {} }); + await host.load([], context); + expect(vi.getTimerCount()).toBe(0); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const [c, d] = memoryLinkPair(); + host.accept(c, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peers = [new RawPeer(b), new RawPeer(d)]; + for (let round = 0; round < 50; round++) peers[0]!.send({ kind: "release", hold: round }); + await transported(); + expect(vi.getTimerCount()).toBe(1); + for (const peer of peers) peer.close(); + await transported(); + expect(vi.getTimerCount()).toBe(0); + }); + + it("takes its numbers from the host's options", async () => { + const host = createHost({ onError: () => {}, heartbeat: { ping: 1_000, close: 2_000 } }); + await host.load([], context); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + await vi.advanceTimersByTimeAsync(1_100); + await transported(); + expect(peer.kinds()).toEqual(["hello", "ping"]); + await vi.advanceTimersByTimeAsync(1_000); + await transported(); + expect(peer.closed).toBe(true); + }); +}); + +describe("trusted links", () => { + it("skip the message schema check: no chord.link.checks, the codec and the owners' checks still run", async () => { + const run = async (trusted: boolean) => { + const { context: counting, counts } = await metered(); + const host = createHost({ onError: () => {} }); + await host.load([], context); + const [a, b] = memoryLinkPair(); + host.accept(a, { principal: { kind: "owner" }, trusted, codec: testCodec() }, counting); + const peer = new RawPeer(b); + peer.send({ kind: "release", hold: 1 }); + peer.sendBytes(new Uint8Array([0xff])); + await transported(); + return { checks: counts()["chord.link.checks"] ?? 0, closed: peer.closed }; + }; + expect(await run(false)).toEqual({ checks: 2, closed: true }); + expect(await run(true)).toEqual({ checks: 0, closed: true }); + }); +}); + +describe("in-order handling", () => { + /** A channel whose handler handles `release` of hold 0 asynchronously (until `finish`), every other message at once. */ + async function holding() { + const [a, b] = memoryLinkPair(); + const handled: string[] = []; + const reports: unknown[] = []; + let finish: { resolve(): void; reject(error: Error): void } | undefined; + let closed = false; + const handler = { + opened: () => {}, + closed: () => { + closed = true; + }, + message: (message: Message): void | Promise => { + const name = message.kind === "release" ? `release ${message.hold}` : message.kind; + if (message.kind !== "release" || message.hold !== 0) { + handled.push(name); + return; + } + const { promise, resolve, reject } = Promise.withResolvers(); + finish = { resolve, reject }; + return promise.then(() => void handled.push(name)); + }, + }; + const options = { + codec: testCodec(), + incarnation: RAW_PEER, + calls: 1, + trusted: true, + report: (e: unknown) => reports.push(e), + }; + new Channel(a, handler, options, context).start(); + const peer = new RawPeer(b); + await peer.next("hello"); + return { peer, handled, reports, finish: () => finish!, isClosed: () => closed }; + } + + forEachCodec(() => { + it("holds every later message, a ping included, until the asynchronous one was handled, then handles them in order", async () => { + const { peer, handled, finish } = await holding(); + peer.send({ kind: "release", hold: 0 }); + peer.send({ kind: "release", hold: 1 }); + peer.send({ kind: "ping" }); + peer.send({ kind: "release", hold: 2 }); + await transported(); + expect(handled).toEqual([]); + expect(peer.kinds()).toEqual([]); + finish().resolve(); + await transported(); + expect(handled).toEqual(["release 0", "release 1", "release 2"]); + expect(peer.kinds()).toEqual(["pong"]); + }); + + it("keeps the rest behind a held message that holds again, in order, a long queue included", async () => { + const { peer, handled, finish } = await holding(); + const release = (hold: number) => testCodec().encode({ kind: "release", hold }); + peer.sendBytes(release(0)); + peer.sendBytes(release(1)); + peer.sendBytes(release(0)); + // Beyond what a spread into function arguments takes: the queue is read by index. + const many = release(2); + for (let i = 0; i < 70_000; i++) peer.sendBytes(many); + await transported(); + expect(handled).toEqual([]); + finish().resolve(); + await transported(); + expect(handled).toEqual(["release 0", "release 1"]); + finish().resolve(); + await transported(); + expect(handled.length).toBe(70_003); + expect(handled.slice(0, 4)).toEqual(["release 0", "release 1", "release 0", "release 2"]); + }); + + it("drops the held message's successors when the link closes meanwhile (race 13)", async () => { + const { peer, handled, finish, isClosed } = await holding(); + peer.send({ kind: "release", hold: 0 }); + peer.send({ kind: "release", hold: 1 }); + await transported(); + peer.close(); + await transported(); + expect(isClosed()).toBe(true); + finish().resolve(); + await transported(); + expect(handled).toEqual(["release 0"]); + }); + + it("keeps no held message once the link closed (GC)", async () => { + const { peer, finish, refs } = await (async () => { + const { peer, finish } = await holding(); + peer.send({ kind: "release", hold: 0 }); + const held = testCodec().encode({ kind: "release", hold: 1 }); + peer.sendBytes(held); + await transported(); + peer.close(); + await transported(); + return { peer, finish, refs: [new WeakRef(held)] }; + })(); + await expectCollected(...refs); + finish().resolve(); + expect(peer.closed).toBe(true); + }); + + it("treats a rejected asynchronous handling as a protocol error: reported, the link closes", async () => { + const { peer, handled, finish, reports } = await holding(); + peer.send({ kind: "release", hold: 0 }); + peer.send({ kind: "release", hold: 1 }); + await transported(); + finish().reject(new Error("corrupt")); + await transported(); + expect(peer.closed).toBe(true); + expect(reports).toEqual([new Error("corrupt")]); + expect(handled).toEqual([]); + }); + }); +}); diff --git a/packages/chord/test/next/link/errors.test.ts b/packages/chord/test/next/link/errors.test.ts new file mode 100644 index 000000000..2303a00b9 --- /dev/null +++ b/packages/chord/test/next/link/errors.test.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from "vitest"; +import { ChordError, PluginError } from "../../../src/next/data/index.ts"; +import { errorFromWire, wireError } from "../../../src/next/link/internal.ts"; + +describe("errors on the wire", () => { + it("carry a ChordError as its JSON, without stack or cause", () => { + const error = new ChordError({ code: "invalid", reason: "args", message: "bad" }, { cause: new Error("x") }); + const wire = wireError(error); + expect(wire).toEqual({ kind: "chord", error: { code: "invalid", reason: "args", message: "bad" } }); + const back = errorFromWire(JSON.parse(JSON.stringify(wire))); + expect(back).toBeInstanceOf(ChordError); + expect(back).toMatchObject({ code: "invalid", reason: "args", message: "bad" }); + }); + + it("carry a PluginError's code, message and details only; the consumer rethrows a PluginError", () => { + const error = new PluginError("quota", "too many notes", { limit: 3 }); + const wire = wireError(error); + expect(wire).toEqual({ kind: "plugin", code: "quota", message: "too many notes", details: { limit: 3 } }); + const back = errorFromWire(wire); + expect(back).toBeInstanceOf(PluginError); + expect(back).toMatchObject({ code: "quota", message: "too many notes", details: { limit: 3 } }); + expect(wireError(new PluginError("c", "m"))).toEqual({ kind: "plugin", code: "c", message: "m" }); + }); + + it("carry anything else as the plugin code internal with its message", () => { + expect(wireError(new TypeError("boom"))).toEqual({ kind: "plugin", code: "internal", message: "boom" }); + expect(wireError("plain")).toEqual({ kind: "plugin", code: "internal", message: "plain" }); + }); + + it("throw on a malformed ChordError envelope (a protocol error)", () => { + expect(() => errorFromWire({ kind: "chord", error: { code: "nope", message: "m" } })).toThrow( + "Invalid ChordError JSON", + ); + }); +}); diff --git a/packages/chord/test/next/link/fixtures/ipc-echo.ts b/packages/chord/test/next/link/fixtures/ipc-echo.ts new file mode 100644 index 000000000..6764f8992 --- /dev/null +++ b/packages/chord/test/next/link/fixtures/ipc-echo.ts @@ -0,0 +1,10 @@ +// A forked child for the IPC link test: echoes every message of its parent link, and exits when the link closes. +import process from "node:process"; +import { ipcLink } from "../../../../src/next/link/index.ts"; + +const link = ipcLink(process as unknown as Parameters[0]); +link.onMessage((message) => { + if (message[0] === 0xff) process.exit(0); + link.send(message); +}); +link.onClose(() => process.exit(0)); diff --git a/packages/chord/test/next/link/ipc.test.ts b/packages/chord/test/next/link/ipc.test.ts new file mode 100644 index 000000000..da9e87d21 --- /dev/null +++ b/packages/chord/test/next/link/ipc.test.ts @@ -0,0 +1,143 @@ +import { fork } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import { type IpcPort, ipcLink } from "../../../src/next/link/index.ts"; +import { flushed } from "../support.ts"; + +/** A port playing a Node IPC channel by hand: what was sent, its listeners, and what the test makes happen. */ +class FakePort implements IpcPort { + connected = true; + readonly sent: Uint8Array[] = []; + /** What `send` returns, or throws when an error. */ + next: boolean | Error = true; + callbacks: ((error: Error | null) => void)[] = []; + disconnects = 0; + readonly #message = new Set<(message: unknown) => void>(); + readonly #disconnect = new Set<() => void>(); + + send(message: Uint8Array, callback: (error: Error | null) => void): boolean { + if (this.next instanceof Error) throw this.next; + this.sent.push(message); + this.callbacks.push(callback); + return this.next; + } + + on(event: "message" | "disconnect", listener: (message: unknown) => void): this { + if (event === "message") this.#message.add(listener); + else this.#disconnect.add(listener as () => void); + return this; + } + + removeListener(event: "message" | "disconnect", listener: (message: unknown) => void): this { + if (event === "message") this.#message.delete(listener); + else this.#disconnect.delete(listener as () => void); + return this; + } + + disconnect(): void { + this.disconnects++; + this.connected = false; + for (const listener of [...this.#disconnect]) listener(); + } + + receive(message: unknown): void { + for (const listener of [...this.#message]) listener(message); + } + + get listeners(): number { + return this.#message.size + this.#disconnect.size; + } +} + +describe("IPC links over a fake port", () => { + it("deliver Uint8Array messages in order, queued until the listener is set; other messages are not the link's", () => { + const port = new FakePort(); + const link = ipcLink(port); + port.receive(new Uint8Array([1])); + port.receive({ start: "the app's own" }); + port.receive(new Uint8Array([2])); + const received: number[] = []; + link.onMessage((message) => received.push(message[0]!)); + port.receive(new Uint8Array([3])); + expect(received).toEqual([1, 2, 3]); + }); + + it("treat send returning false as backpressure, not a close", () => { + const port = new FakePort(); + const link = ipcLink(port); + let closed = 0; + link.onClose(() => closed++); + port.next = false; + link.send(new Uint8Array([1])); + link.send(new Uint8Array([2])); + expect(port.sent).toHaveLength(2); + expect(closed).toBe(0); + }); + + it.each([ + [ + "a thrown send", + (port: FakePort) => { + port.next = new Error("channel closed"); + }, + ], + ["an error in the send callback", (port: FakePort) => port.callbacks[0]!(new Error("EPIPE"))], + ["a disconnect", (port: FakePort) => port.disconnect()], + ])("close once on %s; listeners removed, later messages dropped (race 27)", (_, fail) => { + const port = new FakePort(); + const link = ipcLink(port); + let closed = 0; + const received: Uint8Array[] = []; + link.onClose(() => closed++); + link.onMessage((message) => received.push(message)); + link.send(new Uint8Array([1])); + fail(port); + link.send(new Uint8Array([2])); + port.disconnect(); + port.receive(new Uint8Array([3])); + expect(closed).toBe(1); + expect(received).toEqual([]); + expect(port.listeners).toBe(0); + }); + + it("disconnect the port on close, once; a port not connected gives a closed link", () => { + const port = new FakePort(); + const link = ipcLink(port); + link.close(); + link.close(); + expect(port.disconnects).toBe(1); + let closed = false; + ipcLink(port).onClose(() => { + closed = true; + }); + expect(closed).toBe(true); + }); +}); + +describe("IPC links to a forked child", () => { + it("carry Uint8Array messages both ways and close when the child exits mid-send (race 27)", async () => { + const child = fork(fileURLToPath(new URL("./fixtures/ipc-echo.ts", import.meta.url)), [], { + serialization: "advanced", + execArgv: ["--conditions=source", "--experimental-strip-types", "--no-warnings"], + }); + const link = ipcLink(child); + const echoed = new Promise((resolve) => link.onMessage(resolve)); + let closes = 0; + const closed = new Promise((resolve) => + link.onClose(() => { + closes++; + resolve(); + }), + ); + link.send(new Uint8Array([1, 2, 3])); + const message = await echoed; + expect(message).toBeInstanceOf(Uint8Array); + expect([...message]).toEqual([1, 2, 3]); + // The child exits on this one; the sends after it race the exit. + link.send(new Uint8Array([0xff])); + for (let send = 0; send < 20; send++) link.send(new Uint8Array(1024)); + await closed; + await flushed(); + expect(closes).toBe(1); + }, 20_000); +}); diff --git a/packages/chord/test/next/link/link.test.ts b/packages/chord/test/next/link/link.test.ts new file mode 100644 index 000000000..0f720f196 --- /dev/null +++ b/packages/chord/test/next/link/link.test.ts @@ -0,0 +1,96 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts"; +import { createHost } from "../../../src/next/host/index.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { expectCollected, flushed } from "../support.ts"; + +const bytes = (...values: number[]) => new Uint8Array(values); + +describe("memory links", () => { + it("deliver in order, one task later, and queue what arrives before the listener", async () => { + const [a, b] = memoryLinkPair(); + const got: number[] = []; + a.send(bytes(1)); + a.send(bytes(2)); + await flushed(); + b.onMessage((message) => got.push(message[0]!)); + expect(got).toEqual([1, 2]); + a.send(bytes(3)); + expect(got).toEqual([1, 2]); + await flushed(); + expect(got).toEqual([1, 2, 3]); + }); + + it("close this end at once and the peer one task later; send after close is a no-op", async () => { + const [a, b] = memoryLinkPair(); + const closes: string[] = []; + const got: number[] = []; + a.onClose(() => closes.push("a")); + b.onClose(() => closes.push("b")); + b.onMessage((message) => got.push(message[0]!)); + a.close(); + expect(closes).toEqual(["a"]); + a.send(bytes(1)); + await flushed(); + expect(closes).toEqual(["a", "b"]); + b.send(bytes(2)); + await flushed(); + expect(got).toEqual([]); + const late: string[] = []; + a.onClose(() => late.push("late")); + expect(late).toEqual(["late"]); + }); + + it("drop messages still in flight to a closed end", async () => { + const [a, b] = memoryLinkPair(); + const got: number[] = []; + a.onMessage((message) => got.push(message[0]!)); + b.send(bytes(1)); + a.close(); + await flushed(); + expect(got).toEqual([]); + }); + + it("stop delivering queued messages once a listener closed the link", async () => { + const [a, b] = memoryLinkPair(); + const got: number[] = []; + a.send(bytes(1)); + a.send(bytes(2)); + await flushed(); + b.onMessage((message) => { + got.push(message[0]!); + b.close(); + }); + expect(got).toEqual([1]); + }); + + it("keep no listener once closed", async () => { + const [a, b] = memoryLinkPair(); + const refs = (() => { + const listener = { onMessage: () => {} }; + a.onMessage(listener.onMessage); + a.onClose(listener.onMessage); + return [new WeakRef(listener.onMessage)]; + })(); + a.close(); + await flushed(); + await expectCollected(...refs); + expect(b).toBeDefined(); + }); +}); + +describe("a closed memory link", () => { + it("keeps no listener set after its close: accepting it keeps no host", async () => { + const [a, b] = memoryLinkPair(); + a.close(); + await flushed(); + const refs = await (async () => { + const host = createHost({ onError: () => {} }); + await host.load([], BACKGROUND_CONTEXT); + host.accept(a, { principal: { kind: "owner" } }, BACKGROUND_CONTEXT); + return [new WeakRef(host)]; + })(); + await expectCollected(...refs); + expect([a, b]).toBeDefined(); + }); +}); diff --git a/packages/chord/test/next/link/message.test.ts b/packages/chord/test/next/link/message.test.ts new file mode 100644 index 000000000..6adf76b4d --- /dev/null +++ b/packages/chord/test/next/link/message.test.ts @@ -0,0 +1,540 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { + binaryCodec, + checkMessage, + DeflatedPart, + describeMessage, + jsonPart, + type Message, + parsePart, + parsePayload, + payloadPart, + textCodec, +} from "../../../src/next/link/internal.ts"; +import type { Part } from "../../../src/next/link/part.ts"; +import { forEachCodec, testCodec } from "../support.ts"; + +/** A part as JSON text: a received part's bytes decoded (a deflated part stays as it is). */ +function partText(part: Part): Part { + return part instanceof Uint8Array ? new TextDecoder().decode(part) : part; +} + +const incarnation = "0190f5a2-7c3e-7d4a-9b1c-2d3e4f5a6b7c"; +const error = { kind: "chord", error: { code: "released", message: "gone" } } as const; + +/** One message of every kind, with the optional fields both present and absent; parts are JSON text. */ +const EVERY_KIND: Message[] = [ + { kind: "hello", protocol: 1, incarnation, calls: 256 }, + { kind: "hold", hold: 300, type: "app.notes", version: 2, fingerprint: "0123456789abcdef", params: '{"id":1}' }, + { + kind: "hold", + hold: 1, + type: "app.notes", + version: 1, + fingerprint: "0123456789abcdef", + params: '{"session":"S1"}', + address: { kind: "session", params: { id: "S1" } }, + principal: { kind: "guest" }, + incarnation, + }, + { + kind: "hold", + hold: 2, + type: "app.notes", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + incarnation, + }, + { kind: "release", hold: 0 }, + { kind: "call", call: 3, hold: 0, method: 1, args: '{"text":"x"}' }, + { kind: "call", call: 4, hold: 0, method: 0 }, + { kind: "abort", call: 3 }, + { + kind: "bound", + holds: [{ hold: 0, variant: 1, members: [{ member: 0, value: '[1,"a"]' }], failed: [{ member: 1, error }] }], + }, + { + kind: "bound", + holds: [ + { hold: 0, variant: 0, members: [], failed: [] }, + { hold: 5, variant: 2, members: [{ member: 1, value: "null" }], failed: [] }, + ], + }, + { + kind: "state", + holds: [ + { hold: 0, members: [{ member: 2, ops: '[["s",["n"],1]]' }], failed: [], retired: [] }, + { hold: 1, members: [], failed: [{ member: 0, error }], retired: [] }, + // Retired members (decision "One retirement mechanism"), alone and with a failure. + { hold: 2, members: [{ member: 1, ops: '[["r",{}]]' }], failed: [], retired: [1, 3] }, + { hold: 3, members: [], failed: [{ member: 0, error }], retired: [2] }, + ], + }, + { + kind: "failed", + hold: 0, + cause: { kind: "transient", error: { kind: "chord", error: { code: "unavailable", message: "m" } } }, + }, + { + kind: "failed", + hold: 0, + cause: { kind: "broken", error: { kind: "chord", error: { code: "unavailable", message: "m" } } }, + }, + { + kind: "failed", + hold: 0, + cause: { kind: "transient", error: { kind: "chord", error: { code: "unavailable", message: "m" } } }, + retryIn: 0, + }, + // The wire error format in `failed` and `gone`: a plugin's error travels too. + { kind: "failed", hold: 0, cause: { kind: "broken", error: { kind: "plugin", code: "config", message: "m" } } }, + { kind: "gone", hold: 1, error: { kind: "plugin", code: "config", message: "m", details: { at: 1 } } }, + { + kind: "failed", + hold: 9, + cause: { kind: "transient", error: { kind: "chord", error: { code: "unavailable", message: "m" } } }, + retryIn: 60_000, + }, + { kind: "retry", hold: 300 }, + { kind: "loading", hold: 300 }, + { kind: "result", call: 3, value: '{"n":1}' }, + { kind: "result", call: 3 }, + { kind: "error", call: 3, error: { kind: "plugin", code: "c", message: "m", details: [1] } }, + { kind: "gone", hold: 0, error: { kind: "chord", error: { code: "released", message: "m" } } }, + { kind: "ack" }, + { kind: "ack", origin: 1 }, + { kind: "closed", origin: 300 }, + { kind: "hold", origin: 2, hold: 0, type: "t", version: 1, fingerprint: "0123456789abcdef", params: "{}" }, + { kind: "release", origin: 2, hold: 3 }, + { kind: "call", origin: 2, call: 1, hold: 3, method: 0, args: "[1]" }, + { kind: "abort", origin: 2, call: 1 }, + { kind: "retry", origin: 2, hold: 3 }, + { kind: "bound", origin: 2, holds: [{ hold: 3, variant: 0, members: [], failed: [] }] }, + { kind: "state", origin: 2, holds: [] }, + { + kind: "failed", + origin: 2, + hold: 3, + cause: { kind: "broken", error: { kind: "chord", error: { code: "unavailable", message: "m" } } }, + }, + { kind: "loading", origin: 2, hold: 3 }, + { kind: "result", origin: 2, call: 1 }, + { kind: "error", origin: 2, call: 1, error: { kind: "plugin", code: "c", message: "m" } }, + { kind: "gone", origin: 2, hold: 3, error: { kind: "chord", error: { code: "released", message: "m" } } }, + { + kind: "hold", + hold: 3, + type: "app.notes", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + chain: [incarnation, incarnation], + limit: 65_536, + }, + { kind: "hold", hold: 4, type: "t", version: 1, fingerprint: "0123456789abcdef", params: "{}", limit: 0 }, + { kind: "revoke", revocation: { grant: "g1" } }, + { kind: "ping" }, + { kind: "pong" }, +]; + +/** A decoded message with every part as its text, to compare with one built from text parts. */ +function plain(message: Message): Message { + switch (message.kind) { + case "hold": + return { ...message, params: partText(message.params) }; + case "call": + return message.args === undefined ? message : { ...message, args: partText(message.args) }; + case "result": + return message.value === undefined ? message : { ...message, value: partText(message.value) }; + case "bound": + return { + ...message, + holds: message.holds.map((entry) => ({ + ...entry, + members: entry.members.map(({ member, value }) => ({ member, value: partText(value) })), + })), + }; + case "state": + return { + ...message, + holds: message.holds.map((entry) => ({ + ...entry, + members: entry.members.map(({ member, ops }) => ({ member, ops: partText(ops) })), + })), + }; + default: + return message; + } +} + +forEachCodec(() => { + describe("messages", () => { + it.each(EVERY_KIND)("round-trip %j and pass the message schema", (message) => { + const codec = testCodec(); + const decoded = checkMessage(codec.decode(codec.encode(message)), context); + expect(plain(decoded)).toEqual(message); + }); + + it("round-trip deflated parts in every part's place, still deflated (a relay forwards them so)", () => { + const codec = testCodec(); + const deflated = () => new DeflatedPart(new Uint8Array([0, 1, 2, 255])); + const messages: Message[] = [ + { kind: "hold", hold: 0, type: "t", version: 1, fingerprint: "0123456789abcdef", params: deflated() }, + { kind: "call", call: 0, hold: 0, method: 0, args: deflated() }, + { kind: "result", call: 0, value: deflated(), segments: [new Uint8Array([7])] }, + { + kind: "bound", + holds: [{ hold: 0, variant: 0, members: [{ member: 0, value: deflated() }], failed: [] }], + }, + { kind: "state", holds: [{ hold: 0, members: [{ member: 0, ops: deflated() }], failed: [], retired: [] }] }, + ]; + for (const message of messages) { + const decoded = checkMessage(codec.decode(codec.encode(message)), context); + expect(decoded).toEqual(message); + } + }); + + it("round-trip parts received as bytes (a relay forwards them as they came)", () => { + const codec = testCodec(); + const message: Message = { kind: "result", call: 0, value: new TextEncoder().encode('{"n":1}') }; + expect(plain(codec.decode(codec.encode(message)))).toEqual({ kind: "result", call: 0, value: '{"n":1}' }); + }); + + it("carry byte segments beside a payload, as views of the message", () => { + const codec = testCodec(); + const data = new Uint8Array(1024).fill(7); + const thumbs = [new Uint8Array([1, 2]), new Uint8Array([3])]; + const { json, segments } = payloadPart({ name: "a", data, thumbs }, true); + const message: Message = { kind: "call", call: 0, hold: 0, method: 0, args: json, segments }; + const bytes = codec.encode(message); + const decoded = codec.decode(bytes) as Extract; + expect(decoded.segments).toHaveLength(3); + expect(decoded.segments![0]!.buffer).toBe(bytes.buffer); + const args = parsePayload(decoded.args as Uint8Array, decoded.segments) as { + data: Uint8Array; + thumbs: Uint8Array[]; + }; + expect(args.data).toEqual(data); + expect(args.data.buffer).toBe(bytes.buffer); + expect(args.thumbs).toEqual(thumbs); + // The segments travel raw: the message is barely larger than the bytes it carries. + expect(bytes.byteLength).toBeLessThan(1024 + 3 + 100); + }); + + it("refuse an empty message, an unknown kind and trailing bytes", () => { + const codec = testCodec(); + const release = codec.encode({ kind: "release", hold: 1 }); + const trailing = new Uint8Array([...release, ...(codec === binaryCodec ? [0] : [0x20, 0x31])]); + expect(() => codec.decode(new Uint8Array())).toThrow(); + expect(() => + codec.decode(codec === binaryCodec ? new Uint8Array([99]) : new TextEncoder().encode('["?"]')), + ).toThrow("unknown kind"); + expect(() => codec.decode(trailing)).toThrow(); + }); + + it("refuse an origin on a kind without one, an origin 0, and a closed without one (the message schema)", () => { + const codec = testCodec(); + const check = (message: unknown) => () => + checkMessage(codec.decode(codec.encode(message as Message)), context); + expect(check({ kind: "ping", origin: 1 })).toThrow("message schema"); + expect(check({ kind: "ack", origin: 0 })).toThrow("message schema"); + const closed = codec === binaryCodec ? new Uint8Array([14]) : new TextEncoder().encode('["C"]'); + expect(() => checkMessage(codec.decode(closed), context)).toThrow("message schema"); + }); + + it("keep a leading U+FEFF in strings and parts (it is data, not a byte order mark)", () => { + const codec = testCodec(); + const message: Message = { + kind: "hold", + hold: 0, + type: "\uFEFFthing", + version: 1, + fingerprint: "0123456789abcdef", + params: jsonPart({ "\uFEFFkey": "\uFEFFvalue" }), + }; + const decoded = codec.decode(codec.encode(message)) as Extract; + expect(plain(decoded)).toEqual(message); + expect(parsePart(decoded.params as string | Uint8Array)).toEqual({ "\uFEFFkey": "\uFEFFvalue" }); + const result: Message = { kind: "result", call: 0, value: jsonPart("\uFEFF") }; + expect(plain(codec.decode(codec.encode(result)))).toEqual(result); + }); + + it("leave the envelope to the message schema: a misshapen error passes the codec and fails the schema", () => { + const codec = testCodec(); + const decoded = codec.decode(codec.encode({ kind: "error", call: 0, error: { kind: "nope" } } as never)); + expect(() => checkMessage(decoded, context)).toThrow("message: fails the message schema"); + expect(() => checkMessage({ kind: "release", hold: -1 }, context)).toThrow( + "message: fails the message schema", + ); + }); + }); +}); + +describe("the binary codec", () => { + it("writes ids as varints, the incarnation as 16 and the fingerprint as 8 raw bytes, no outer length", () => { + expect(binaryCodec.encode({ kind: "hello", protocol: 1, incarnation, calls: 1 })).toHaveLength(1 + 1 + 16 + 1); + expect([...binaryCodec.encode({ kind: "release", hold: 300 })]).toEqual([2, 0xac, 0x02]); + const hold = binaryCodec.encode({ + kind: "hold", + hold: 0, + type: "t", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + }); + // kind, id, type, version, fingerprint, params, the optional fields' flags + expect(hold).toHaveLength(1 + 1 + 2 + 1 + 8 + 3 + 1); + expect([...binaryCodec.encode({ kind: "ack" })]).toEqual([13]); + // An origin: the kind byte's high bit, then the origin as a varint. + expect([...binaryCodec.encode({ kind: "ack", origin: 300 })]).toEqual([13 | 0x80, 0xac, 0x02]); + expect([...binaryCodec.encode({ kind: "closed", origin: 1 })]).toEqual([14 | 0x80, 1]); + }); + + it("costs a state entry without retirements no byte: the failure count is doubled, plus one when they follow", () => { + const state = (retired: number[]): Message => ({ + kind: "state", + holds: [{ hold: 0, members: [], failed: [], retired }], + }); + // kind, entry count, hold, member count, failures (0 doubled) + expect([...binaryCodec.encode(state([]))]).toEqual([6, 1, 0, 0, 0]); + // the low bit set, then the retired count and member + expect([...binaryCodec.encode(state([3]))]).toEqual([6, 1, 0, 0, 1, 1, 3]); + const first = binaryCodec.decode(binaryCodec.encode(state([]))); + const second = binaryCodec.decode(binaryCodec.encode(state([]))); + const retired = (message: Message) => (message as Extract).holds[0]!.retired; + // No allocation per message: decoded entries without retirements share one empty list. + expect(retired(first)).toBe(retired(second)); + }); + + it("prefixes a part with its length doubled, plus one when it is deflated", () => { + const plain = binaryCodec.encode({ kind: "result", call: 0, value: "12" }); + const deflated = binaryCodec.encode({ kind: "result", call: 0, value: new DeflatedPart(new Uint8Array([7, 8])) }); + // kind, call, payload flags, prefix, bytes + expect([...plain]).toEqual([8, 0, 1, 4, 0x31, 0x32]); + expect([...deflated]).toEqual([8, 0, 1, 5, 7, 8]); + }); + + it("decodes parts as views of the message, never parsing them: a malformed part decodes", () => { + const part = new TextEncoder().encode("{not json"); + const bytes = binaryCodec.encode({ kind: "result", call: 0, value: part }); + const decoded = binaryCodec.decode(bytes) as Extract; + const value = decoded.value as Uint8Array; + expect(value.buffer).toBe(bytes.buffer); + expect(() => parsePart(value)).toThrow(); + }); + + it("refuses varints past a safe integer and lengths past the message", () => { + expect(() => binaryCodec.decode(new Uint8Array([2, ...new Array(8).fill(0xff), 0x7f]))).toThrow( + "a varint does not fit", + ); + expect(() => binaryCodec.decode(new Uint8Array([1, 0, 9, 0x61]))).toThrow("the message ends early"); + }); + + it("refuses a varint of more than 8 bytes, which would read as no number", () => { + const state = new Uint8Array([6, ...new Array(150).fill(0x80), 0]); + expect(() => binaryCodec.decode(state)).toThrow("a varint does not fit"); + }); + + it("refuses unknown hold flags", () => { + const hold = binaryCodec.encode({ + kind: "hold", + hold: 0, + type: "t", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + }); + const forged = hold.slice(); + forged[forged.length - 1] = 32; + expect(() => binaryCodec.decode(forged)).toThrow("unknown hold flags"); + }); + + it("returns a right-sized message: a large one keeps no larger backing store", () => { + const value = jsonPart("x".repeat(500_000)); + // Small fields after the large value: they must not double the buffer the message keeps. + const members = [ + { member: 0, value }, + { member: 1, value: "1" }, + ]; + const bytes = binaryCodec.encode({ kind: "bound", holds: [{ hold: 0, variant: 0, members, failed: [] }] }); + expect(bytes.buffer.byteLength - bytes.byteLength).toBeLessThanOrEqual(4096); + const small = binaryCodec.encode({ kind: "release", hold: 1 }); + expect(small.buffer.byteLength).toBe(small.byteLength); + // A message of about two thirds of the starting buffer keeps no spare room either. + const medium = binaryCodec.encode({ kind: "result", call: 0, value: jsonPart("x".repeat(160)) }); + expect(medium.buffer.byteLength).toBe(medium.byteLength); + }); + + it("refuses invalid UTF-8 in strings", () => { + expect(() => binaryCodec.decode(new Uint8Array([1, 0, 1, 0xff]))).toThrow(); + }); + + it("writes retry and loading as their kind and hold, and failed's retryIn as one more than it (0: absent)", () => { + expect([...binaryCodec.encode({ kind: "retry", hold: 300 })]).toEqual([11, 0xac, 0x02]); + expect([...binaryCodec.encode({ kind: "loading", hold: 1 })]).toEqual([12, 1]); + const cause = { kind: "broken", error: { kind: "chord", error: { code: "released", message: "" } } } as const; + const absent = binaryCodec.encode({ kind: "failed", hold: 0, cause }); + const present = binaryCodec.encode({ kind: "failed", hold: 0, cause, retryIn: 0 }); + expect(absent.at(-1)).toBe(0); + expect(present.at(-1)).toBe(1); + }); +}); + +describe("the text codec", () => { + const text = (message: Message) => new TextDecoder().decode(textCodec.encode(message)); + + it("writes one JSON tuple per message with short tags and positional fields, parts spliced in", () => { + expect(text({ kind: "release", hold: 3 })).toBe('["R",3]'); + expect(text({ kind: "call", call: 1, hold: 0, method: 2 })).toBe('["c",1,0,2]'); + const members = [{ member: 0, value: "5" }]; + expect(text({ kind: "bound", holds: [{ hold: 0, variant: 1, members, failed: [] }] })).toBe( + '["b",[[0,1,[[0,[0,5]]]]]]', + ); + expect(text({ kind: "retry", hold: 2 })).toBe('["T",2]'); + expect(text({ kind: "loading", hold: 2 })).toBe('["l",2]'); + const cause = { + kind: "transient", + error: { kind: "chord", error: { code: "unavailable", message: "m" } }, + } as const; + expect(text({ kind: "failed", hold: 2, cause })).toBe( + '["f",2,"t",{"kind":"chord","error":{"code":"unavailable","message":"m"}}]', + ); + expect(text({ kind: "failed", hold: 2, cause, retryIn: 750 })).toBe( + '["f",2,"t",{"kind":"chord","error":{"code":"unavailable","message":"m"}},750]', + ); + expect(text({ kind: "ack" })).toBe('["a"]'); + // An origin wraps the message's own tuple. + expect(text({ kind: "release", origin: 4, hold: 3 })).toBe('["@",4,["R",3]]'); + expect(text({ kind: "closed", origin: 4 })).toBe('["@",4,["C"]]'); + }); + + it("tags every part with its codec, [0, json] or [1, base64] for a deflated one, so no value is reserved", () => { + const message: Message = { kind: "result", call: 1, value: new DeflatedPart(new Uint8Array([1, 2, 3])) }; + expect(text(message)).toBe('["r",1,[1,"AQID"]]'); + expect(textCodec.decode(textCodec.encode(message))).toEqual(message); + // A value of any shape is data, the one-key `$deflated` object the first builder reserved included. + const shaped: Message = { kind: "result", call: 1, value: '{"$deflated":"AQID"}' }; + expect(text(shaped)).toBe('["r",1,[0,{"$deflated":"AQID"}]]'); + expect(textCodec.decode(textCodec.encode(shaped))).toEqual(shaped); + expect(text({ kind: "revoke", revocation: { grant: "g" } })).toBe('["V",{"grant":"g"}]'); + }); + + it("refuses a part that is no [codec, payload] pair, or of an unknown codec", () => { + const decode = (tuple: string) => textCodec.decode(new TextEncoder().encode(tuple)); + expect(() => decode('["r",1,{"n":1}]')).toThrow("no [codec, payload] pair"); + expect(() => decode('["r",1,[2,"AQID"]]')).toThrow("unknown codec"); + expect(() => decode('["r",1,[1,5]]')).toThrow("unknown codec"); + }); + + it("carries a binary link's deflated part over a text hop and back unchanged (a relay between codecs)", () => { + const part = new DeflatedPart(new Uint8Array([9, 8, 7])); + const message: Message = { + kind: "state", + holds: [{ hold: 0, members: [{ member: 0, ops: part }], failed: [], retired: [] }], + }; + const overText = textCodec.decode(textCodec.encode(binaryCodec.decode(binaryCodec.encode(message)))); + expect(binaryCodec.decode(binaryCodec.encode(overText))).toEqual(message); + }); + + it("leaves out a hold's absent optional fields at the end, null in the middle", () => { + const hold = { + kind: "hold", + hold: 0, + type: "t", + version: 1, + fingerprint: "0123456789abcdef", + params: "{}", + } as const; + expect(text(hold)).toBe('["H",0,"t",1,"0123456789abcdef",[0,{}]]'); + expect(text({ ...hold, incarnation })).toBe(`["H",0,"t",1,"0123456789abcdef",[0,{}],null,null,"${incarnation}"]`); + expect(text({ ...hold, limit: 5 })).toBe('["H",0,"t",1,"0123456789abcdef",[0,{}],null,null,null,null,5]'); + }); + + it("puts byte segments after the tuple and a newline, their lengths the tuple's last field", () => { + const bytes = textCodec.encode({ + kind: "result", + call: 0, + value: '{"$bytes":0}', + segments: [new Uint8Array([1, 2])], + }); + expect([...bytes]).toEqual([...new TextEncoder().encode('["r",0,[0,{"$bytes":0}],[2]]\n'), 1, 2]); + }); + + it("refuses a tuple of the wrong arity and segments without a payload carrying them", () => { + expect(() => textCodec.decode(new TextEncoder().encode('["R",1,2]'))).toThrow('a "R" message with 2 fields'); + expect(() => textCodec.decode(new TextEncoder().encode('["s",[[0,[],[],[],9]]]'))).toThrow( + "a state entry with 5 fields", + ); + expect(() => textCodec.decode(new TextEncoder().encode('["s",[[0]]]'))).toThrow("a state entry with 1 fields"); + expect(() => textCodec.decode(new Uint8Array([...new TextEncoder().encode('["R",1]\n'), 1]))).toThrow( + "byte segments without a payload", + ); + expect(() => textCodec.decode(new TextEncoder().encode('["@",1,["@",2,["a"]]]'))).toThrow("wraps one message"); + expect(() => textCodec.decode(new TextEncoder().encode('["@",1]'))).toThrow("wraps one message"); + }); +}); + +describe("describeMessage", () => { + it("prints a message in the text tuple form", () => { + expect(describeMessage({ kind: "abort", call: 7 })).toBe('["x",7]'); + expect(describeMessage({ kind: "result", call: 0, value: '{"$bytes":0}', segments: [new Uint8Array(3)] })).toBe( + '["r",0,[0,{"$bytes":0}],[3]]', + ); + }); +}); + +describe("a codec mismatch", () => { + it("decodes the other codec's hello as no message of its own", () => { + const hello: Message = { kind: "hello", protocol: 1, incarnation, calls: 256 }; + expect(() => binaryCodec.decode(textCodec.encode(hello))).toThrow(); + expect(() => textCodec.decode(binaryCodec.encode(hello))).toThrow(); + }); +}); + +describe("the binary codec's strings", () => { + it("survive the buffer growing around them, at every offset near the initial size", () => { + for (let length = 0; length < 400; length++) { + // A long value of three-byte characters (filling what it reserved), then a short one that lands at every offset around the buffer's first growth. + const members = [ + { member: 0, value: jsonPart("€".repeat(length)) }, + { member: 1, value: "1" }, + ]; + const message: Message = { kind: "bound", holds: [{ hold: 1, variant: 0, members, failed: [] }] }; + expect(plain(binaryCodec.decode(binaryCodec.encode(message)))).toEqual(message); + } + }); +}); + +describe("the binary codec's long strings", () => { + const texts = [ + ["ASCII", "x".repeat(10_000)], + ["three-byte characters", "€".repeat(10_000)], + ["surrogate pairs", "😀".repeat(5_000)], + ["mixed, at the in-place threshold", `${"a".repeat(4_093)}😀€`], + ["a leading U+FEFF", `\uFEFF${"y".repeat(5_000)}`], + ]; + + it.each(texts)("round-trip %s, encoded in place behind a padded length", (_, text) => { + const message: Message = { kind: "result", call: 0, value: jsonPart(text) }; + const bytes = binaryCodec.encode(message); + expect(plain(binaryCodec.decode(bytes))).toEqual(message); + expect(bytes.buffer.byteLength - bytes.byteLength).toBeLessThanOrEqual(4096); + }); +}); + +describe("the binary codec's short strings", () => { + it.each([0, 1, 41, 42, 43])("round-trip at %i three-byte characters, around the one-byte length", (length) => { + const type = "€".repeat(length); + const message: Message = { + kind: "hold", + hold: 0, + type, + version: 1, + fingerprint: "0123456789abcdef", + params: jsonPart(type), + }; + expect(plain(binaryCodec.decode(binaryCodec.encode(message)))).toEqual(message); + }); +}); diff --git a/packages/chord/test/next/link/part.test.ts b/packages/chord/test/next/link/part.test.ts new file mode 100644 index 000000000..83d76f54e --- /dev/null +++ b/packages/chord/test/next/link/part.test.ts @@ -0,0 +1,196 @@ +// Tests of parts on links: the limits per part (in UTF-8 bytes), compression at the origin, opening a part where it is +// owned, and the asynchronous inflate with its limit, its corrupt outcome and its stop when the link closes. + +import { describe, expect, it } from "vitest"; +import { + COMPRESS_ABOVE, + type Compression, + DeflatedPart, + inflateAsync, + linkLimits, + openPart, + originPart, + oversized, +} from "../../../src/next/link/part.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; +import { metered } from "../support.ts"; + +const UTF8 = new TextEncoder(); + +const NONE = { part: Number.POSITIVE_INFINITY, decoded: Number.POSITIVE_INFINITY }; + +/** A link that never closes. */ +const NEVER = new AbortController().signal; + +/** A JSON text of `length` characters that deflates well. */ +const json = (length: number) => JSON.stringify("x".repeat(length - 2)); + +/** What `fn` threw. */ +function thrown(fn: () => unknown): unknown { + try { + fn(); + } catch (error) { + return error; + } + throw new Error("expected a throw"); +} + +describe("limits per part", () => { + it("default to none on trusted links and to 16 MiB on the wire, 64 MiB decoded on untrusted ones", () => { + expect(linkLimits(true)).toEqual(NONE); + expect(linkLimits(false)).toEqual({ part: 16 * 1024 * 1024, decoded: 64 * 1024 * 1024 }); + expect(linkLimits(false, { part: 10 })).toEqual({ part: 10, decoded: 64 * 1024 * 1024 }); + expect(linkLimits(true, { decoded: 5 })).toEqual({ part: Number.POSITIVE_INFINITY, decoded: 5 }); + }); + + it("check a part's size on the wire with one comparison, nothing parsed", () => { + expect(oversized("[1,2]", 5)).toBeUndefined(); + expect(oversized("[1,2,3]", 5)?.toJSON()).toEqual({ + code: "too_large", + reason: "part", + message: "a part of 7 bytes is over the wire limit of 5 bytes", + }); + expect(oversized(new Uint8Array(6), 5)?.code).toBe("too_large"); + expect(oversized(new DeflatedPart(new Uint8Array(5)), 5)).toBeUndefined(); + }); + + it("count text in UTF-8 bytes, not UTF-16 units (a text link bounds what a binary link does)", () => { + const text = JSON.stringify("界".repeat(6000)); + expect(UTF8.encode(text).byteLength).toBe(18_002); + expect(oversized(text, 18_002)).toBeUndefined(); + expect(oversized(text, 18_001)?.message).toBe("a part of 18002 bytes is over the wire limit of 18001 bytes"); + // Two-byte characters and surrogate pairs (four bytes for two units). + expect(oversized('"é😀"', 8)).toBeUndefined(); + expect(oversized('"é😀"', 7)?.code).toBe("too_large"); + }); +}); + +describe("compression at the origin", () => { + it("deflates a JSON part above 16 KiB when the host compresses, and counts it", async () => { + const { context, counts } = await metered(); + const small = json(COMPRESS_ABOVE / 3); + expect(originPart(small, nodeCompression, context)).toBe(small); + const large = json(COMPRESS_ABOVE + 1); + const part = originPart(large, nodeCompression, context); + expect(part).toBeInstanceOf(DeflatedPart); + expect((part as DeflatedPart).bytes.byteLength).toBeLessThan(200); + expect(counts()["chord.link.deflates"]).toBe(1); + }); + + it("measures the threshold in UTF-8 bytes: a text of few units but many bytes is compressed", async () => { + const { context } = await metered(); + const text = JSON.stringify("界".repeat(6000)); + expect(text.length).toBeLessThan(COMPRESS_ABOVE); + expect(originPart(text, nodeCompression, context)).toBeInstanceOf(DeflatedPart); + // Within the threshold in bytes: its bytes, encoded once (the codec copies them). + const within = json(COMPRESS_ABOVE); + expect(originPart(within, nodeCompression, context)).toEqual(UTF8.encode(within)); + }); + + it("sends a part as it is without a compressor, and its bytes when deflating does not make it smaller", async () => { + const { context } = await metered(); + const large = json(COMPRESS_ABOVE * 2); + expect(originPart(large, undefined, context)).toBe(large); + const useless: Compression = { + deflate: (bytes) => bytes, + inflate: (bytes) => ({ kind: "bytes", bytes }), + }; + expect(originPart(large, useless, context)).toEqual(UTF8.encode(large)); + }); +}); + +describe("opening a part", () => { + it("returns a plain part within its limits; past either it throws too_large/part", async () => { + const { context } = await metered(); + expect(openPart("[1]", NONE, undefined, context)).toBe("[1]"); + expect(() => openPart("[1,2]", { part: 4, decoded: 100 }, undefined, context)).toThrow( + "a part of 5 bytes is over the wire limit of 4 bytes", + ); + // Not deflated: its decoded size is its size. + expect(() => openPart(UTF8.encode("[1,2]"), { part: 100, decoded: 4 }, undefined, context)).toThrow( + "a part of 5 bytes is over the decoded limit of 4 bytes", + ); + }); + + it("inflates a deflated part synchronously within the decoded limit, and counts it", async () => { + const { context, counts } = await metered(); + const text = json(100_000); + const part = new DeflatedPart(nodeCompression.deflate(UTF8.encode(text))); + const opened = openPart(part, { part: 1000, decoded: 100_000 }, nodeCompression, context); + expect(new TextDecoder().decode(opened as Uint8Array)).toBe(text); + expect(counts()["chord.link.inflates"]).toBe(1); + }); + + it("refuses a compression bomb: a small part inflating past the decoded limit", async () => { + const { context } = await metered(); + const bomb = new DeflatedPart(nodeCompression.deflate(new Uint8Array(10 * 1024 * 1024))); + expect(bomb.bytes.byteLength).toBeLessThan(50_000); + const error = thrown(() => openPart(bomb, { part: 1_000_000, decoded: 1024 * 1024 }, nodeCompression, context)); + expect(error).toMatchObject({ code: "too_large", reason: "part" }); + }); + + it("checks the wire size before inflating anything", async () => { + const { context, counts } = await metered(); + const part = new DeflatedPart(nodeCompression.deflate(UTF8.encode(json(100_000)))); + const error = thrown(() => openPart(part, { part: 10, decoded: 1_000_000 }, nodeCompression, context)); + expect(error).toMatchObject({ code: "too_large", message: expect.stringContaining("wire limit") }); + expect(counts()["chord.link.inflates"]).toBeUndefined(); + }); + + it("opens what an asynchronous inflate left on the part", async () => { + const { context } = await metered(); + const part = new DeflatedPart(nodeCompression.deflate(UTF8.encode(json(1000)))); + await inflateAsync(part, 2000, NEVER, context); + expect(openPart(part, { part: 1000, decoded: 2000 }, undefined, context)).toHaveLength(1000); + const bomb = new DeflatedPart(nodeCompression.deflate(UTF8.encode(json(1000)))); + await inflateAsync(bomb, 999, NEVER, context); + expect(() => openPart(bomb, { part: 1000, decoded: 999 }, undefined, context)).toThrow("decoded limit of 999"); + }); + + it("throws a plain error for a deflated part nothing inflated, or no raw deflate stream (a protocol error)", async () => { + const { context } = await metered(); + const part = new DeflatedPart(new Uint8Array([255, 255, 255])); + expect(() => openPart(part, NONE, undefined, context)).toThrow("a deflated part this host did not inflate"); + expect(() => openPart(part, NONE, nodeCompression, context)).toThrow("no raw deflate stream"); + expect(thrown(() => openPart(part, NONE, nodeCompression, context))).not.toMatchObject({ code: "too_large" }); + }); +}); + +describe("the asynchronous inflate", () => { + it("inflates with DecompressionStream, counting output bytes", async () => { + const { context, counts } = await metered(); + const text = json(200_000); + const part = new DeflatedPart(nodeCompression.deflate(UTF8.encode(text))); + await inflateAsync(part, Number.POSITIVE_INFINITY, NEVER, context); + expect(part.inflated?.kind).toBe("bytes"); + expect(new TextDecoder().decode(openPart(part, NONE, undefined, context) as Uint8Array)).toBe(text); + expect(counts()["chord.link.inflates"]).toBe(1); + }); + + it("stops past the limit: a bomb leaves too_large, never the whole output; a limit of 0 too", async () => { + const { context } = await metered(); + const bomb = new DeflatedPart(nodeCompression.deflate(new Uint8Array(64 * 1024 * 1024))); + await inflateAsync(bomb, 1024 * 1024, NEVER, context); + expect(bomb.inflated).toEqual({ kind: "too_large" }); + const one = new DeflatedPart(nodeCompression.deflate(new Uint8Array(1))); + await inflateAsync(one, 0, NEVER, context); + expect(one.inflated).toEqual({ kind: "too_large" }); + }); + + it("leaves corrupt for data that is no raw deflate stream", async () => { + const { context } = await metered(); + const part = new DeflatedPart(new Uint8Array([255, 255, 255])); + await inflateAsync(part, 100, NEVER, context); + expect(part.inflated?.kind).toBe("corrupt"); + }); + + it("stops at once when its link closes (the signal), leaving no outcome", async () => { + const { context } = await metered(); + const closing = new AbortController(); + const part = new DeflatedPart(nodeCompression.deflate(new Uint8Array(64 * 1024 * 1024))); + const inflating = inflateAsync(part, Number.POSITIVE_INFINITY, closing.signal, context); + closing.abort(); + await inflating; + expect(part.inflated).toBeUndefined(); + }); +}); diff --git a/packages/chord/test/next/link/values.test.ts b/packages/chord/test/next/link/values.test.ts new file mode 100644 index 000000000..84656b516 --- /dev/null +++ b/packages/chord/test/next/link/values.test.ts @@ -0,0 +1,64 @@ +import { describe, expect, it } from "vitest"; +import { isChordError } from "../../../src/next/data/index.ts"; +import { jsonPart, parsePart, parsePayload, payloadPart, resolveSegments } from "../../../src/next/link/values.ts"; + +describe("parts", () => { + it("are JSON text at their origin, parsed from text or from UTF-8 bytes where they are owned", () => { + expect(jsonPart({ a: [1, "x"] })).toBe('{"a":[1,"x"]}'); + expect(jsonPart(undefined)).toBe("null"); + expect(parsePart('{"a":1}')).toEqual({ a: 1 }); + expect(parsePart(new TextEncoder().encode('{"a":"\u00e9"}'))).toEqual({ a: "\u00e9" }); + }); + + it("that are no strict UTF-8 JSON throw (a protocol error where they are parsed)", () => { + expect(() => parsePart("{")).toThrow(); + expect(() => parsePart(new Uint8Array([0x22, 0xff, 0x22]))).toThrow(); + }); +}); + +describe("payloads", () => { + it("marked as carrying bytes write each byte leaf as a segment referenced by {$bytes: i}", () => { + const data = new Uint8Array(1024).fill(7); + const thumbs = [new Uint8Array([1, 2]), new Uint8Array([3])]; + const { json, segments } = payloadPart({ name: "chart.png", data, thumbs }, true); + expect(json).toBe('{"name":"chart.png","data":{"$bytes":0},"thumbs":[{"$bytes":1},{"$bytes":2}]}'); + expect(segments).toEqual([data, ...thumbs]); + expect(segments![0]).toBe(data); + const read = parsePayload(json, segments) as { data: Uint8Array; thumbs: Uint8Array[] }; + expect(read.data).toBe(data); + expect(read.thumbs).toEqual(thumbs); + }); + + it("marked as carrying bytes refuse the reserved key $bytes with invalid/args", () => { + let thrown: unknown; + try { + payloadPart({ meta: { $bytes: 0 }, data: new Uint8Array(1) }, true); + } catch (error) { + thrown = error; + } + expect(isChordError(thrown, "invalid") && thrown.reason).toBe("args"); + }); + + it("not marked are plain JSON, $bytes included; undefined is null", () => { + expect(payloadPart({ $bytes: 0 }, false)).toEqual({ json: '{"$bytes":0}', segments: undefined }); + expect(payloadPart(undefined, false).json).toBe("null"); + expect(parsePayload('{"$bytes":0}', undefined)).toEqual({ $bytes: 0 }); + }); + + it.each([ + ["an index past the segments", { $bytes: 1 }], + ["a non-integer index", { $bytes: "0" }], + ["other keys next to $bytes", { $bytes: 0, x: 1 }], + ])("refuse a malformed reference: %s", (_, value) => { + expect(() => resolveSegments({ a: [value] }, [new Uint8Array(1)])).toThrow( + "payload: a malformed byte segment reference", + ); + }); + + it("define a resolved key __proto__ as an own property", () => { + const parsed = JSON.parse('{"__proto__":{"$bytes":0}}'); + const resolved = resolveSegments(parsed, [new Uint8Array(1)]) as object; + expect(Object.getPrototypeOf(resolved)).toBe(Object.prototype); + expect(Object.getOwnPropertyDescriptor(resolved, "__proto__")?.value).toEqual(new Uint8Array(1)); + }); +}); diff --git a/packages/chord/test/next/link/websocket.test.ts b/packages/chord/test/next/link/websocket.test.ts new file mode 100644 index 000000000..821c1468e --- /dev/null +++ b/packages/chord/test/next/link/websocket.test.ts @@ -0,0 +1,159 @@ +import type { AddressInfo } from "node:net"; +import { describe, expect, it } from "vitest"; +import { WebSocket, WebSocketServer } from "ws"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { type WebSocketLike, webSocketLink } from "../../../src/next/link/index.ts"; +import { Notes, serveNotes, type TestPrincipal, transported } from "../support.ts"; + +/** A socket playing a WebSocket by hand. */ +class FakeSocket implements WebSocketLike { + readyState: number; + binaryType = "blob"; + readonly sent: Uint8Array[] = []; + closes = 0; + readonly #listeners = new Map void>>(); + + constructor(readyState: number) { + this.readyState = readyState; + } + + send(data: Uint8Array): void { + this.sent.push(data); + } + + close(): void { + this.closes++; + this.readyState = 3; + } + + addEventListener(type: string, listener: (event: { readonly data: unknown }) => void): void { + const listeners = this.#listeners.get(type) ?? new Set(); + listeners.add(listener); + this.#listeners.set(type, listeners); + } + + removeEventListener(type: string, listener: (event: { readonly data: unknown }) => void): void { + this.#listeners.get(type)?.delete(listener); + } + + emit(type: string, data?: unknown): void { + if (type === "open") this.readyState = 1; + for (const listener of [...(this.#listeners.get(type) ?? [])]) listener({ data }); + } + + get listeners(): number { + return [...this.#listeners.values()].reduce((count, listeners) => count + listeners.size, 0); + } +} + +describe("WebSocket links over a fake socket", () => { + it("buffer messages sent while connecting and send them on open, as binary frames (race 29)", () => { + const socket = new FakeSocket(0); + const link = webSocketLink(socket); + expect(socket.binaryType).toBe("arraybuffer"); + link.send(new Uint8Array([1])); + link.send(new Uint8Array([2])); + expect(socket.sent).toEqual([]); + socket.emit("open"); + expect(socket.sent.map((message) => message[0])).toEqual([1, 2]); + const received: number[] = []; + link.onMessage((message) => received.push(message[0]!)); + socket.emit("message", new Uint8Array([7]).buffer); + expect(received).toEqual([7]); + }); + + it("give a closed link for a socket closing or closed already", () => { + for (const state of [2, 3]) { + let closed = false; + webSocketLink(new FakeSocket(state)).onClose(() => { + closed = true; + }); + expect(closed).toBe(true); + } + }); + + it.each([ + ["close", (socket: FakeSocket) => socket.emit("close")], + ["error", (socket: FakeSocket) => socket.emit("error")], + ["a text frame", (socket: FakeSocket) => socket.emit("message", "text")], + ])("close once on %s, listeners removed", (_, fail) => { + const socket = new FakeSocket(1); + const link = webSocketLink(socket); + let closes = 0; + link.onClose(() => closes++); + fail(socket); + socket.emit("close"); + link.send(new Uint8Array([1])); + expect(closes).toBe(1); + expect(socket.sent).toEqual([]); + expect(socket.listeners).toBe(0); + }); +}); + +/** A `ws` server on a free loopback port whose connections a server host accepts, and a URL to it. */ +async function loopback(options: { readonly maxPayload?: number } = {}) { + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + await server.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); + const wss = new WebSocketServer({ port: 0, host: "127.0.0.1", maxPayload: options.maxPayload }); + await new Promise((resolve) => wss.once("listening", resolve)); + wss.on("connection", (socket) => server.accept(webSocketLink(socket), { principal: { kind: "owner" } }, context)); + const url = `ws://127.0.0.1:${(wss.address() as AddressInfo).port}`; + return { server, wss, url }; +} + +describe("WebSocket links over a loopback ws server", () => { + it("survive closing a socket that still connects: ws's late error is handled", async () => { + const errors: unknown[] = []; + const record = (error: unknown) => errors.push(error); + process.on("uncaughtException", record); + try { + const socket = new WebSocket("ws://127.0.0.1:1"); + let closed = false; + const link = webSocketLink(socket); + link.onClose(() => { + closed = true; + }); + link.close(); + expect(closed).toBe(true); + await new Promise((resolve) => socket.once("close", () => resolve())); + expect(errors).toEqual([]); + expect(socket.listenerCount("error")).toBe(0); + } finally { + process.off("uncaughtException", record); + } + }); + + it("carry a live reference both ways; the client's hello waits for open", async () => { + const { server, wss, url } = await loopback(); + const tab = createHost({ onError: () => {} }); + await tab.load([], context); + tab.connect(webSocketLink(new WebSocket(url)), { principal: { kind: "server" } }, context); + const notes = await tab.get(Notes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("expected the editor"); + expect(await notes.add({ text: "over ws" }, context)).toBe(1); + await transported(); + expect(notes.notes.value).toEqual([{ text: "over ws" }]); + await tab.close(context); + await server.close(context); + await new Promise((resolve) => wss.close(resolve)); + }); + + it("close the connection on a frame above maxPayload (1009): references go loading (race 28)", async () => { + const { server, wss, url } = await loopback({ maxPayload: 4096 }); + const tab = createHost({ onError: () => {} }); + await tab.load([], context); + const socket = new WebSocket(url); + const code = new Promise((resolve) => socket.once("close", resolve)); + tab.connect(webSocketLink(socket), { principal: { kind: "server" } }, context); + const notes = await tab.get(Notes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("expected the editor"); + const call = notes.echo({ data: new Uint8Array(10_000) }, context).catch((error: unknown) => error); + expect(await code).toBe(1009); + expect(await call).toMatchObject({ code: "unavailable" }); + expect(notes.state.value).toEqual({ kind: "loading" }); + await tab.close(context); + await server.close(context); + await new Promise((resolve) => wss.close(resolve)); + }); +}); diff --git a/packages/chord/test/next/remote/calls.test.ts b/packages/chord/test/next/remote/calls.test.ts new file mode 100644 index 000000000..79f3981da --- /dev/null +++ b/packages/chord/test/next/remote/calls.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it } from "vitest"; +import { CallWindow, type QueuedCall } from "../../../src/next/remote/calls.ts"; +import { expectCollected } from "../support.ts"; + +/** A call of `hold` that records its name in `sent` when it goes out. */ +function callOf(name: string, sent: string[], hold = 0): QueuedCall { + return { hold, send: () => void sent.push(name) }; +} + +describe("the call window", () => { + it("sends nothing before the peer's hello, then up to its N; the rest waits", () => { + const sent: string[] = []; + const window = new CallWindow(); + for (const name of ["a", "b", "c"]) window.submit(callOf(name, sent)); + expect(sent).toEqual([]); + window.opened(2); + expect(sent).toEqual(["a", "b"]); + window.freed(); + expect(sent).toEqual(["a", "b", "c"]); + }); + + it("sends waiting calls in order as slots free", () => { + const sent: string[] = []; + const window = new CallWindow(); + window.opened(1); + for (const name of ["a", "b", "c", "d"]) window.submit(callOf(name, sent)); + for (let slot = 0; slot < 3; slot++) window.freed(); + expect(sent).toEqual(["a", "b", "c", "d"]); + }); + + it("never preempts: a call that waits does not overtake while every slot is taken", () => { + const sent: string[] = []; + const window = new CallWindow(); + window.opened(1); + window.submit(callOf("long", sent)); + window.submit(callOf("short", sent)); + expect(sent).toEqual(["long"]); + }); + + it("takes out a cancelled waiting call, and every waiting call of a hold that rebinds", () => { + const sent: string[] = []; + const window = new CallWindow(); + window.opened(1); + window.submit(callOf("first", sent)); + const cancelled = callOf("cancelled", sent); + window.submit(cancelled); + const rebinding = [callOf("hold 1 a", sent, 1), callOf("hold 1 b", sent, 1)]; + for (const call of rebinding) window.submit(call); + window.submit(callOf("last", sent)); + window.remove(cancelled); + // A second remove of the same call changes nothing. + window.remove(cancelled); + expect(window.removeHold(1)).toEqual(rebinding); + window.freed(); + expect(sent).toEqual(["first", "last"]); + expect(window.clear()).toEqual([]); + }); + + it("keeps no call that went out or was taken", async () => { + const window = new CallWindow(); + window.opened(1); + const refs = (() => { + const sent = callOf("a", []); + const next = callOf("b", []); + const waiting = callOf("c", []); + for (const call of [sent, next, waiting]) window.submit(call); + window.freed(); + window.remove(waiting); + return [new WeakRef(sent), new WeakRef(waiting)]; + })(); + await expectCollected(...refs); + expect(window).toBeDefined(); + }); +}); diff --git a/packages/chord/test/next/remote/consumer.test.ts b/packages/chord/test/next/remote/consumer.test.ts new file mode 100644 index 000000000..32e4fb839 --- /dev/null +++ b/packages/chord/test/next/remote/consumer.test.ts @@ -0,0 +1,718 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local, map, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { PluginError } from "../../../src/next/data/index.ts"; +import { createHost, defineFacet, type HostError } from "../../../src/next/host/index.ts"; +import { resolveWatchable } from "../../../src/next/host/reference.ts"; +import { memoryLinkPair } from "../../../src/next/link/index.ts"; +import { DeflatedPart, jsonPart, type Part, wireError } from "../../../src/next/link/internal.ts"; +import { RemoteObject } from "../../../src/next/remote/remote-object.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; +import { + addNote, + bindNext, + EDITOR, + expectCollected, + forEachCodec, + framesOf, + instancesOf, + linked, + metered, + Notes, + RawPeer, + rawParent, + rawProvider, + rejection, + serveNotes, + testCodec, + transported, +} from "../support.ts"; + +/** Every remote object bound while this file runs, weakly. */ +const remoteObjects = instancesOf(RemoteObject, "hold"); + +forEachCodec(() => { + describe("references over a link", () => { + it("are live and hydrated when got, with the variant of the link's principal", async () => { + const { client, server } = await linked((h) => void serveNotes(h), { + child: { kind: "guest", session: "S1" }, + }); + await addNote(server, "S1", "a"); + const notes = await client.get(Notes, { session: "S1" }, context); + expect(notes.variant).toBe("reader"); + expect(notes.notes.value).toEqual([{ text: "a" }]); + expect(notes.count.value).toBe(1); + const frames = framesOf(await notes.notes.watch(context)); + await addNote(server, "S1", "b"); + await transported(); + expect(frames).toEqual([[[{ text: "a" }, { text: "b" }], [["p", [], 1, 0, [{ text: "b" }]]]]]); + expect(notes.count.value).toBe(2); + }); + + it("apply one state message as one batch: a derive over two members never sees half a commit", async () => { + const View = local("app.view", { version: 1, params: {}, members: { pair: str() } }); + const { server } = await linked((h) => void serveNotes(h)); + const seen: string[] = []; + const client = createHost({ onError: () => {} }); + // Connected first (W6): the view's factory references a server type. + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + client.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); + await client.load( + [ + { + name: "@test/view", + facet: defineFacet((h) => + h.serve(View, (_, self) => { + const notes = self.get(Notes, { session: "S1" }); + return { + pair: self.derive([notes.notes, notes.count], ([list, count]) => { + seen.push(`${list.length}/${count}`); + return `${list.length}/${count}`; + }), + }; + }), + ), + }, + ], + context, + ); + const view = await client.get(View, {}, context); + const frames = framesOf(await view.pair.watch(context)); + await addNote(server, "S1", "a"); + await addNote(server, "S1", "b"); + await transported(); + expect(frames.map(([value]) => value)).toEqual(["1/1", "2/2"]); + expect(seen.every((pair) => pair.split("/")[0] === pair.split("/")[1])).toBe(true); + }); + + it("call methods: results, ChordErrors, PluginErrors and other throws", async () => { + const { client } = await linked((h) => void serveNotes(h)); + const notes = await client.get(Notes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("expected the editor"); + expect(await notes.add({ text: "x" }, context)).toBe(1); + const chord = await rejection(notes.fail({ how: "chord" }, context)); + expect(chord).toEqual({ code: "forbidden", message: "not yours" }); + const plugin = await notes.fail({ how: "plugin" }, context).catch((error: unknown) => error); + expect(plugin).toBeInstanceOf(PluginError); + expect(plugin).toMatchObject({ code: "quota", message: "too many", details: { limit: 1 } }); + expect(await notes.fail({ how: "plain" }, context).catch((error: unknown) => error)).toMatchObject({ + name: "PluginError", + code: "internal", + message: "plain", + }); + expect(await rejection(notes.add({ text: 1 } as never, context))).toMatchObject({ + code: "invalid", + reason: "args", + }); + }); + + it("carry bytes as raw segments both ways", async () => { + const { client } = await linked((h) => void serveNotes(h)); + const notes = await client.get(Notes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("expected the editor"); + const data = new Uint8Array([1, 2, 3]); + const echoed = await notes.echo({ data }, context); + expect(echoed).toBeInstanceOf(Uint8Array); + expect([...echoed]).toEqual([1, 2, 3]); + }); + + it("cancel a call: the caller is rejected at once, the provider's handler aborted", async () => { + let probe: ReturnType | undefined; + const { client } = await linked((h) => { + probe = serveNotes(h); + }); + const notes = await client.get(Notes, { session: "S1" }, context); + const { context: cancellable, cancel } = withCancel(context); + const call = notes.wait(cancellable); + await transported(); + cancel(new Error("user cancelled")); + await expect(call).rejects.toThrow("user cancelled"); + await transported(); + expect(probe!.waits[0]!.abortSignal?.aborted).toBe(true); + }); + + it("release: pending calls reject released, the provider aborts their handlers and lets the object go", async () => { + let probe: ReturnType | undefined; + const { client } = await linked((h) => { + probe = serveNotes(h); + }); + const notes = await client.get(Notes, { session: "S1" }, context); + const watch = await notes.notes.watch(context); + const call = notes.wait(context); + await transported(); + notes.release(); + expect(await rejection(call)).toEqual({ code: "released", message: "app.notes(S1): released" }); + expect(await watch.closed).toEqual({ kind: "ended" }); + await transported(); + expect(probe!.waits[0]!.abortSignal?.aborted).toBe(true); + // The object ended with its last reference: a new get runs the factory again. + await client.get(Notes, { session: "S1" }, context); + expect(probe!.factories).toEqual(["S1 owner", "S1 owner"]); + }); + + it("a closing link rejects sent calls unavailable; references go loading, gets keep waiting", async () => { + const { client, peer } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + await bindNext(peer); + const notes = await got; + const call = notes.wait(context); + const pending = client.get(Notes, { session: "S2" }, context, { readyTimeout: 50 }); + await peer.next("call"); + peer.close(); + expect(await rejection(call)).toEqual({ code: "unavailable", message: "the link closed" }); + expect(notes.state.value).toEqual({ kind: "loading" }); + expect(await rejection(pending)).toEqual({ + code: "unavailable", + message: "app.notes(S2): did not become live within 50 ms", + }); + }); + }); + + describe("races at the consumer", () => { + it("ignores a bound and a state for a hold it released, and a result crossing its cancel", async () => { + const { client, peer, errors } = await rawProvider(); + const late = client.get(Notes, { session: "S1" }, context, { readyTimeout: 10 }); + const { hold } = await peer.next("hold"); + await expect(late).rejects.toThrow("did not become live"); + const members = [ + { member: EDITOR.state.notes!, value: "[]" }, + { member: EDITOR.state.count!, value: "0" }, + ]; + peer.send({ kind: "bound", holds: [{ hold, variant: EDITOR.variant, members, failed: [] }] }); + peer.send({ + kind: "state", + holds: [ + { hold, members: [{ member: EDITOR.state.count!, ops: jsonPart([["r", 1]]) }], failed: [], retired: [] }, + ], + }); + const got = client.get(Notes, { session: "S1" }, context); + await bindNext(peer); + const notes = await got; + const { context: cancellable, cancel } = withCancel(context); + const call = notes.wait(cancellable); + const { call: id } = await peer.next("call"); + cancel(new Error("cancelled")); + await expect(call).rejects.toThrow("cancelled"); + await peer.next("abort"); + peer.send({ kind: "result", call: id }); + await transported(); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + }); + }); + + describe("protocol errors at the consumer", () => { + it.each([ + ["wire ops failing Delta's checks", [{ member: EDITOR.state.count!, ops: '[["x"]]' }]], + ["a member index outside the variant", [{ member: 9, ops: '[["r",1]]' }]], + [ + "a member twice", + [ + { member: EDITOR.state.count!, ops: '[["r",1]]' }, + { member: EDITOR.state.count!, ops: '[["r",2]]' }, + ], + ], + ["a member's changes that are no list", [{ member: EDITOR.state.count!, ops: "1" }]], + ["ops that are no JSON", [{ member: EDITOR.state.count!, ops: "[" }]], + ["ops that do not apply to the member", [{ member: EDITOR.state.count!, ops: '[["s",["x"],1]]' }]], + ])("close the link: %s", async (_, members) => { + const { client, peer, errors } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + const hold = await bindNext(peer); + const notes = await got; + peer.send({ kind: "state", holds: [{ hold, members, failed: [], retired: [] }] }); + await transported(); + expect(peer.closed).toBe(true); + expect(errors.map(({ kind }) => kind)).toEqual(["protocol_error"]); + // Nothing of the message was applied; the reference waits for the next link. + expect(notes.state.value).toEqual({ kind: "loading" }); + expect(notes.count.value).toBe(0); + }); + + it.each([ + ["a result that is no JSON", { kind: "result", call: 0, value: "{" }], + ["a $bytes reference without its segment", { kind: "result", call: 0, value: '{"$bytes":0}', segments: [] }], + ["a malformed ChordError envelope", { kind: "error", call: 0, error: { kind: "chord", error: { code: 1 } } }], + ] as const)("reject the call and close the link on a malformed reply: %s", async (_, reply) => { + const { client, peer, errors } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + await bindNext(peer); + const notes = await got; + const call = notes.wait(context); + await peer.next("call"); + peer.send(reply); + // The text codec cannot even decode a result that is no JSON: the link closes and the call with it. + expect(await rejection(call)).toMatchObject({ code: "unavailable" }); + expect(peer.closed).toBe(true); + expect(errors.map(({ kind }) => kind)).toEqual(["protocol_error"]); + }); + + it("ignore a gone for a hold it does not know before reading its error (Q1)", async () => { + const { client, peer, errors } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + await bindNext(peer); + await got; + peer.send({ kind: "gone", hold: 9, error: { kind: "chord", error: { code: "nope" } } }); + await transported(); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + }); + + it("close the link on a malformed error envelope", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + peer.send({ + kind: "failed", + hold, + cause: { kind: "broken", error: { kind: "chord", error: { code: "nope", message: "m" } } }, + }); + await transported(); + expect(peer.closed).toBe(true); + expect(notes.state.value).toEqual({ kind: "loading" }); + }); + }); + + describe("the wire format of a hold", () => { + it("is the snapshot by member, then each commit's changed members with their ops", async () => { + const { server } = await linked((h) => void serveNotes(h)); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + peer.send({ + kind: "hold", + hold: 7, + type: Notes.name, + version: Notes.version, + fingerprint: Notes.fingerprint, + params: jsonPart({ session: "S1" }), + }); + const [bound] = (await peer.next("bound")).holds; + expect(bound!.variant).toBe(EDITOR.variant); + expect(EDITOR.named(bound!.members)).toEqual({ notes: [], count: 0 }); + await addNote(server, "S1", "a"); + const state = await peer.next("state"); + expect(state.holds).toMatchObject([{ hold: 7, failed: [] }]); + expect(EDITOR.named(state.holds[0]!.members)).toEqual({ + notes: [["p", [], 0, 0, [{ text: "a" }]]], + count: [["r", 1]], + }); + }); + }); + + describe("cost guards at the consumer", () => { + it("applies state without validating, walking or copying it: unchanged parts stay the same objects", async () => { + const { context: counting, counts } = await metered(); + const { server } = await linked((h) => void serveNotes(h)); + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + client.connect(b, { principal: { kind: "server" }, codec: testCodec() }, counting); + await addNote(server, "S1", "a"); + const notes = await client.get(Notes, { session: "S1" }, context); + const first = notes.notes.value[0]; + await addNote(server, "S1", "b"); + await transported(); + expect(notes.notes.value).toEqual([{ text: "a" }, { text: "b" }]); + expect(notes.notes.value[0]).toBe(first); + expect(counts()["chord.data.validations"]).toBeUndefined(); + expect(counts()["chord.data.walks"]).toBeUndefined(); + }); + }); + + describe("GC at the consumer", () => { + it("a closed link retained by the test, with its host, keeps no hold, replica or released reference", async () => { + const { client, link, refs } = await (async () => { + const { client, peer } = await rawProvider(); + const made = remoteObjects.length; + const got = client.get(Notes, { session: "S1" }, context); + await bindNext(peer); + const notes = await got; + const refs: WeakRef[] = [new WeakRef(resolveWatchable(notes.notes)!), new WeakRef(notes)]; + refs.push(...remoteObjects.slice(made)); + expect(refs).toHaveLength(3); + peer.close(); + await transported(); + // Across the drop the host keeps the reference's object for the next link; released, nothing stays. + notes.release(); + return { client, link: peer, refs }; + })(); + await expectCollected(...refs); + expect([client, link]).toBeDefined(); + }); + }); + + describe("one state message over several holds", () => { + const Pair = local("app.pair", { version: 1, params: {}, members: { pair: str() } }); + + /** A client whose `Pair` derives over the counts of two holds (S1, S2), both bound by a raw provider. */ + async function pair() { + const seen: string[] = []; + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const [a, b] = memoryLinkPair(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const facet = defineFacet((h) => + h.serve(Pair, (_, self) => { + const one = self.get(Notes, { session: "S1" }); + const two = self.get(Notes, { session: "S2" }); + return { + pair: self.derive([one.count, two.count], ([x, y]) => { + seen.push(`${x}/${y}`); + return `${x}/${y}`; + }), + }; + }), + ); + const loading = client.load([{ name: "@test/pair", facet }], context); + await bindNext(peer); + await bindNext(peer); + await loading; + const view = await client.get(Pair, {}, context); + const frames = framesOf(await view.pair.watch(context)); + return { client, peer, seen, frames, errors, view }; + } + + it("is applied as one batch: a derive over both holds never sees half of it, a failed member included", async () => { + const { peer, seen, frames, view } = await pair(); + const failed = [{ member: EDITOR.state.notes!, error: wireError(new PluginError("derive", "notes threw")) }]; + const count = [{ member: EDITOR.state.count!, ops: '[["r",1]]' }]; + peer.send({ + kind: "state", + holds: [ + { hold: 0, members: count, failed: [], retired: [] }, + { hold: 1, members: count, failed, retired: [] }, + ], + }); + await transported(); + expect(frames.map(([value]) => value)).toEqual(["1/1"]); + expect(seen).toEqual(["0/0", "1/1"]); + expect(view.pair.value).toBe("1/1"); + }); + + it("with a malformed later entry applies nothing and closes the link", async () => { + const { peer, seen, frames, errors } = await pair(); + peer.send({ + kind: "state", + holds: [ + { hold: 0, members: [{ member: EDITOR.state.count!, ops: '[["r",1]]' }], failed: [], retired: [] }, + { hold: 1, members: [{ member: EDITOR.state.count!, ops: '[["x"]]' }], failed: [], retired: [] }, + ], + }); + await transported(); + expect(frames).toEqual([]); + expect(seen).toEqual(["0/0"]); + expect(peer.closed).toBe(true); + expect(errors.map(({ kind }) => kind)).toEqual(["protocol_error"]); + }); + }); + + describe("$bytes outside methods with bytes", () => { + it("is plain data: {$bytes: 0} round-trips unchanged through a method without bytes", async () => { + const Echo = remote("app.echo", { + version: 1, + params: {}, + variants: { v: { methods: { echo: method(map(int()), map(int())) } } }, + }); + const { client } = await linked((h) => + h.serve(Echo, () => ({ variant: "v", members: { echo: async (args) => args } })), + ); + const echo = await client.get(Echo, {}, context); + expect(await echo.echo({ $bytes: 0 }, context)).toEqual({ $bytes: 0 }); + }); + }); +}); + +forEachCodec(() => { + describe("the call window at the consumer", () => { + /** A client connected to a raw provider announcing one concurrent call, and two bound references. */ + async function narrow() { + const client = createHost({ onError: () => {} }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const peer = new RawPeer(b, { calls: 1 }); + const one = client.get(Notes, { session: "S1" }, context); + const first = await bindNext(peer); + const two = client.get(Notes, { session: "S2" }, context); + const second = await bindNext(peer); + return { peer, one: await one, two: await two, holds: [first, second] as const }; + } + + /** The calls the provider received, by hold. */ + const calls = (peer: RawPeer) => + peer.received.flatMap((message) => (message.kind === "call" ? [message.hold] : [])); + + it("frees a slot once when a hold with a call in flight and one waiting is released", async () => { + const { peer, one, two, holds } = await narrow(); + const sent = one.wait(context); + const waiting = one.wait(context); + const other = [two.wait(context), two.wait(context)]; + await transported(); + expect(calls(peer)).toEqual([holds[0]]); + one.release(); + expect(await rejection(sent)).toMatchObject({ code: "released" }); + expect(await rejection(waiting)).toMatchObject({ code: "released" }); + await transported(); + expect(calls(peer)).toEqual([holds[0], holds[1]]); + expect(other).toHaveLength(2); + }); + + it("frees a slot once on gone", async () => { + const { peer, one, two, holds } = await narrow(); + const sent = one.wait(context); + const other = [two.wait(context), two.wait(context)]; + await transported(); + peer.send({ + kind: "gone", + hold: holds[0], + error: { kind: "chord", error: { code: "unavailable", message: "revoked" } }, + }); + expect(await rejection(sent)).toMatchObject({ message: "revoked" }); + await transported(); + expect(calls(peer)).toEqual([holds[0], holds[1]]); + expect(other).toHaveLength(2); + }); + + it("rejects a call still waiting when its hold rebinds: unavailable (race 26)", async () => { + const { peer, one, holds } = await narrow(); + const sent = one.wait(context); + const waiting = one.wait(context); + await transported(); + peer.send({ kind: "loading", hold: holds[0] }); + expect(await rejection(waiting)).toEqual({ code: "unavailable", message: `hold ${holds[0]}: rebinding` }); + const { call } = await peer.next("call"); + peer.send({ kind: "result", call }); + expect(await sent).toBe(null); + }); + }); +}); + +forEachCodec(() => { + describe("limits per part at the consumer", () => { + /** `[member, value]` of `Notes`'s editor variant: `notes` and `count`, the notes' JSON given as text. */ + const members = (notes: string): { member: number; value: Part }[] => [ + { member: EDITOR.state.notes!, value: notes }, + { member: EDITOR.state.count!, value: "1" }, + ]; + + it("rejects a call whose result is past the link's limits with too_large/part; the link stays", async () => { + const { client, peer } = await rawProvider({ limits: { part: 50 } }); + const notes = client.hold(Notes, { session: "S1" }, context); + await bindNext(peer); + await transported(); + if (notes.variant !== "editor") throw new Error("bound as the editor"); + const added = notes.add({ text: "x" }, context); + const { call } = await peer.next("call"); + peer.send({ kind: "result", call, value: jsonPart("y".repeat(60)) }); + expect(await rejection(added)).toMatchObject({ code: "too_large", reason: "part" }); + const again = notes.add({ text: "x" }, context); + peer.send({ kind: "result", call: (await peer.next("call")).call, value: "2" }); + expect(await again).toBe(2); + }); + + it("fails a hold whose snapshot is past the limits broken, releases it, and binds the message's others", async () => { + const { client, peer } = await rawProvider({ limits: { part: 50 } }); + const big = client.hold(Notes, { session: "S1" }, context); + const small = client.hold(Notes, { session: "S2" }, context); + const [one, two] = [await peer.next("hold"), await peer.next("hold")]; + const holds = [ + { + hold: one.hold, + variant: EDITOR.variant, + members: members(jsonPart([{ text: "z".repeat(60) }])), + failed: [], + }, + { hold: two.hold, variant: EDITOR.variant, members: members('[{"text":"a"}]'), failed: [] }, + ]; + peer.send({ kind: "bound", holds }); + expect(await peer.next("release")).toEqual({ kind: "release", hold: one.hold }); + expect(await peer.next("ack")).toEqual({ kind: "ack" }); + expect(big.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: { code: "too_large" } }, + }); + expect(small.notes.value).toEqual([{ text: "a" }]); + // No automatic re-hold; a retry holds again. + await transported(); + expect(peer.kinds()).toEqual([]); + big.retry(); + expect(big.state.value.kind).toBe("loading"); + const again = await peer.next("hold"); + expect(again.hold).toBeGreaterThan(two.hold); + }); + + it("acknowledges a message waiting for its asynchronous inflate only once applied, in order (race 14)", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + const list = [{ text: "q".repeat(30_000) }]; + const value = new DeflatedPart(nodeCompression.deflate(new TextEncoder().encode(JSON.stringify(list)))); + const bound = { hold, variant: EDITOR.variant, members: [...members("[]")], failed: [] }; + bound.members[0] = { member: EDITOR.state.notes!, value }; + peer.send({ kind: "bound", holds: [bound] }); + const ops = jsonPart([["r", 2]]); + peer.send({ + kind: "state", + holds: [{ hold, members: [{ member: EDITOR.state.count!, ops }], failed: [], retired: [] }], + }); + await peer.next("ack"); + expect(notes.notes.value).toEqual(list); + await peer.next("ack"); + expect(notes.count.value).toBe(2); + }); + + it("drops a message whose asynchronous inflate is in flight when the link closes, and those behind it (race 13)", async () => { + const { client, peer, errors } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + const list = [{ text: "q".repeat(30_000) }]; + const value = new DeflatedPart(nodeCompression.deflate(new TextEncoder().encode(JSON.stringify(list)))); + const bound = { hold, variant: EDITOR.variant, members: [...members("[]")], failed: [] }; + bound.members[0] = { member: EDITOR.state.notes!, value }; + peer.send({ kind: "bound", holds: [bound] }); + peer.close(); + for (let i = 0; i < 20; i++) await new Promise((resolve) => setTimeout(resolve, 1)); + expect(notes.state.value.kind).toBe("loading"); + expect(errors).toEqual([]); + }); + + it("inflates nothing it gave up (Q1): a corrupt deflated result of an unknown call, an entry of a released hold (both paths)", async () => { + for (const compression of [nodeCompression, undefined]) { + const { client, peer, errors } = await rawProvider({ compression }); + const kept = client.hold(Notes, { session: "S1" }, context); + const released = client.hold(Notes, { session: "S2" }, context); + const [a, b] = [await bindNext(peer), await bindNext(peer)]; + await transported(); + released.release(); + await peer.next("release"); + const corrupt = () => new DeflatedPart(new Uint8Array([255, 255, 255])); + peer.send({ kind: "result", call: 999, value: corrupt() }); + const ops = jsonPart([["r", 4]]); + peer.send({ + kind: "state", + holds: [ + { hold: b, members: [{ member: EDITOR.state.count!, ops: corrupt() }], failed: [], retired: [] }, + { hold: a, members: [{ member: EDITOR.state.count!, ops }], failed: [], retired: [] }, + ], + }); + await peer.next("ack"); + expect(kept.count.value).toBe(4); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + } + }); + + it("checks a deflated part's wire size before inflating it: corrupt data past the limit only fails its hold", async () => { + for (const compression of [nodeCompression, undefined]) { + const { client, peer, errors } = await rawProvider({ compression, limits: { part: 10 } }); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + const value = new DeflatedPart(new Uint8Array(20).fill(255)); + peer.send({ + kind: "bound", + holds: [ + { + hold, + variant: EDITOR.variant, + members: [ + { member: EDITOR.state.notes!, value }, + { member: EDITOR.state.count!, value: "1" }, + ], + failed: [], + }, + ], + }); + expect(await peer.next("release")).toEqual({ kind: "release", hold }); + expect(notes.state.value).toMatchObject({ kind: "failed", cause: { error: { code: "too_large" } } }); + expect(errors).toEqual([]); + } + }); + + it("keeps a hold failed for a part past the limits off the next link until retry, offline retry included", async () => { + const { client, peer } = await rawProvider({ limits: { part: 50 } }); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + peer.send({ + kind: "bound", + holds: [ + { hold, variant: EDITOR.variant, members: members(jsonPart([{ text: "z".repeat(60) }])), failed: [] }, + ], + }); + await peer.next("release"); + peer.close(); + await transported(); + const next = await rawParent(client, { part: 50 }); + await transported(); + // A reconnect holds nothing again: the failure is final until retry. + expect(next.kinds()).toEqual([]); + expect(notes.state.value.kind).toBe("failed"); + next.close(); + await transported(); + // A retry while offline holds it on the next link. + notes.retry(); + expect(notes.state.value.kind).toBe("loading"); + const third = await rawParent(client); + expect((await third.next("hold")).type).toBe(Notes.name); + }); + + it("stops an asynchronous inflate at once when the link closes: nothing of the link stays (GC)", async () => { + // Decompression that never yields: only the close ends it. + const streams: WeakRef[] = []; + class Stalled { + readonly writable: WritableStream; + readonly readable: ReadableStream; + constructor() { + const stream = new TransformStream({ transform() {} }); + [this.writable, this.readable] = [stream.writable, stream.readable]; + streams.push(new WeakRef(this)); + } + } + vi.stubGlobal("DecompressionStream", Stalled); + try { + const refs = await (async () => { + const { client, peer } = await rawProvider(); + client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + const value = new DeflatedPart(nodeCompression.deflate(new TextEncoder().encode("[]"))); + const entry = { hold, variant: EDITOR.variant, members: members("[]"), failed: [] }; + entry.members[0] = { member: EDITOR.state.notes!, value }; + peer.send({ kind: "bound", holds: [entry] }); + await transported(); + expect(streams).toHaveLength(1); + peer.close(); + await transported(); + await client.close(context); + return [new WeakRef(client), ...streams]; + })(); + await expectCollected(...refs); + } finally { + vi.unstubAllGlobals(); + } + }); + + it("fails only the hold of a state part past the limits (race 10); the others of the message apply", async () => { + const { client, peer } = await rawProvider({ limits: { part: 50 } }); + const one = client.hold(Notes, { session: "S1" }, context); + const two = client.hold(Notes, { session: "S2" }, context); + const [a, b] = [await bindNext(peer), await bindNext(peer)]; + await transported(); + const ops = (count: number, pad: number) => jsonPart([["r", count], ...Array(pad).fill(["r", count])]); + peer.send({ + kind: "state", + holds: [ + { hold: a, members: [{ member: EDITOR.state.count!, ops: ops(5, 10) }], failed: [], retired: [] }, + { hold: b, members: [{ member: EDITOR.state.count!, ops: ops(7, 0) }], failed: [], retired: [] }, + ], + }); + expect(await peer.next("release")).toEqual({ kind: "release", hold: a }); + await transported(); + expect(one.state.value).toMatchObject({ kind: "failed", cause: { error: { code: "too_large" } } }); + expect(two.count.value).toBe(7); + expect(peer.closed).toBe(false); + }); + }); +}); diff --git a/packages/chord/test/next/remote/hold.test.ts b/packages/chord/test/next/remote/hold.test.ts new file mode 100644 index 000000000..ab1c07b8a --- /dev/null +++ b/packages/chord/test/next/remote/hold.test.ts @@ -0,0 +1,694 @@ +import { describe, expect, it, vi } from "vitest"; +import { type Context, BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import type * as Delta from "../../../src/delta/index.ts"; +import { decoder } from "../../../src/delta/index.ts"; +import { bool, int, local, method, obj, remote } from "../../../src/next/contracts/index.ts"; +import { broken } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type Host, type Self } from "../../../src/next/host/index.ts"; +import type { ChordHost } from "../../../src/next/host/internal.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { Hold } from "../../../src/next/remote/hold.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + changesOf, + expectCollected, + fakeTimers, + forEachCodec, + instancesOf, + linked, + RawPeer, + rawLinked, + type TestPrincipal, + testCodec, + transported, + wireNames, +} from "../support.ts"; + +/** Every Delta encoder made while this file runs, weakly: the holds' member encoders among them. */ +const encoders = vi.hoisted(() => [] as WeakRef[]); + +vi.mock("../../../src/delta/index.ts", async (importOriginal) => { + const original = await importOriginal(); + return { + ...original, + encoder: () => { + const made = original.encoder(); + encoders.push(new WeakRef(made)); + return made; + }, + }; +}); + +/** Every hold started while this file runs, weakly. */ +const holds = instancesOf(Hold, "start"); + +const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + +/** `half` throws once `n` is 2: a member failing after the object is live (W1). */ +const Halves = remote("app.halves", { + version: 1, + params: {}, + variants: { v: { state: { n: obj({ n: int() }), half: int() }, methods: { set: method(int()), wait: method() } } }, +}); + +/** The wire's indexes of `Halves`. */ +const HALVES = wireNames(Halves, "v"); + +/** A server serving `Halves` and a raw peer; `set` writes `n` through hold 0, the object of the link's principal. */ +async function halves() { + const linked = await rawLinked((h) => + h.serve(Halves, (_, _principal, self) => { + const n = self.state(N); + return { + variant: "v", + members: { + n, + half: self.derive([n], ([{ n }]) => { + if (n === 2) throw new Error("half threw"); + return n / 2; + }), + set: async (value, ctx) => { + await n.change((draft) => { + draft.n = value; + }, ctx); + return null; + }, + wait: (ctx) => new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", () => resolve(null))), + }, + }; + }), + ); + let calls = 0; + const set = async (value: number) => { + const call = calls++; + linked.peer.send({ kind: "call", call, hold: 0, method: HALVES.methods.set!, args: jsonPart(value) }); + expect((await linked.peer.next("result")).call).toBe(call); + }; + return { ...linked, set }; +} + +function hold(peer: RawPeer, id: number, Type: { name: string; version: number; fingerprint: string }) { + const { name: type, version, fingerprint } = Type; + peer.send({ kind: "hold", hold: id, type, version, fingerprint, params: "{}" }); +} + +/** The provider object behind a hold, reached through the host's own lookup (to end it by hand). */ +function providerObject(server: unknown) { + const host = server as ChordHost; + const { name: type, version, fingerprint } = Halves; + return host.serving({ type, version, fingerprint, params: "{}" }, {}, { kind: "owner" }, context); +} + +/** A remote type whose activation never completes: its open waits for its abort. */ +const Pending = remote("app.pending", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + +function servePending(h: Parameters[0]>[0]) { + h.serve(Pending, (_, _principal, self) => { + self.open( + (ctx) => new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", resolve)), + () => {}, + ); + return { variant: "v", members: { n: constant(1) } }; + }); +} + +forEachCodec(() => { + describe("the parts of a hold", () => { + it("are its members, each with its own encoder: the snapshot by member, then the changed members' ops", async () => { + const { peer, set } = await halves(); + hold(peer, 0, Halves); + expect(HALVES.named((await peer.next("bound")).holds[0]!.members)).toEqual({ n: { n: 0 }, half: 0 }); + await set(1); + const first = HALVES.named((await peer.next("state")).holds[0]!.members); + expect(first).toEqual({ n: [["s", ["n"], 1]], half: [["r", 0.5]] }); + await set(4); + // The member's encoder interns a path on its second use. + const n = decoder(); + n.decode(first.n as unknown[]); + const second = HALVES.named((await peer.next("state")).holds[0]!.members); + expect(n.decode(second.n as unknown[])).toEqual([["s", ["n"], 4]]); + }); + }); + + describe("failed members", () => { + it("are listed in the header of the state that carries the failure, never in its part; they recover in the same hold", async () => { + const { peer, set } = await halves(); + hold(peer, 0, Halves); + hold(peer, 1, Halves); + await peer.next("bound"); + await peer.next("bound"); + await set(2); + const state = await peer.next("state"); + const failed = [ + { member: HALVES.state.half!, error: { kind: "plugin", code: "internal", message: "half threw" } }, + ]; + // Race 12: each hold's entry carries its healthy members' ops and the failure. + expect(state.holds.map(({ hold, failed }) => ({ hold, failed }))).toEqual([ + { hold: 0, failed }, + { hold: 1, failed }, + ]); + for (const entry of state.holds) expect(HALVES.named(entry.members)).toEqual({ n: [["s", ["n"], 2]] }); + await set(3); + const next = await peer.next("state"); + expect(next.holds.map(({ hold, failed }) => ({ hold, failed }))).toEqual([ + { hold: 0, failed: [] }, + { hold: 1, failed: [] }, + ]); + // The derive healed: its next value replaces the root. + expect(HALVES.named(next.holds[0]!.members).half).toEqual([["r", 1.5]]); + }); + + it("reach a client's watch over the link as one failure call, then the healed value: the watch stays open", async () => { + const { server, client } = await linked((h) => + h.serve(Halves, (_, _principal, self) => { + const n = self.state(N); + return { + variant: "v", + members: { + n, + half: self.derive([n], ([{ n }]) => { + if (n === 2) throw new Error("half threw"); + return n / 2; + }), + set: async (value, ctx) => { + await n.change((draft) => { + draft.n = value; + }, ctx); + return null; + }, + wait: async () => null, + }, + }; + }), + ); + const halves = await client.get(Halves, {}, context); + if (halves.variant !== "v") throw new Error("expected variant v"); + const calls = changesOf(await halves.half.watch(context)); + await halves.set(2, context); + await transported(); + expect(calls).toEqual([{ kind: "failed", error: expect.objectContaining({ message: "half threw" }) }]); + await halves.set(6, context); + await transported(); + expect(halves.half.value).toBe(3); + expect(calls.at(-1)).toEqual({ kind: "value", value: 3, ops: [["r", 3]], retired: false }); + await client.close(context); + await server.close(context); + }); + + it("are listed in bound for a hold made after the failure", async () => { + const { peer, set } = await halves(); + hold(peer, 0, Halves); + await peer.next("bound"); + await set(2); + await peer.next("state"); + hold(peer, 1, Halves); + const [bound] = (await peer.next("bound")).holds; + expect(bound!.failed).toEqual([ + { member: HALVES.state.half!, error: { kind: "plugin", code: "internal", message: "half threw" } }, + ]); + expect(HALVES.named(bound!.members)).toEqual({ n: { n: 2 } }); + // Followed though it failed at the snapshot: it recovers in this hold. + await set(4); + const next = await peer.next("state"); + expect(HALVES.named(next.holds.find(({ hold }) => hold === 1)!.members).half).toEqual([["r", 2]]); + }); + }); + + describe("transient failures", () => { + // Timers stand still: the failed object's backoff window does not end while the test runs. + fakeTimers(); + + it("a new hold of an object failed transiently is sent the failure at once", async () => { + const Flaky = remote("app.flaky", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const { peer } = await rawLinked((h) => + h.serve(Flaky, (_, _principal, self) => { + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + return { variant: "v", members: { n: constant(1) } }; + }), + ); + hold(peer, 0, Flaky); + const cause = { + kind: "transient", + error: { kind: "chord", error: { code: "unavailable", message: "app.flaky: offline" } }, + }; + expect(await peer.next("failed")).toMatchObject({ hold: 0, cause }); + hold(peer, 1, Flaky); + // The object stays failed for its backoff window (1 s); the new hold does not wait for the next attempt. + expect(await peer.next("failed")).toMatchObject({ hold: 1, cause }); + }); + }); + + // A hold whose object ends sends `gone` and nothing for its pending calls (race 4): through revocation, the path + // that ends a held object (`host/revocation.test.ts`, "aborts running calls"). + + describe("GC", () => { + it("a closed link retained by the test keeps no hold, encoder or object", async () => { + const { server, refs, link } = await (async () => { + const { server } = await halves(); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const [held, made] = [holds.length, encoders.length]; + hold(peer, 0, Halves); + await peer.next("bound"); + const refs: WeakRef[] = [new WeakRef(providerObject(server)), ...holds.slice(held)]; + refs.push(...encoders.slice(made)); + expect(refs).toHaveLength(4); + peer.close(); + await transported(); + return { server, refs, link: a }; + })(); + await expectCollected(...refs); + expect([server, link]).toBeDefined(); + }); + + it("a released pending hold is kept neither by its object, still activating for another hold, nor by its waits", async () => { + const { server, peer, refs } = await (async () => { + const { server, peer } = await rawLinked((h) => servePending(h)); + hold(peer, 0, Pending); + await transported(); + const held = holds.length; + hold(peer, 1, Pending); + await transported(); + const refs = holds.slice(held); + expect(refs).toHaveLength(1); + peer.send({ kind: "release", hold: 1 }); + await transported(); + return { server, peer, refs }; + })(); + await expectCollected(...refs); + expect([server, peer]).toBeDefined(); + }); + + it("a pending activation on a closed link keeps no hold", async () => { + const { server, link, refs } = await (async () => { + const { server } = await rawLinked((h) => servePending(h)); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const held = holds.length; + hold(peer, 0, Pending); + await transported(); + const refs = holds.slice(held); + peer.close(); + await transported(); + return { server, link: a, refs }; + })(); + await expectCollected(...refs); + expect([server, link]).toBeDefined(); + }); + + it("a hold released while waiting for load is not kept by the wait", async () => { + const Slow = local("app.slow", { version: 1, params: {}, members: { n: int() } }); + const { server, refs } = await (async () => { + const server = createHost({ + principal: { kind: "server" }, + readyTimeout: 2_000, + onError: () => {}, + }); + const facet = defineFacet((h) => { + servePending(h); + // A host-lifetime object whose open never resolves: `load` waits for it (until its readyTimeout). + h.serve(Slow, (_, self) => { + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { n: constant(1) }; + }); + }); + void server.load([{ name: "@test/app", facet }], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const held = holds.length; + hold(peer, 0, Pending); + await transported(); + peer.send({ kind: "release", hold: 0 }); + await transported(); + return { server, refs: holds.slice(held) }; + })(); + expect(refs).toHaveLength(1); + await expectCollected(...refs); + expect(server).toBeDefined(); + }); + }); +}); + +/** A remote type whose open throws `offline` (broken when `failing.broken`) while `failing.left` is above 0. */ +const Flaky = remote("app.flaky", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + +function serveFlaky(h: Parameters[0]>[0], failing: { left: number; broken: boolean }) { + h.serve(Flaky, (_, _principal, self) => { + self.open( + async () => { + if (failing.left-- <= 0) return; + throw failing.broken ? broken(new Error("offline")) : new Error("offline"); + }, + () => {}, + ); + return { variant: "v", members: { n: constant(1) } }; + }); +} + +/** A second raw peer on `server`, stamped with the same principal as `rawLinked`'s: it holds the same objects. */ +function secondPeer(server: Host): RawPeer { + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + return new RawPeer(b); +} + +const offline = { code: "unavailable", message: "app.flaky: offline" }; + +forEachCodec(() => { + describe("notices of a failed object", () => { + fakeTimers(); + + it("its holds get failed with the rest of its window, loading for each attempt, then bound (race 6)", async () => { + const { peer } = await rawLinked((h) => serveFlaky(h, { left: 2, broken: false })); + hold(peer, 0, Flaky); + expect(await peer.next("failed")).toEqual({ + kind: "failed", + hold: 0, + cause: { kind: "transient", error: { kind: "chord", error: offline } }, + retryIn: 1_000, + }); + await vi.advanceTimersByTimeAsync(600); + hold(peer, 1, Flaky); + expect(await peer.next("failed")).toMatchObject({ hold: 1, retryIn: 400 }); + await vi.advanceTimersByTimeAsync(400); + await transported(); + expect(peer.kinds()).toEqual(["hello", "loading", "loading", "failed", "failed"]); + expect((await peer.next("failed")).retryIn).toBe(2_000); + await vi.advanceTimersByTimeAsync(2_000); + await peer.next("bound"); + await peer.next("bound"); + expect(peer.kinds()).toEqual(["hello", "loading", "loading", "failed", "loading", "loading"]); + }); + + it("a retry from one consumer renews a broken object for every holder: loading, then bound (races 12, 13)", async () => { + const failing = { left: 1, broken: true }; + const { server, peer } = await rawLinked((h) => serveFlaky(h, failing)); + const other = secondPeer(server); + hold(peer, 0, Flaky); + hold(other, 0, Flaky); + const broken = { + kind: "failed", + hold: 0, + cause: { kind: "broken", error: { kind: "chord", error: offline } }, + }; + expect([await peer.next("failed"), await other.next("failed")]).toEqual([broken, broken]); + peer.send({ kind: "retry", hold: 0 }); + for (const consumer of [peer, other]) { + expect(await consumer.next("loading")).toEqual({ kind: "loading", hold: 0 }); + expect((await consumer.next("bound")).holds[0]).toMatchObject({ hold: 0 }); + } + }); + + it("a retry skips a transient failure's window for every holder (race 12)", async () => { + const { server, peer } = await rawLinked((h) => serveFlaky(h, { left: 1, broken: false })); + const other = secondPeer(server); + hold(peer, 0, Flaky); + await peer.next("failed"); + hold(other, 0, Flaky); + await other.next("failed"); + other.send({ kind: "retry", hold: 0 }); + for (const consumer of [peer, other]) { + await consumer.next("loading"); + await consumer.next("bound"); + } + }); + + it("a retry of a live or an unknown hold does nothing", async () => { + const { peer } = await rawLinked((h) => serveFlaky(h, { left: 0, broken: false })); + hold(peer, 0, Flaky); + await peer.next("bound"); + peer.send({ kind: "retry", hold: 0 }); + peer.send({ kind: "retry", hold: 9 }); + await transported(); + expect([peer.kinds(), peer.closed]).toEqual([["hello"], false]); + }); + }); + + describe("a reconnecting consumer", () => { + const Tally = remote("app.tally", { + version: 1, + params: {}, + variants: { v: { state: { n: obj({ n: int() }) }, methods: { bump: method() } } }, + }); + const Fresh = remote("app.fresh", { + version: 1, + params: {}, + variants: { v: { state: { n: obj({ n: int() }) }, methods: { bump: method() } } }, + }); + + /** A server serving `Tally` (lingering a minute) and `Fresh` (not lingering, its native's close waits for `closing`). */ + async function served(closing: Promise) { + const serve = (self: Self) => { + const doc = self.state(N); + return { + variant: "v" as const, + members: { + n: doc, + bump: async (ctx: Context) => { + await doc.change((draft) => { + draft.n++; + }, ctx); + return null; + }, + }, + }; + }; + return linked((h) => { + h.serve(Tally, (_, _principal, self) => serve(self), { linger: 60_000 }); + h.serve(Fresh, (_, _principal, self) => { + self.open( + async () => null, + () => closing, + ); + return serve(self); + }); + }); + } + + /** Drops the client's link and connects it again. */ + async function reconnect(server: Host, client: Host, link: Link) { + link.close(); + await transported(); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + client.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); + await transported(); + } + + it("finds a lingering object as it left it: its in-memory documents kept; one without linger is new (race 7)", async () => { + const closing = Promise.resolve(); + const { server, client, links } = await served(closing); + const tally = await client.get(Tally, {}, context); + const fresh = await client.get(Fresh, {}, context); + await tally.bump(context); + await fresh.bump(context); + await reconnect(server, client, links[0]); + expect([tally.state.value, fresh.state.value]).toEqual([{ kind: "live" }, { kind: "live" }]); + expect([tally.n.value, fresh.n.value]).toEqual([{ n: 1 }, { n: 0 }]); + }); + + it("re-holds an object whose previous instance still tears down: bound once that ended (race 17)", async () => { + const closing = Promise.withResolvers(); + const { server, client, links } = await served(closing.promise); + const fresh = await client.get(Fresh, {}, context); + await fresh.bump(context); + await reconnect(server, client, links[0]); + expect([fresh.state.value, fresh.n.value]).toEqual([{ kind: "loading" }, { n: 1 }]); + closing.resolve(); + await transported(); + expect([fresh.state.value, fresh.n.value]).toEqual([{ kind: "live" }, { n: 0 }]); + }); + }); +}); + +/** + * A remote type whose document can be retired; `big` derives a flag that a retirement of a small value leaves as is, + * `t` is `n` and throws while it is negative. + */ +const Retiring = remote("app.retiring", { + version: 1, + params: {}, + variants: { + v: { + state: { doc: obj({ n: int() }), big: bool(), t: int() }, + methods: { set: method(int()), retire: method() }, + }, + }, +}); + +const RETIRING = wireNames(Retiring, "v"); + +/** Every member of `Retiring` retired with its document, sorted. */ +const retiredAll = [RETIRING.state.doc!, RETIRING.state.big!, RETIRING.state.t!].sort(); + +/** A server serving `Retiring` to a raw peer that acks only when told; `call` calls a method of hold 0. */ +async function retiring() { + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + await server.load( + [ + { + name: "@test/app", + facet: defineFacet((h) => + h.serve(Retiring, (_, _principal, self) => { + const doc = self.state(N); + return { + variant: "v", + members: { + doc, + big: self.derive([doc], ([{ n }]) => n > 10), + t: self.derive([doc], ([{ n }]) => { + if (n < 0) throw new Error(`failed ${n}`); + return n; + }), + set: async (n, ctx) => { + await doc.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + retire: async (ctx) => { + await doc.retire(ctx); + return null; + }, + }, + }; + }), + ), + }, + ], + context, + ); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b, { ack: false }); + await transported(); + let calls = 0; + const call = async (method: "set" | "retire", args?: number) => { + const id = calls++; + const payload = args === undefined ? {} : { args: jsonPart(args) }; + peer.send({ kind: "call", call: id, hold: 0, method: RETIRING.methods[method]!, ...payload }); + expect((await peer.next("result")).call).toBe(id); + }; + const { name: type, version, fingerprint } = Retiring; + peer.send({ kind: "hold", hold: 0, type, version, fingerprint, params: "{}" }); + await peer.next("bound"); + peer.ack(); + return { peer, call }; +} + +forEachCodec(() => { + describe('retired members (decision "One retirement mechanism")', () => { + it("are listed in the state that carries the retirement, a derive whose value stayed without ops", async () => { + const { peer, call } = await retiring(); + await call("set", 1); + const set = await peer.next("state"); + expect(set.holds[0]!.retired).toEqual([]); + peer.ack(); + await call("retire"); + const retired = (await peer.next("state")).holds[0]!; + expect(RETIRING.named(retired.members)).toEqual({ doc: [["r", { n: 0 }]], t: [["r", 0]] }); + expect(retired.retired.toSorted()).toEqual(retiredAll); + }); + + it("are kept while the window is full and sent once with the folded ops", async () => { + const { peer, call } = await retiring(); + await call("set", 1); + await call("set", 2); + await peer.next("state"); + await peer.next("state"); + // The window is full: the retirement and the next change wait, folded into one entry. + await call("retire"); + await call("set", 4); + await transported(); + expect(peer.kinds().filter((kind) => kind === "state")).toEqual([]); + peer.ack(); + const pending = (await peer.next("state")).holds[0]!; + expect(RETIRING.named(pending.members)).toEqual({ + doc: [ + ["r", { n: 0 }], + ["s", ["n"], 4], + ], + t: [["r", 4]], + }); + expect(pending.retired.toSorted()).toEqual(retiredAll); + peer.ack(); + peer.ack(); + await call("set", 5); + expect((await peer.next("state")).holds[0]!.retired).toEqual([]); + }); + + it("of a member failed again before its entry leaves wait for its next value, never with its failure", async () => { + const { peer, call } = await retiring(); + await call("set", 1); + await call("set", 2); + await peer.next("state"); + await peer.next("state"); + // The window is full: `t` fails, heals with the retirement (n is 0 again), and fails again. + await call("set", -1); + await call("retire"); + await call("set", -3); + peer.ack(); + const failed = (await peer.next("state")).holds[0]!; + expect(failed.failed).toEqual([ + { member: RETIRING.state.t!, error: { kind: "plugin", code: "internal", message: "failed -3" } }, + ]); + expect(Object.keys(RETIRING.named(failed.members))).toEqual(["doc"]); + expect(failed.retired.toSorted()).toEqual([RETIRING.state.doc!, RETIRING.state.big!].sort()); + peer.ack(); + peer.ack(); + await call("set", 5); + const healed = (await peer.next("state")).holds[0]!; + expect(RETIRING.named(healed.members).t).toEqual([["r", 5]]); + expect(healed.retired).toEqual([RETIRING.state.t!]); + }); + }); + + describe("failures while the window is full", () => { + it("fold per member: 1000 failing commits leave one failure entry, the member's ops dropped; bounded", async () => { + const { peer, call } = await retiring(); + await call("set", 1); + await call("set", 2); + await peer.next("state"); + await peer.next("state"); + // An earlier episode, healed: its failure leaves with the heal. + await call("set", -1); + await call("set", 3); + for (let n = 1; n <= 1000; n++) await call("set", -1000 - n); + peer.ack(); + const pending = (await peer.next("state")).holds[0]!; + // One failure call per episode: the error of the commit that started it. + expect(pending.failed).toEqual([ + { member: RETIRING.state.t!, error: { kind: "plugin", code: "internal", message: "failed -1001" } }, + ]); + expect(Object.keys(RETIRING.named(pending.members))).not.toContain("t"); + expect(peer.codec.encode({ kind: "state", holds: [pending] }).byteLength).toBeLessThan(200); + }); + + it("are replaced by a value that follows them: the value as a root replacement, no failure", async () => { + const { peer, call } = await retiring(); + await call("set", 1); + await call("set", 2); + await peer.next("state"); + await peer.next("state"); + await call("set", -1); + await call("set", 3); + peer.ack(); + const pending = (await peer.next("state")).holds[0]!; + expect(pending.failed).toEqual([]); + expect(RETIRING.named(pending.members).t).toEqual([["r", 3]]); + }); + }); +}); diff --git a/packages/chord/test/next/remote/host-link.test.ts b/packages/chord/test/next/remote/host-link.test.ts new file mode 100644 index 000000000..7eb0a8551 --- /dev/null +++ b/packages/chord/test/next/remote/host-link.test.ts @@ -0,0 +1,429 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method, remote, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet, type FacetHost, type HostError } from "../../../src/next/host/index.ts"; +import { type Compression, memoryLinkPair, type PartLimits } from "../../../src/next/link/index.ts"; +import { jsonPart } from "../../../src/next/link/internal.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; +import { + addNote, + countInflated, + delivered, + EDITOR, + forEachCodec, + framesOf, + incarnationOf, + Notes, + PlacedNotes, + partValue, + RAW_PEER, + RawPeer, + rawWorkers, + rejection, + relayed, + Session, + serveNotes, + type TestPrincipal, + testCodec, + transported, +} from "../support.ts"; + +/** A server type that tells who holds it, and echoes as whom it is called. */ +const ServerWho = remote("app.server-who", { + version: 1, + params: {}, + variants: { v: { state: { who: str() }, methods: { echo: method(str(), str()) } } }, +}); + +/** A worker type that asks the server, from a handler (`ask`) and from its factory (`who`). */ +const Asker = remote("app.asker", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { v: { state: { who: str() }, methods: { ask: method(str(), str()) } } }, +}); + +/** A worker type that reads another session's notes through the server. */ +const Neighbour = remote("app.neighbour", { + version: 1, + params: { session: str(), other: str() }, + placement: ({ session }) => Session({ id: session }), + variants: { v: { state: { count: int() } } }, +}); + +function serveWho(h: FacetHost): void { + h.serve(ServerWho, (_, principal) => ({ + variant: "v", + members: { who: constant(principal.kind), echo: async (text, ctx) => `${ctx.principal?.kind}:${text}` }, + })); +} + +function serveAsker(h: FacetHost): void { + h.serve(Asker, (_, _principal, self) => { + const server = self.get(ServerWho, {}); + return { + variant: "v", + members: { + who: server.who, + ask: async (text, ctx) => (await ctx.get(ServerWho, {})).echo(text, ctx), + }, + }; + }); + h.serve(Neighbour, ({ other }, _principal, self) => { + const notes = self.get(PlacedNotes, { session: other }); + return { variant: "v", members: { count: notes.count } }; + }); +} + +forEachCodec(() => { + describe("two-sided links", () => { + it("carry the server's holds of a worker type and the worker's holds of a server type over one route link", async () => { + const { tab, routeLinks } = await relayed({ server: serveWho, worker: serveAsker }); + const asker = await tab.get(Asker, { session: "S1" }, context); + // The worker's factory held `ServerWho` over its parent link, the route link the server made. + expect(asker.who.value).toBe("worker"); + expect(routeLinks).toHaveLength(1); + }); + + it("carry calls both ways with equal ids: each side's ids are its own", async () => { + const { tab } = await relayed({ server: serveWho, worker: serveAsker }); + const asker = await tab.get(Asker, { session: "S1" }, context); + // Server to worker: call 0 on the route link; inside it, worker to server: call 0 the other way. + expect(await asker.ask("x", context)).toBe("worker:x"); + expect(await Promise.all([asker.ask("a", context), asker.ask("b", context)])).toEqual([ + "worker:a", + "worker:b", + ]); + }); + + it("let a worker reach another worker through the server", async () => { + const { tab, workers, resolves } = await relayed({ worker: serveAsker }); + const neighbour = await tab.get(Neighbour, { session: "S1", other: "S2" }, context); + expect(resolves).toEqual(["session|S1", "session|S2"]); + await addNote(workers.get("session|S2")!, "S2", "a"); + await transported(); + expect(neighbour.count.value).toBe(1); + }); + + it("apply this host's own messages and forward the tab origin's, each acknowledged by its consumer", async () => { + const peers: RawPeer[] = []; + const { tab, server } = await relayed({ resolve: rawWorkers(peers) }); + const own = server.get(PlacedNotes, { session: "S1" }, context); + await transported(); + const [worker] = peers; + const first = await worker!.next("hold"); + bindPlaced(worker!, first); + const relayedHold = tab.get(PlacedNotes, { session: "S1" }, context); + const second = await worker!.next("hold"); + expect([first.origin, second.origin]).toEqual([undefined, 1]); + bindPlaced(worker!, second); + const [serverNotes, tabs] = [await own, await relayedHold]; + await transported(); + worker!.received.length = 0; + const ops = jsonPart([["r", 5]]); + for (const { hold, origin } of [first, second]) { + const from = origin === undefined ? {} : { origin }; + const members = [{ member: EDITOR.state.count!, ops }]; + worker!.send({ kind: "state", ...from, holds: [{ hold, members, failed: [], retired: [] }] }); + } + await transported(); + expect(serverNotes.count.value).toBe(5); + expect(tabs.count.value).toBe(5); + expect(worker!.received).toEqual([{ kind: "ack" }, { kind: "ack", origin: 1 }]); + }); + }); + + describe("principals on holds (section 4's table)", () => { + /** A server serving a guest-facing gateway and a local view, both reaching `PlacedNotes` on a raw worker. */ + async function principals(tab: TestPrincipal = { kind: "guest", session: "S1" }) { + const peers: RawPeer[] = []; + const Gateway = remote("app.gateway", { + version: 1, + params: {}, + variants: { v: { state: { n: int() }, methods: { open: method(str(), int()) } } }, + }); + // With params: not activated by `load`, which would wait for the raw worker. + const View = local("app.view", { version: 1, params: { id: int() }, members: { n: int() } }); + const relay = await relayed({ + tab, + resolve: rawWorkers(peers), + server: (h) => { + h.serve(Gateway, (_, _principal, self) => { + // `self.get` in a remote object: the object's principal, this host's incarnation. + const own = self.get(PlacedNotes, { session: "own" }); + return { + variant: "v", + members: { + n: own.count, + // `ctx.get` in a handler: the caller's principal and incarnation. + open: async (session, ctx) => (await ctx.get(PlacedNotes, { session })).count.value, + }, + }; + }); + h.serve(View, (_, self) => ({ n: self.get(PlacedNotes, { session: "view" }).count })); + }, + }); + return { ...relay, peers, Gateway, View }; + } + + /** The next hold the raw worker of `peers[index]` receives, bound at once, with whom it acts for. */ + async function nextHold(peers: RawPeer[], index = 0) { + for (let tries = 0; peers[index] === undefined && tries < 10; tries++) await transported(); + const hold = await peers[index]!.next("hold"); + bindPlaced(peers[index]!, hold); + const { session } = partValue(hold.params) as { session: string }; + return { session, principal: hold.principal, incarnation: hold.incarnation }; + } + + it("a relayed hold acts as the link's principal: a forged principal from a tab is ignored (race 9)", async () => { + const { server, peers } = await principals(); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "guest", session: "S1" }, codec: testCodec() }, context); + const tab = new RawPeer(b); + tab.send({ + kind: "hold", + hold: 0, + type: PlacedNotes.name, + version: PlacedNotes.version, + fingerprint: PlacedNotes.fingerprint, + params: jsonPart({ session: "S1" }), + address: { kind: "session", params: { id: "S1" } }, + principal: { kind: "owner" }, + }); + expect(await nextHold(peers)).toEqual({ + session: "S1", + principal: { kind: "guest", session: "S1" }, + incarnation: RAW_PEER, + }); + }); + + it("a hold relayed over a parent path names nobody: a child's principal and incarnation stay behind", async () => { + const worker = createHost({ principal: { kind: "worker", session: "S1" }, onError: () => {} }); + await worker.load([], context); + const [up, parentEnd] = memoryLinkPair(); + worker.connect(up, { principal: { kind: "server" }, codec: testCodec() }, context); + const parent = new RawPeer(parentEnd); + const [a, b] = memoryLinkPair(); + worker.accept(a, { principal: { kind: "guest", session: "S1" }, codec: testCodec() }, context); + const child = new RawPeer(b); + child.send({ + kind: "hold", + hold: 0, + type: PlacedNotes.name, + version: PlacedNotes.version, + fingerprint: PlacedNotes.fingerprint, + params: jsonPart({ session: "S1" }), + address: { kind: "session", params: { id: "S1" } }, + principal: { kind: "owner" }, + incarnation: RAW_PEER, + }); + const relayedHold = await parent.next("hold"); + expect(relayedHold).toMatchObject({ hold: 0, origin: 1, type: PlacedNotes.name }); + expect("principal" in relayedHold).toBe(false); + expect("incarnation" in relayedHold).toBe(false); + await worker.close(context); + }); + + it("host.get through a route acts as the host: its principal and incarnation", async () => { + const { server, peers } = await principals(); + void server.get(PlacedNotes, { session: "S1" }, context); + expect(await nextHold(peers)).toEqual({ + session: "S1", + principal: { kind: "server" }, + incarnation: incarnationOf(server), + }); + }); + + it("ctx.get in a guest's call acts as the guest, with the tab's incarnation (race 10)", async () => { + const { tab, peers, Gateway } = await principals(); + const got = tab.get(Gateway, {}, context); + expect(await nextHold(peers)).toMatchObject({ session: "own" }); + const opened = (await got).open("S1", context); + // Another address: another worker. + expect(await nextHold(peers, 1)).toEqual({ + session: "S1", + principal: { kind: "guest", session: "S1" }, + incarnation: incarnationOf(tab), + }); + expect(await opened).toBe(0); + }); + + it("self.get in a remote object acts as that object's principal, with this host's incarnation", async () => { + const { tab, server, peers, Gateway } = await principals(); + tab.hold(Gateway, {}, context); + expect(await nextHold(peers)).toEqual({ + session: "own", + principal: { kind: "guest", session: "S1" }, + incarnation: incarnationOf(server), + }); + }); + + it("self.get in a local object acts as the host", async () => { + const { server, peers, View } = await principals(); + void server.get(View, { id: 1 }, context); + expect(await nextHold(peers)).toEqual({ + session: "view", + principal: { kind: "server" }, + incarnation: incarnationOf(server), + }); + }); + + it("two origins with equal principal keys but other incarnations are two upstream holds", async () => { + const { tab, server, peers } = await principals({ kind: "owner" }); + const other = createHost({ onError: () => {} }); + await other.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + other.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); + void tab.get(PlacedNotes, { session: "S1" }, context); + const first = await nextHold(peers); + void other.get(PlacedNotes, { session: "S1" }, context); + const second = await nextHold(peers); + expect([first, second]).toEqual([ + { session: "S1", principal: { kind: "owner" }, incarnation: incarnationOf(tab) }, + { session: "S1", principal: { kind: "owner" }, incarnation: incarnationOf(other) }, + ]); + }); + + it("upward is a boundary: a worker's hold of a server type acts as the worker, whoever it serves (race 11)", async () => { + const { tab } = await relayed({ tab: { kind: "guest", session: "S1" }, server: serveWho, worker: serveAsker }); + const asker = await tab.get(Asker, { session: "S1" }, context); + expect(await asker.ask("x", context)).toBe("worker:x"); + }); + }); + + describe("trusted links", () => { + it("work as untrusted ones do", async () => { + const { tab } = await relayed({ trusted: true }); + expect((await tab.get(PlacedNotes, { session: "S1" }, context)).variant).toBe("editor"); + }); + }); +}); + +/** Binds a hold a raw worker received (its id and origin): `PlacedNotes`'s editor variant, no notes. */ +function bindPlaced(peer: RawPeer, { hold, origin }: { readonly hold: number; readonly origin?: number }): void { + const members = [ + { member: EDITOR.state.notes!, value: "[]" }, + { member: EDITOR.state.count!, value: "0" }, + ]; + const from = origin === undefined ? {} : { origin }; + peer.send({ kind: "bound", ...from, holds: [{ hold, variant: EDITOR.variant, members, failed: [] }] }); +} + +/** A compressor that counts what it does (`nodeCompression` underneath). */ +function counting(): Compression & { deflates: number; inflates: number } { + const counts = { + deflates: 0, + inflates: 0, + deflate: (bytes: Uint8Array) => { + counts.deflates++; + return nodeCompression.deflate(bytes); + }, + inflate: (bytes: Uint8Array, limit: number) => { + counts.inflates++; + return nodeCompression.inflate(bytes, limit); + }, + }; + return counts; +} + +/** A note text large enough that a notes list with it is deflated at its origin. */ +const LARGE = "lorem ipsum ".repeat(2000); + +/** + * A server serving `Notes` with `compression`, and a client with its own `compression` (none: it inflates + * asynchronously), its parent link's limits `limits`. + */ +async function compressing(options: { + readonly server?: Compression; + readonly client?: Compression; + readonly limits?: PartLimits; +}) { + const errors: HostError[] = []; + const server = createHost({ + principal: { kind: "server" }, + compression: options.server, + onError: (error) => errors.push(error), + }); + await server.load([{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], context); + const client = createHost({ compression: options.client, onError: (error) => errors.push(error) }); + await client.load([], context); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + client.connect(b, { principal: { kind: "server" }, codec: testCodec(), limits: options.limits }, context); + return { server, client, errors }; +} + +forEachCodec(() => { + describe("parts: compression at the origin and limits", () => { + it("deflate JSON parts above 16 KiB at their origin and inflate them where owned, synchronously; never up a parent link's arguments", async () => { + const [server, client] = [counting(), counting()]; + const hosts = await compressing({ server, client }); + await addNote(hosts.server, "S1", LARGE); + const notes = await hosts.client.get(Notes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("an owner's notes are the editor's"); + expect(notes.notes.value).toEqual([{ text: LARGE }]); + expect([server.deflates, client.inflates]).toEqual([1, 1]); + // A large argument goes up the client's parent link uncompressed: child links refuse compressed parts. + await notes.add({ text: LARGE }, context); + expect([client.deflates, server.inflates]).toEqual([0, 0]); + await transported(); + expect(notes.notes.value).toEqual([{ text: LARGE }, { text: LARGE }]); + }); + + it("inflate asynchronously without a compressor: later messages of the link wait, in order", async () => { + const hosts = await compressing({ server: nodeCompression }); + await addNote(hosts.server, "S1", LARGE); + const notes = await hosts.client.get(Notes, { session: "S1" }, context); + expect(notes.notes.value).toEqual([{ text: LARGE }]); + const counts = framesOf(await notes.count.watch(context)); + // Each commit sends the whole changed list (a derive's diff appends one note): large ones deflated. + for (let i = 0; i < 5; i++) await addNote(hosts.server, "S1", i % 2 === 0 ? LARGE : "small"); + for (let i = 0; i < 100 && notes.count.value < 6; i++) await new Promise((resolve) => setTimeout(resolve, 5)); + expect(notes.count.value).toBe(6); + expect(notes.notes.value.map(({ text }) => text.length)).toEqual( + [LARGE, LARGE, "small", LARGE, "small", LARGE].map((text) => text.length), + ); + await delivered(); + const seen = counts.map(([count]) => count); + expect(seen).toEqual([...seen].sort()); + }); + + it("fail only the hold of a compressed bomb, on either inflate path (race 11): the link stays", async () => { + for (const client of [nodeCompression, undefined]) { + const hosts = await compressing({ server: nodeCompression, client, limits: { decoded: 100_000 } }); + await addNote(hosts.server, "S1", "x".repeat(200_000)); + await addNote(hosts.server, "S2", "small"); + const error = await rejection(hosts.client.get(Notes, { session: "S1" }, context)); + expect(error).toMatchObject({ code: "too_large", reason: "part" }); + const other = await hosts.client.get(Notes, { session: "S2" }, context); + expect(other.notes.value).toEqual([{ text: "small" }]); + } + }); + + it("cost nothing for uncompressed messages at a host without a compressor: no inflate, no stream", async () => { + const inflated = countInflated(); + try { + const hosts = await compressing({ server: nodeCompression }); + const notes = await hosts.client.get(Notes, { session: "S1" }, context); + for (let i = 0; i < 5; i++) await addNote(hosts.server, "S1", "small"); + await transported(); + expect(notes.count.value).toBe(5); + expect(inflated.bytes).toBe(0); + expect(hosts.errors).toEqual([]); + } finally { + inflated.restore(); + } + }); + + it("send every part uncompressed from a host without a compressor", async () => { + const server = counting(); + const hosts = await compressing({ server }); + const notes = await hosts.client.get(Notes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("an owner's notes are the editor's"); + await notes.add({ text: LARGE }, context); + expect(server.inflates).toBe(0); + }); + }); +}); diff --git a/packages/chord/test/next/remote/path.test.ts b/packages/chord/test/next/remote/path.test.ts new file mode 100644 index 000000000..c3efb197d --- /dev/null +++ b/packages/chord/test/next/remote/path.test.ts @@ -0,0 +1,70 @@ +import { describe, expect, it } from "vitest"; +import type { HostLink } from "../../../src/next/remote/host-link.ts"; +import { Path, type Upstream } from "../../../src/next/remote/path.ts"; + +/** A holder recording where it was held and what it was told, as `name@link` and `name dropped`. */ +function holder(name: string, log: string[]): Upstream { + // The path calls only `attach` and `dropped`. + return { + attach: (link: { readonly name: string }) => void log.push(`${name}@${link.name}`), + dropped: () => void log.push(`${name} dropped`), + } as unknown as Upstream; +} + +/** A link known by its name; the path only passes it on. */ +function linkOf(name: string): HostLink { + return { name } as unknown as HostLink; +} + +describe("a path", () => { + it("asks for a link when it gets a holder without one; holds every holder on each link it gets", () => { + const log: string[] = []; + let changes = 0; + const path = new Path(() => changes++); + path.attach(holder("a", log)); + path.attach(holder("b", log)); + expect(changes).toBe(2); + expect(log).toEqual([]); + const first = linkOf("1"); + path.connected(first); + expect(log).toEqual(["a@1", "b@1"]); + path.attach(holder("c", log)); + expect(log.at(-1)).toBe("c@1"); + expect(changes).toBe(2); + }); + + it("tells every holder when its link drops, then asks for the next; a stale link's drop is ignored", () => { + const log: string[] = []; + let changes = 0; + const path = new Path(() => changes++); + const [first, second] = [linkOf("1"), linkOf("2")]; + path.connected(first); + path.attach(holder("a", log)); + path.disconnected(first); + expect(log).toEqual(["a@1", "a dropped"]); + expect(changes).toBe(1); + expect(path.link).toBeUndefined(); + path.connected(second); + path.disconnected(first); + expect(log).toEqual(["a@1", "a dropped", "a@2"]); + expect(changes).toBe(1); + expect(path.link).toBe(second); + }); + + it("reports its last holder's detach: no later link holds it", () => { + const log: string[] = []; + let changes = 0; + const path = new Path(() => changes++); + const [a, b] = [holder("a", log), holder("b", log)]; + path.connected(linkOf("1")); + path.attach(a); + path.attach(b); + path.detach(a); + expect(changes).toBe(0); + path.detach(b); + expect(changes).toBe(1); + expect(path.held).toBe(false); + path.connected(linkOf("2")); + expect(log).toEqual(["a@1", "b@1"]); + }); +}); diff --git a/packages/chord/test/next/remote/provider.test.ts b/packages/chord/test/next/remote/provider.test.ts new file mode 100644 index 000000000..b1be51b9e --- /dev/null +++ b/packages/chord/test/next/remote/provider.test.ts @@ -0,0 +1,974 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { PendingOps } from "../../../src/delta/index.ts"; +import { type Contract, int, local, map, method, obj, remote } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { createHost, defineFacet, type FacetHost } from "../../../src/next/host/index.ts"; +import { type Compression, memoryLinkPair, type PartLimits } from "../../../src/next/link/index.ts"; +import { DeflatedPart, jsonPart, type Message } from "../../../src/next/link/internal.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; +import { + addNote, + countInflated, + EDITOR, + expectCollected, + forEachCodec, + metered, + Notes, + type NotesProbe, + partValue, + RAW_PEER, + RawPeer, + rawLinked, + serveNotes, + type TestPrincipal, + testCodec, + transported, + wireNames, +} from "../support.ts"; + +/** Sends a hold of `Type` (its own version and fingerprint unless overridden), of `origin` when given. */ +function hold( + peer: RawPeer, + id: number, + Type: Contract, + params: unknown, + override: Partial = {}, + origin?: number, +) { + const { name: type, version, fingerprint } = { ...Type, ...override }; + const from = origin === undefined ? {} : { origin }; + peer.send({ kind: "hold", ...from, hold: id, type, version, fingerprint, params: jsonPart(params) }); +} + +/** Sends a call of the method at index `method` of the bound variant, with `args` (none when left out). */ +function call(peer: RawPeer, id: number, holdId: number, method: number, args?: unknown, origin?: number) { + const from = origin === undefined ? {} : { origin }; + const payload = args === undefined ? {} : { args: jsonPart(args) }; + peer.send({ kind: "call", ...from, call: id, hold: holdId, method, ...payload }); +} + +/** A remote type kept per id. */ +const Kept = remote("app.kept", { version: 1, params: { id: int() }, variants: { v: { state: { n: int() } } } }); + +/** The wire's indexes of `Notes`'s reader variant. */ +const READER = wireNames(Notes, "reader"); + +const { add, fail, wait } = EDITOR.methods as { add: number; fail: number; wait: number }; + +/** A server serving `Notes` and a raw consumer peer stamped `child`. */ +async function notesServer(child: TestPrincipal = { kind: "owner" }) { + let probe: NotesProbe | undefined; + const linked = await rawLinked( + (h) => { + probe = serveNotes(h); + }, + { child }, + ); + return { ...linked, probe: probe! }; +} + +forEachCodec(() => { + describe("holds arriving on a link", () => { + it("bind as the link's principal: bound with the variant and the snapshot", async () => { + const { peer, server } = await notesServer({ kind: "guest", session: "S1" }); + await addNote(server, "S1", "a"); + hold(peer, 0, Notes, { session: "S1" }); + const [bound] = (await peer.next("bound")).holds; + expect(bound).toMatchObject({ hold: 0, variant: READER.variant, failed: [] }); + expect(READER.named(bound!.members)).toEqual({ notes: [{ text: "a" }], count: 1 }); + }); + + it.each([ + [ + "a type not served here", + { type: "app.nope" }, + { code: "unavailable", message: "app.nope is not served here" }, + ], + [ + "a local type", + { type: "app.note-store" }, + { code: "unavailable", message: "app.note-store is not served here" }, + ], + [ + "another version", + { version: 2 }, + { code: "contract_mismatch", message: "app.notes v2 does not match the served v1" }, + ], + [ + "another fingerprint", + { fingerprint: "0000000000000000" }, + { code: "contract_mismatch", message: "app.notes v1 does not match the served v1" }, + ], + ])("refuse %s with a broken failed", async (_, override, error) => { + const { peer } = await notesServer(); + const { type, ...rest } = override as { type?: string }; + hold(peer, 0, { ...Notes, name: type ?? Notes.name } as Contract, { session: "S1" }, rest); + expect(await peer.next("failed")).toMatchObject({ + hold: 0, + cause: { kind: "broken", error: { kind: "chord", error } }, + }); + }); + + it("refuse invalid params with a broken failed", async () => { + const { peer } = await notesServer(); + hold(peer, 0, Notes, { session: 1 }); + expect(await peer.next("failed")).toMatchObject({ + cause: { kind: "broken", error: { kind: "chord", error: { code: "invalid", reason: "params" } } }, + }); + }); + + it("arriving before load resolved wait for it", async () => { + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + hold(peer, 0, Notes, { session: "S1" }); + await transported(); + expect(peer.kinds()).toEqual(["hello"]); + await server.load( + [{ name: "@test/app", facet: defineFacet((h) => void serveNotes(h)) }], + context, + ); + expect((await peer.next("bound")).holds[0]).toMatchObject({ hold: 0, variant: EDITOR.variant }); + }); + }); + + describe("calls arriving on a link", () => { + it("run as the link's principal with the child's incarnation; results and errors go back", async () => { + const { peer, probe } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + call(peer, 0, 0, add, { text: "x" }); + expect(partValue((await peer.next("result")).value!)).toBe(1); + call(peer, 1, 0, fail, { how: "plugin" }); + expect((await peer.next("error")).error).toEqual({ + kind: "plugin", + code: "quota", + message: "too many", + details: { limit: 1 }, + }); + call(peer, 2, 0, wait); + await transported(); + expect(probe.waits[0]).toMatchObject({ principal: { kind: "owner" }, incarnation: RAW_PEER }); + }); + + it("outside the binding's variant cannot be named: an index past its methods, forged at the wire, closes the link", async () => { + const { peer, errors } = await notesServer({ kind: "guest", session: "S1" }); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + // The reader variant has one method, `wait`; the editor's `add` has no index in it. + call(peer, 0, 0, Object.keys(READER.methods).length, { text: "x" }); + await transported(); + expect(peer.closed).toBe(true); + expect(errors).toMatchObject([{ kind: "protocol_error", error: { message: "hold 0: no method 1" } }]); + }); + + it("check their arguments", async () => { + const { peer } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + call(peer, 0, 0, add, { text: "x", extra: 1 }); + expect((await peer.next("error")).error).toMatchObject({ + kind: "chord", + error: { code: "invalid", reason: "args" }, + }); + }); + + it("before bound are refused not_ready (race 3)", async () => { + const Slow = remote("app.slow", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const { peer } = await rawLinked((h) => + h.serve(Slow, (_, _principal, self) => { + self.open( + () => new Promise(() => {}), + () => {}, + ); + return { variant: "v", members: { n: constant(1) } }; + }), + ); + hold(peer, 0, Slow, {}); + call(peer, 0, 0, 0); + expect((await peer.next("error")).error).toMatchObject({ kind: "chord", error: { code: "not_ready" } }); + }); + + it("cancelled are aborted and get no reply; a result crossing the cancel is never sent (race 1)", async () => { + const { peer, probe } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + call(peer, 0, 0, wait); + await transported(); + peer.send({ kind: "abort", call: 0 }); + await transported(); + expect(probe.waits[0]!.abortSignal?.aborted).toBe(true); + expect(peer.kinds()).toEqual(["hello"]); + }); + }); + + describe("ids", () => { + it("a link ignores ids it does not know and messages for the other direction", async () => { + const { peer, errors } = await notesServer(); + peer.send({ kind: "release", hold: 9 }); + peer.send({ kind: "abort", call: 9 }); + call(peer, 0, 9, add, { text: "x" }); + peer.send({ kind: "gone", hold: 9, error: { kind: "chord", error: { code: "released", message: "m" } } }); + await transported(); + expect(peer.closed).toBe(false); + expect(peer.kinds()).toEqual(["hello"]); + expect(errors).toEqual([]); + }); + + it("a hold reusing a live id closes the link", async () => { + const { peer, errors } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + hold(peer, 0, Notes, { session: "S1" }); + await transported(); + expect(peer.closed).toBe(true); + expect(errors).toMatchObject([{ kind: "protocol_error", error: { message: "hold 0: the id is in use" } }]); + }); + + it("a call reusing a live id closes the link", async () => { + const { peer } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + call(peer, 0, 0, wait); + call(peer, 0, 0, wait); + await transported(); + expect(peer.closed).toBe(true); + }); + + it("a message the codec cannot decode closes the link", async () => { + const { peer } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + peer.sendBytes(new Uint8Array([0xff, 0xff])); + await transported(); + expect(peer.closed).toBe(true); + }); + }); + + describe("one message per link per commit (W1)", () => { + it("three holds on one link: one state message, one part per hold, one encode each", async () => { + const { peer, server } = await notesServer(); + for (const id of [0, 1, 2]) hold(peer, id, Notes, { session: "S1" }); + for (const _ of [0, 1, 2]) await peer.next("bound"); + const { context: counting, counts } = await metered(); + const store = await server.get(Notes, { session: "S1" }, context); + if (store.variant === "editor") await store.add({ text: "a" }, counting); + await transported(); + const states = peer.received.filter((message) => message.kind === "state"); + expect(states).toHaveLength(1); + expect(states[0]).toMatchObject({ holds: [{ hold: 0 }, { hold: 1 }, { hold: 2 }] }); + expect(counts()["chord.link.encodes"]).toBe(3); + // No validation or walk of state on the provider: only the argument checks of `add` and the store's `add`. + expect(counts()["chord.data.validations"]).toBe(2); + expect(counts()["chord.data.walks"]).toBeUndefined(); + }); + + it("a derive travels with its source in the same part", async () => { + const { peer, server } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + await addNote(server, "S1", "a"); + const state = await peer.next("state"); + expect(Object.keys(EDITOR.named(state.holds[0]!.members)).sort()).toEqual(["count", "notes"]); + }); + + it("two links: one message each; when one closes, the other keeps updating (race 8)", async () => { + const { peer, server } = await notesServer(); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const other = new RawPeer(b); + hold(peer, 0, Notes, { session: "S1" }); + hold(other, 0, Notes, { session: "S1" }); + await peer.next("bound"); + await other.next("bound"); + await addNote(server, "S1", "a"); + await peer.next("state"); + await other.next("state"); + peer.close(); + await transported(); + await addNote(server, "S1", "b"); + const next = await other.next("state"); + expect(Object.keys(EDITOR.named(next.holds[0]!.members)).sort()).toEqual(["count", "notes"]); + }); + }); + + describe("races at the provider", () => { + /** Serves `Kept`, recording each factory run and each native close. */ + function serveKept(h: Parameters[0]>[0], events: string[]) { + h.serve(Kept, ({ id }, _principal, self) => { + events.push(`factory ${id}`); + self.open( + async () => null, + (_value, reason) => void events.push(`close ${reason.kind}`), + ); + return { variant: "v", members: { n: constant(id) } }; + }); + } + + it("a new hold and the old one's release in one turn hand the object over without ending it (R18)", async () => { + const events: string[] = []; + const { peer } = await rawLinked((h) => serveKept(h, events)); + hold(peer, 0, Kept, { id: 1 }); + await peer.next("bound"); + hold(peer, 1, Kept, { id: 1 }); + peer.send({ kind: "release", hold: 0 }); + expect((await peer.next("bound")).holds[0]).toMatchObject({ hold: 1 }); + expect(events).toEqual(["factory 1"]); + }); + + it("a hold released right after it was sent keeps no object", async () => { + const events: string[] = []; + const { peer } = await rawLinked((h) => serveKept(h, events)); + hold(peer, 0, Kept, { id: 1 }); + await Promise.resolve(); + peer.send({ kind: "release", hold: 0 }); + await transported(); + // The object ended: a new hold makes a new one. + hold(peer, 1, Kept, { id: 1 }); + await peer.next("bound"); + expect(events.filter((event) => event.startsWith("factory"))).toEqual(["factory 1", "factory 1"]); + }); + + it("a hold released while it waits for load takes nothing once load resolves", async () => { + const events: string[] = []; + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + await transported(); + hold(peer, 0, Kept, { id: 1 }); + await Promise.resolve(); + peer.send({ kind: "release", hold: 0 }); + await transported(); + await server.load( + [{ name: "@test/app", facet: defineFacet((h) => serveKept(h, events)) }], + context, + ); + await transported(); + expect(events).toEqual([]); + }); + + it("a release while activating unholds; bound is never sent (race 6)", async () => { + const Slow = remote("app.slow", { + version: 1, + params: { id: int() }, + variants: { v: { state: { n: int() } } }, + }); + const aborted: string[] = []; + const { peer } = await rawLinked((h) => + h.serve(Slow, ({ id }, _principal, self) => { + self.open( + (ctx) => + new Promise((resolve) => + ctx.abortSignal?.addEventListener("abort", () => resolve(aborted.push(`${id}`))), + ), + () => {}, + ); + return { variant: "v", members: { n: constant(id) } }; + }), + ); + hold(peer, 0, Slow, { id: 1 }); + await transported(); + peer.send({ kind: "release", hold: 0 }); + await transported(); + expect(aborted).toEqual(["1"]); + expect(peer.kinds()).toEqual(["hello"]); + }); + + it("a hold while the previous instance of its identity tears down waits for it (race 7)", async () => { + const Closing = remote("app.closing", { + version: 1, + params: { id: int() }, + variants: { v: { state: { n: int() } } }, + }); + const events: string[] = []; + let finish: () => void = () => {}; + const { peer } = await rawLinked((h) => + h.serve(Closing, ({ id }, _principal, self) => { + self.open( + async () => void events.push("open"), + () => + new Promise((resolve) => { + events.push("closing"); + finish = () => { + events.push("closed"); + resolve(); + }; + }), + ); + return { variant: "v", members: { n: constant(id) } }; + }), + ); + hold(peer, 0, Closing, { id: 1 }); + await peer.next("bound"); + peer.send({ kind: "release", hold: 0 }); + hold(peer, 1, Closing, { id: 1 }); + await transported(); + expect(events).toEqual(["open", "closing"]); + finish(); + await peer.next("bound"); + expect(events).toEqual(["open", "closing", "closed", "open"]); + }); + + it("a commit right after the binding continues exactly from the snapshot (race 2)", async () => { + const { peer, server } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + const adding = addNote(server, "S1", "a"); + const [bound] = (await peer.next("bound")).holds; + await adding; + await transported(); + const snapshot = EDITOR.named(bound!.members) as { notes: unknown[] }; + const states = peer.received.filter((message) => message.kind === "state"); + const added = states.flatMap((message) => (EDITOR.named(message.holds[0]!.members).notes ?? []) as unknown[]); + expect(snapshot.notes.length + added.length).toBe(1); + }); + + it("closing a link during a batch drops its state message and releases its holds (race 9)", async () => { + const Counter = remote("app.counter", { + version: 1, + params: { id: int() }, + variants: { v: { state: { n: int(), seen: int() }, methods: { bump: method() } } }, + }); + const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + // Called inside the batch, from a derive the hold follows. + let during: () => void = () => {}; + const ends: string[] = []; + const { server, errors } = await rawLinked((h) => + h.serve(Counter, ({ id }, _principal, self) => { + const n = self.state(N); + self.open( + async () => null, + () => void ends.push(`counter ${id} ended`), + ); + return { + variant: "v", + members: { + n: self.derive([n], ([{ n }]) => n), + seen: self.derive([n], ([{ n }]) => { + if (n > 0) during(); + return n; + }), + bump: async (ctx) => { + await n.change((draft) => void draft.n++, ctx); + return null; + }, + }, + }; + }), + ); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + hold(peer, 0, Counter, { id: 1 }); + await peer.next("bound"); + during = () => a.close(); + call(peer, 0, 0, 0); + await transported(); + expect(peer.kinds()).toEqual(["hello"]); + expect(ends).toEqual(["counter 1 ended"]); + expect(errors).toEqual([]); + }); + }); + + describe("local types over a link", () => { + it("never cross one", async () => { + const Store = local("app.store", { version: 1, params: {}, members: { n: int() } }); + const { peer } = await rawLinked((h) => h.serve(Store, () => ({ n: constant(1) }))); + hold(peer, 0, Store, {}); + expect(await peer.next("failed")).toMatchObject({ + cause: { kind: "broken", error: { kind: "chord", error: { code: "unavailable" } } }, + }); + }); + }); +}); + +/** A store whose `fill` sets `cells[0..n)` to a value in one commit: `n` leaf ops on distinct paths. */ +const Grid = remote("app.grid", { + version: 1, + params: {}, + variants: { + v: { + state: { grid: obj({ cells: map(int()) }), one: int() }, + methods: { fill: method(obj({ n: int(), value: int() })) }, + }, + }, +}); + +const GridDoc = defineState("grid", { schema: obj({ cells: map(int()) }), version: 1, initial: () => ({ cells: {} }) }); + +function serveGrid(h: FacetHost): void { + h.serve(Grid, (_, _principal, self) => { + const grid = self.state(GridDoc); + return { + variant: "v", + members: { + grid, + one: constant(1), + fill: async ({ n, value }, ctx) => { + await grid.change((draft) => { + for (let cell = 0; cell < n; cell++) draft.cells[`c${cell}`] = value; + }, ctx); + return null; + }, + }, + }; + }); +} + +/** A server serving `Notes` and `Grid` to a raw peer that acks only when told, the link's context counting. */ +async function slowConsumer() { + const counting = await metered(); + const server = createHost({ principal: { kind: "server" }, onError: () => {} }); + await server.load( + [ + { + name: "@test/app", + facet: defineFacet((h) => { + serveNotes(h); + serveGrid(h); + }), + }, + ], + context, + ); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, counting.context); + const peer = new RawPeer(b, { ack: false }); + await transported(); + return { server, peer, counts: counting.counts }; +} + +/** The `state` messages received and not taken. */ +function statesOf(peer: RawPeer): Extract[] { + return peer.received.filter((message) => message.kind === "state"); +} + +/** The `bound` and `state` messages received and not taken, by kind. */ +function windowed(peer: RawPeer): string[] { + return peer.kinds().filter((kind) => kind === "bound" || kind === "state"); +} + +const GRID = wireNames(Grid, "v"); + +forEachCodec(() => { + describe("the window at the provider", () => { + it("holds at most two unacknowledged bound and state messages; what waits goes out folded, encoded once", async () => { + const { server, peer, counts } = await slowConsumer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + for (const text of ["a", "b", "c", "d"]) await addNote(server, "S1", text); + await transported(); + expect(windowed(peer)).toEqual(["state"]); + const before = counts()["chord.link.encodes"] ?? 0; + peer.ack(); + await transported(); + const [, pending] = peer.received.filter(({ kind }) => kind === "state") as Extract< + Message, + { kind: "state" } + >[]; + // Three commits waited: one message, one entry, `count`'s three replacements folded into the last. + expect(EDITOR.named(pending!.holds[0]!.members).count).toEqual([["r", 4]]); + expect(counts()["chord.link.encodes"]! - before).toBe(1); + expect(windowed(peer)).toEqual(["state", "state"]); + await addNote(server, "S1", "e"); + await transported(); + expect(windowed(peer)).toEqual(["state", "state"]); + }); + + it("ignores acks that cross nothing: they never open the window past two (race 18)", async () => { + const { server, peer } = await slowConsumer(); + for (const _ of [0, 1, 2]) peer.ack(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + for (const text of ["a", "b", "c"]) await addNote(server, "S1", text); + await transported(); + expect(windowed(peer)).toEqual(["state"]); + }); + + it("lets a hold that binds while the window is full wait: the waiting state first, its bound at the next slot", async () => { + const { server, peer } = await slowConsumer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + await addNote(server, "S1", "a"); + await addNote(server, "S1", "b"); + hold(peer, 1, Notes, { session: "S1" }); + await transported(); + expect(windowed(peer)).toEqual(["state"]); + peer.ack(); + await transported(); + expect(windowed(peer)).toEqual(["state", "state"]); + await addNote(server, "S1", "c"); + peer.ack(); + await transported(); + // Hold 1's snapshot is taken now, with every commit so far. + const bound = peer.received.findLast(({ kind }) => kind === "bound") as Extract; + expect(bound.holds.map(({ hold }) => hold)).toEqual([1]); + expect(EDITOR.named(bound.holds[0]!.members).count).toBe(3); + }); + + it("overflows past 10,000 pending ops to a replacement with the current value (race 23)", async () => { + const { peer } = await slowConsumer(); + hold(peer, 0, Grid, {}); + await peer.next("bound"); + const fill = (id: number, value: number) => + peer.send({ + kind: "call", + call: id, + hold: 0, + method: GRID.methods.fill!, + args: jsonPart({ n: 6_000, value }), + }); + fill(0, 1); + fill(1, 2); + fill(2, 3); + for (const _ of [0, 1, 2]) await peer.next("result"); + // The first fill took the second slot; the next two overflowed the queue. + peer.ack(); + await transported(); + const states = peer.received.filter(({ kind }) => kind === "state") as Extract[]; + const last = GRID.named(states.at(-1)!.holds[0]!.members).grid as [string, { cells: object }][]; + expect(last[0]![0]).toBe("r"); + expect(Object.keys(last[0]![1].cells)).toHaveLength(6_000); + }); + }); + + describe("origins at the provider", () => { + it("key holds and calls by (origin, id): the same ids on two origins are two holds; answers carry the origin", async () => { + const { peer } = await slowConsumer(); + hold(peer, 0, Notes, { session: "S1" }); + hold(peer, 0, Notes, { session: "S2" }, {}, 3); + const [own, relayed] = [await peer.next("bound"), await peer.next("bound")]; + expect(own.origin).toBeUndefined(); + expect(relayed.origin).toBe(3); + call(peer, 0, 0, add, { text: "a" }); + call(peer, 0, 0, add, { text: "b" }, 3); + const results = [await peer.next("result"), await peer.next("result")]; + expect(results.map(({ origin, call }) => ({ origin, call }))).toEqual([ + { origin: undefined, call: 0 }, + { origin: 3, call: 0 }, + ]); + }); + + it("keep a window per origin: a stalled origin folds its commits, another keeps receiving each", async () => { + const { server, peer } = await slowConsumer(); + hold(peer, 0, Notes, { session: "S1" }); + hold(peer, 0, Notes, { session: "S1" }, {}, 1); + await peer.next("bound"); + await peer.next("bound"); + for (const text of ["a", "b", "c", "d"]) { + await addNote(server, "S1", text); + // Only origin 0 acknowledges. + peer.ack(); + } + await transported(); + const states = statesOf(peer); + expect(states.filter(({ origin }) => origin === undefined)).toHaveLength(4); + // Origin 1: its bound and one state fill its window; three commits wait, folded into its next state. + expect(states.filter(({ origin }) => origin === 1)).toHaveLength(1); + peer.ack(1); + peer.ack(1); + await transported(); + const folded = statesOf(peer) + .filter(({ origin }) => origin === 1) + .at(-1)!; + expect(EDITOR.named(folded.holds[0]!.members).count).toEqual([["r", 4]]); + }); + + it("count calls per origin against N: 256 on each of two origins run, one more on one closes the link", async () => { + const { peer, errors } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + hold(peer, 0, Notes, { session: "S1" }, {}, 1); + await peer.next("bound"); + await peer.next("bound"); + for (let id = 0; id < 256; id++) { + call(peer, id, 0, wait); + call(peer, id, 0, wait, undefined, 1); + } + await transported(); + expect(peer.closed).toBe(false); + call(peer, 256, 0, wait, undefined, 1); + await transported(); + expect(peer.closed).toBe(true); + expect(errors).toMatchObject([ + { kind: "protocol_error", error: { message: "call 256: more than 256 calls" } }, + ]); + }); + + it("ignore a call on a hold they do not know before counting it (Q1)", async () => { + const { peer, errors } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + for (let id = 0; id < 256; id++) call(peer, id, 0, wait); + call(peer, 256, 9, wait); + call(peer, 257, 0, wait, undefined, 4); + await transported(); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + }); + + it("closed releases the origin's holds and aborts its calls; another origin keeps its own", async () => { + const { peer, probe, server } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + hold(peer, 0, Notes, { session: "S2" }, {}, 2); + await peer.next("bound"); + await peer.next("bound"); + call(peer, 0, 0, wait, undefined, 2); + await transported(); + const [waiting] = probe.waits; + peer.send({ kind: "closed", origin: 2 }); + await transported(); + expect(waiting!.abortSignal?.aborted).toBe(true); + // Its ids are free: nothing answers them, a new hold of origin 2 starts over. + call(peer, 1, 0, add, { text: "x" }, 2); + await addNote(server, "S1", "a"); + await transported(); + expect(peer.kinds().filter((kind) => kind === "result" || kind === "error")).toEqual([]); + expect(statesOf(peer).map(({ origin }) => origin)).toEqual([undefined]); + }); + + it("carry constants and failed members, and an empty variant's binding, in a first bound (race 24)", async () => { + const Empty = remote("app.empty", { version: 1, params: {}, variants: { v: {} } }); + const N = defineState("n", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const { peer } = await rawLinked((h) => { + h.serve(Empty, () => ({ variant: "v", members: {} })); + h.serve(Halves, (_, _principal, self) => { + const n = self.state(N); + return { + variant: "v", + members: { + one: constant(1), + half: self.derive([n], ([{ n }]) => { + if (n === 2) throw new Error("half threw"); + return n / 2; + }), + set: async (value, ctx) => { + await n.change((draft) => { + draft.n = value; + }, ctx); + return null; + }, + }, + }; + }); + }); + hold(peer, 0, Halves, {}); + await peer.next("bound"); + call(peer, 0, 0, HALVES.methods.set!, 2); + await peer.next("result"); + hold(peer, 1, Empty, {}); + hold(peer, 2, Halves, {}); + const bounds = [await peer.next("bound"), await peer.next("bound")]; + const [empty, halves] = [1, 2].map((id) => bounds.find(({ holds }) => holds[0]!.hold === id)!); + expect(empty.holds).toEqual([{ hold: 1, variant: 0, members: [], failed: [] }]); + expect(halves.holds).toEqual([ + { + hold: 2, + variant: 0, + members: [{ member: HALVES.state.one!, value: expect.anything() }], + failed: [ + { member: HALVES.state.half!, error: { kind: "plugin", code: "internal", message: "half threw" } }, + ], + }, + ]); + expect(HALVES.named(halves.holds[0]!.members)).toEqual({ one: 1 }); + }); + }); + + describe("GC of the window", () => { + it("a closed origin drops its pending queue's ops: nothing keeps them", async () => { + const push = PendingOps.prototype.push; + const pushed: WeakRef[] = []; + // Patched by hand: a spy would keep every argument in its call log. + PendingOps.prototype.push = function (this: PendingOps, ops) { + pushed.push(new WeakRef(ops)); + push.call(this, ops); + }; + try { + const { server, peer } = await slowConsumer(); + hold(peer, 0, Notes, { session: "S1" }, {}, 1); + await peer.next("bound"); + for (const text of ["a", "b", "c"]) await addNote(server, "S1", text); + expect(pushed.length).toBeGreaterThan(0); + peer.send({ kind: "closed", origin: 1 }); + await transported(); + await expectCollected(...pushed); + expect(peer.closed).toBe(false); + } finally { + PendingOps.prototype.push = push; + } + }); + }); + + describe("the call limit", () => { + it("closes a link whose peer sends more calls than this side announced", async () => { + const { peer, errors } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + for (let id = 0; id <= 256; id++) call(peer, id, 0, wait); + await transported(); + expect(peer.closed).toBe(true); + expect(errors).toMatchObject([ + { kind: "protocol_error", error: { message: "call 256: more than 256 calls" } }, + ]); + }); + + it("frees a slot per reply, cancel and release", async () => { + const { peer, errors } = await notesServer(); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + for (let id = 0; id < 256; id++) call(peer, id, 0, wait); + peer.send({ kind: "abort", call: 0 }); + call(peer, 256, 0, wait); + peer.send({ kind: "release", hold: 0 }); + hold(peer, 1, Notes, { session: "S1" }); + await peer.next("bound"); + for (let id = 300; id < 556; id++) call(peer, id, 1, wait); + await transported(); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + }); + }); +}); + +/** `Halves`: a constant, and a member that fails once `set(2)`. */ +const Halves = remote("app.halves", { + version: 1, + params: {}, + variants: { v: { state: { one: int(), half: int() }, methods: { set: method(int()) } } }, +}); + +const HALVES = wireNames(Halves, "v"); + +forEachCodec(() => { + describe("limits per part at the provider", () => { + /** + * A server serving `Notes` behind a raw consumer whose link has `limits`, compressing with `compression`; the raw + * peer is its child, or its parent (only a parent may send compressed arguments). + */ + async function limited(limits: PartLimits, compression?: Compression, role?: "child" | "parent") { + return rawLinked((h) => void serveNotes(h), { limits, compression, role }); + } + + /** Deflated JSON of `{ text }` with `length` characters. */ + const deflatedText = (length: number) => + new DeflatedPart(nodeCompression.deflate(UTF8.encode(JSON.stringify({ text: "x".repeat(length) })))); + + it("fails a call whose argument part is past the wire limit with too_large/part; the link stays", async () => { + const { peer } = await limited({ part: 100 }); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + call(peer, 0, 0, add, { text: "x".repeat(200) }); + expect(await peer.next("error")).toMatchObject({ + call: 0, + error: { kind: "chord", error: { code: "too_large", reason: "part" } }, + }); + call(peer, 1, 0, add, { text: "fits" }); + expect(partValue((await peer.next("result")).value!)).toBe(1); + expect(peer.closed).toBe(false); + }); + + it("fails a hold whose params are past the limit broken too_large/part", async () => { + const { peer } = await limited({ part: 30 }); + hold(peer, 0, Notes, { session: "s".repeat(40) }); + expect(await peer.next("failed")).toMatchObject({ + hold: 0, + cause: { kind: "broken", error: { kind: "chord", error: { code: "too_large", reason: "part" } } }, + }); + }); + + it("fails a call whose deflated argument inflates past the decoded limit, on both inflate paths", async () => { + for (const compression of [nodeCompression, undefined]) { + const { peer } = await limited({ decoded: 10_000 }, compression, "parent"); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + peer.send({ kind: "call", call: 0, hold: 0, method: add, args: deflatedText(50_000) }); + expect(await peer.next("error")).toMatchObject({ error: { error: { code: "too_large" } } }); + call(peer, 1, 0, add, { text: "fits" }); + expect((await peer.next("result")).call).toBe(1); + } + }); + + it("inflates a deflated argument within the hold's stamped decoded limit, a bomb included (both paths)", async () => { + const output = countInflated(); + try { + for (const compression of [nodeCompression, undefined]) { + // The stamped limit is smaller than the link's own: it bounds the inflate. + const { peer } = await limited({}, compression, "parent"); + peer.send({ ...notesHold(0), limit: 1000 }); + await peer.next("bound"); + const bomb = new DeflatedPart( + nodeCompression.deflate(UTF8.encode(JSON.stringify("x".repeat(64 << 20)))), + ); + peer.send({ kind: "call", call: 0, hold: 0, method: add, args: bomb }); + expect(await peer.next("error")).toMatchObject({ error: { error: { code: "too_large" } } }); + peer.send({ kind: "call", call: 1, hold: 0, method: add, args: deflatedText(500) }); + expect((await peer.next("result")).call).toBe(1); + } + // The asynchronous inflate stopped within a chunk of the stamped limit, never near the bomb's 64 MiB. + expect(output.bytes).toBeLessThan(1024 * 1024); + } finally { + output.restore(); + } + }); + + it("checks a deflated argument's wire size before inflating it (both paths)", async () => { + for (const compression of [nodeCompression, undefined]) { + const { peer, errors } = await limited({ part: 18 }, compression, "parent"); + hold(peer, 0, Notes, { session: "S1" }); + await peer.next("bound"); + // Corrupt data past the wire limit: refused by its size, never inflated (no protocol error). + peer.send({ kind: "call", call: 0, hold: 0, method: add, args: new DeflatedPart(new Uint8Array(20)) }); + expect(await peer.next("error")).toMatchObject({ error: { error: { code: "too_large" } } }); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + } + }); + + it("inflates nothing for a call it ignores: an unknown hold (Q1), a refused one (both paths)", async () => { + for (const compression of [nodeCompression, undefined]) { + const { peer, errors } = await limited({}, compression, "parent"); + const corrupt = () => new DeflatedPart(new Uint8Array([255, 255, 255])); + peer.send({ kind: "call", call: 0, hold: 9, method: add, args: corrupt() }); + peer.send({ ...notesHold(0), type: "app.kept" }); + await peer.next("failed"); + peer.send({ kind: "call", call: 1, hold: 0, method: 0, args: corrupt() }); + await transported(); + expect(peer.closed).toBe(false); + expect(errors).toEqual([]); + } + }); + + it("refuses compressed arguments from a child, and compressed params on any link: protocol errors", async () => { + const child = await limited({}); + hold(child.peer, 0, Notes, { session: "S1" }); + await child.peer.next("bound"); + child.peer.send({ kind: "call", call: 0, hold: 0, method: add, args: deflatedText(10) }); + await transported(); + expect(child.peer.closed).toBe(true); + expect(child.errors).toMatchObject([{ kind: "protocol_error" }]); + const parent = await limited({}, undefined, "parent"); + parent.peer.send({ ...notesHold(0), params: new DeflatedPart(nodeCompression.deflate(UTF8.encode("{}"))) }); + await transported(); + expect(parent.peer.closed).toBe(true); + expect(parent.errors).toMatchObject([{ kind: "protocol_error" }]); + }); + + it("honours a stamped decoded limit only on a parent link: a child's is ignored (race 12)", async () => { + const { peer } = await limited({}); + peer.send({ ...notesHold(0), limit: 10 }); + await peer.next("bound"); + call(peer, 0, 0, add, { text: "longer than ten bytes" }); + expect((await peer.next("result")).call).toBe(0); + }); + }); +}); + +/** A hold of `Notes` for S1 as a message. */ +function notesHold(id: number): Extract { + const { name: type, version, fingerprint } = Notes; + return { kind: "hold", hold: id, type, version, fingerprint, params: jsonPart({ session: "S1" }) }; +} + +const UTF8 = new TextEncoder(); diff --git a/packages/chord/test/next/remote/relay.test.ts b/packages/chord/test/next/remote/relay.test.ts new file mode 100644 index 000000000..78fbddb0c --- /dev/null +++ b/packages/chord/test/next/remote/relay.test.ts @@ -0,0 +1,732 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { local, str } from "../../../src/next/contracts/index.ts"; +import { createHost, defineFacet } from "../../../src/next/host/index.ts"; +import { type Compression, type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { binaryCodec, jsonPart, type Message, textCodec } from "../../../src/next/link/internal.ts"; +import type * as Values from "../../../src/next/link/values.ts"; +import { RelayedHold } from "../../../src/next/remote/relay.ts"; +import { nodeCompression } from "../../../src/next-node/compression.ts"; +import { + addNote, + countInflated, + EDITOR, + expectCollected, + fakeTimers, + forEachCodec, + instancesOf, + PlacedNotes, + RawPeer, + rawWorkers, + rejection, + relayed, + type TestPrincipal, + testCodec, + transported, + wireNames, +} from "../support.ts"; + +/** How often a part was parsed in this file, by anyone. */ +const parses = vi.hoisted(() => ({ count: 0 })); + +vi.mock("../../../src/next/link/values.ts", async (importOriginal) => { + const original = await importOriginal(); + return { + ...original, + parsePart: (part: string | Uint8Array) => { + parses.count++; + return original.parsePart(part); + }, + parsePayload: (part: string | Uint8Array, segments: readonly Uint8Array[] | undefined) => { + parses.count++; + return original.parsePayload(part, segments); + }, + }; +}); + +/** Every relayed hold started in this file, weakly. */ +const relayedHolds = instancesOf(RelayedHold, "start"); + +const READER = wireNames(PlacedNotes, "reader"); + +const { wait, add } = EDITOR.methods as { wait: number; add: number }; + +/** A server routing to raw workers, and a raw tab on a link it accepted as `principal`. */ +async function rawRelay(options: { readonly principal?: TestPrincipal; readonly calls?: number } = {}) { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers, { calls: options.calls }) }); + const tab = rawTab(relay.server, options.principal); + return { ...relay, workers, raw: tab }; +} + +/** A raw tab on a new link the server accepts as `principal` (owner by default). */ +function rawTab(server: Awaited>["server"], principal?: TestPrincipal): RawPeer { + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: principal ?? { kind: "owner" }, codec: testCodec() }, context); + return new RawPeer(b); +} + +/** Sends a hold of `PlacedNotes` for `session` with id `hold`, naming its address. */ +function holdPlaced(peer: RawPeer, hold: number, session = "S1"): void { + peer.send({ + kind: "hold", + hold, + type: PlacedNotes.name, + version: PlacedNotes.version, + fingerprint: PlacedNotes.fingerprint, + params: jsonPart({ session }), + address: { kind: "session", params: { id: session } }, + }); +} + +/** The raw worker at `index` once it exists (a resolve runs a turn after the hold arrived). */ +async function workerAt(workers: RawPeer[], index = 0): Promise { + for (let tries = 0; workers[index] === undefined && tries < 10; tries++) await transported(); + return workers[index]!; +} + +/** A hold as a raw worker received it: its id and its origin, which every answer names. */ +type Held = { readonly hold: number; readonly origin?: number }; + +/** The origin field of an answer to `held`. */ +function originOf({ origin }: Held): { origin?: number } { + return origin === undefined ? {} : { origin }; +} + +/** Binds `held` on a raw worker with `PlacedNotes`'s editor (or reader) variant and `count`. */ +function bindPlaced(peer: RawPeer, held: Held, count = 0, variant = EDITOR): void { + const members = [ + { member: variant.state.notes!, value: "[]" }, + { member: variant.state.count!, value: jsonPart(count) }, + ]; + peer.send({ + kind: "bound", + ...originOf(held), + holds: [{ hold: held.hold, variant: variant.variant, members, failed: [] }], + }); +} + +/** A `state` of `count` for `holds` (of one origin) on a raw worker. */ +function countState(holds: readonly Held[], count: number): Extract { + const ops = jsonPart([["r", count]]); + return { + kind: "state", + ...originOf(holds[0]!), + holds: holds.map(({ hold }) => ({ + hold, + members: [{ member: EDITOR.state.count!, ops }], + failed: [], + retired: [], + })), + }; +} + +forEachCodec(() => { + describe("ids and origins", () => { + it("keep the downstream ids upstream, under the downstream link's origin; answers come back by it", async () => { + const { raw, workers } = await rawRelay(); + holdPlaced(raw, 7); + const worker = await workerAt(workers); + const held = await worker.next("hold"); + expect(held).toMatchObject({ hold: 7, origin: 1 }); + bindPlaced(worker, held, 3); + const bound = await raw.next("bound"); + expect(bound.origin).toBeUndefined(); + expect(bound.holds[0]).toMatchObject({ hold: 7, variant: EDITOR.variant }); + raw.send({ kind: "call", call: 5, hold: 7, method: add, args: jsonPart({ text: "x" }) }); + // Calls are forwarded at once, with the same id. + expect(await worker.next("call")).toMatchObject({ origin: 1, call: 5, hold: 7, method: add }); + worker.send({ kind: "result", origin: 1, call: 5, value: "1" }); + const result = await raw.next("result"); + expect(result).toMatchObject({ kind: "result", call: 5 }); + expect(result.origin).toBeUndefined(); + raw.send({ kind: "retry", hold: 7 }); + expect(await worker.next("retry")).toEqual({ kind: "retry", origin: 1, hold: 7 }); + }); + + it("give each downstream link its own origin, never reused", async () => { + const { raw, server, workers } = await rawRelay(); + holdPlaced(raw, 0); + const worker = await workerAt(workers); + const one = await worker.next("hold"); + const other = rawTab(server); + holdPlaced(other, 0); + const two = await worker.next("hold"); + expect([one.origin, two.origin]).toEqual([1, 2]); + other.close(); + await transported(); + holdPlaced(rawTab(server), 0); + expect((await worker.next("hold")).origin).toBe(3); + }); + + it("drop entries, notices and replies for holds of the origin relayed over another link", async () => { + const { raw, workers } = await rawRelay(); + holdPlaced(raw, 0, "S1"); + holdPlaced(raw, 1, "S2"); + const one = await workerAt(workers, 0); + const x = await one.next("hold"); + const two = await workerAt(workers, 1); + const y = await two.next("hold"); + expect([x.origin, y.origin]).toEqual([1, 1]); + bindPlaced(one, x); + await raw.next("bound"); + raw.send({ kind: "call", call: 0, hold: 0, method: wait }); + await one.next("call"); + // Worker two speaks for worker one's hold of the same tab origin. + two.send(countState([x], 9)); + two.send({ kind: "loading", origin: 1, hold: 0 }); + two.send({ kind: "result", origin: 1, call: 0 }); + two.send({ + kind: "gone", + origin: 1, + hold: 0, + error: { kind: "chord", error: { code: "unavailable", message: "forged" } }, + }); + bindPlaced(two, x); + await transported(); + expect(raw.kinds()).toEqual(["hello"]); + // Its windowed messages, left empty, are acknowledged to it at once. + expect(two.kinds().filter((kind) => kind === "ack")).toEqual(["ack", "ack"]); + // A message with its own hold and the other's: only its own entry is forwarded. + bindPlaced(two, y); + await raw.next("bound"); + two.send({ kind: "state", origin: 1, holds: [...countState([x], 9).holds, ...countState([y], 3).holds] }); + const state = await raw.next("state"); + expect(state.holds.map(({ hold }) => hold)).toEqual([1]); + one.send({ kind: "result", origin: 1, call: 0 }); + expect(await raw.next("result")).toMatchObject({ call: 0 }); + }); + + it("ignore answers for an origin from a link it relayed nothing to", async () => { + const { raw, server, workers } = await rawRelay(); + holdPlaced(raw, 0); + const worker = await workerAt(workers); + const held = await worker.next("hold"); + // Another tab speaks for the first tab's origin on its own link. + const other = rawTab(server); + await transported(); + bindPlaced(other, held); + await transported(); + expect(raw.kinds()).toEqual(["hello"]); + bindPlaced(worker, held); + await raw.next("bound"); + }); + }); + + describe("parts at a relay", () => { + it("are never parsed: params, snapshots, ops, arguments and results travel as they came", async () => { + const { raw, workers } = await rawRelay(); + await transported(); + const before = parses.count; + holdPlaced(raw, 0); + const worker = await workerAt(workers); + const held = await worker.next("hold"); + bindPlaced(worker, held, 1); + await raw.next("bound"); + worker.send(countState([held], 2)); + await raw.next("state"); + raw.send({ kind: "call", call: 0, hold: 0, method: add, args: jsonPart({ text: "x" }) }); + await worker.next("call"); + worker.send({ kind: "result", origin: held.origin, call: 0, value: "2" }); + await raw.next("result"); + expect(parses.count).toBe(before); + }); + }); + + describe("the window through a relay", () => { + it("forwards every bound and state at once and relays each ack to its worker", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers, { ack: false }) }); + const [a, b] = memoryLinkPair(); + relay.server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const tab = new RawPeer(b, { ack: false }); + holdPlaced(tab, 0); + const worker = await workerAt(workers); + const held = await worker.next("hold"); + bindPlaced(worker, held); + for (const count of [1, 2, 3]) worker.send(countState([held], count)); + await transported(); + // The relay holds nothing back: the worker's window for this origin is the limit. + expect(tab.kinds()).toEqual(["hello", "bound", "state", "state", "state"]); + expect(worker.kinds().filter((kind) => kind === "ack")).toEqual([]); + tab.ack(); + tab.ack(); + await transported(); + expect(worker.received.filter(({ kind }) => kind === "ack")).toEqual([ + { kind: "ack", origin: held.origin }, + { kind: "ack", origin: held.origin }, + ]); + }); + + it("acknowledges to the right worker in order: two workers sharing one tab", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers, { ack: false }) }); + const [a, b] = memoryLinkPair(); + relay.server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const tab = new RawPeer(b, { ack: false }); + holdPlaced(tab, 0, "S1"); + const one = await workerAt(workers, 0); + const x = await one.next("hold"); + holdPlaced(tab, 1, "S2"); + const two = await workerAt(workers, 1); + const y = await two.next("hold"); + bindPlaced(one, x); + await transported(); + bindPlaced(two, y); + await transported(); + tab.ack(); + await transported(); + expect(one.kinds().filter((kind) => kind === "ack")).toEqual(["ack"]); + expect(two.kinds().filter((kind) => kind === "ack")).toEqual([]); + tab.ack(); + await transported(); + expect(two.kinds().filter((kind) => kind === "ack")).toEqual(["ack"]); + }); + + it("never tears a commit: a slow tab's states fold at the worker, a derive over two holds sees both at one moment (race 21)", async () => { + const Pair = local("app.pair", { version: 1, params: {}, members: { pair: str() } }); + const relay = await relayed(); + const seen: string[] = []; + const tab = createHost({ onError: () => {} }); + const [a, b] = memoryLinkPair(); + relay.server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + const slow = pausable(b); + tab.connect(slow.link, { principal: { kind: "server" }, codec: testCodec() }, context); + await tab.load( + [ + { + name: "@test/pair", + facet: defineFacet((h) => + h.serve(Pair, (_, self) => { + const one = self.get(PlacedNotes, { session: "S1" }); + const two = self.get(PlacedNotes, { session: "S1" }); + return { + pair: self.derive([one.count, two.count], ([x, y]) => { + seen.push(`${x}/${y}`); + return `${x}/${y}`; + }), + }; + }), + ), + }, + ], + context, + ); + const view = await tab.get(Pair, {}, context); + const watch = await view.pair.watch(context); + watch.start(async () => {}); + const worker = relay.workers.get("session|S1")!; + slow.pause(); + slow.kinds.length = 0; + for (const text of ["a", "b", "c", "d", "e"]) await addNote(worker, "S1", text); + await transported(); + // Two states in the tab origin's window at the worker, the other commits folded there. + expect(slow.kinds).toEqual(["state", "state"]); + slow.resume(); + await transported(); + await transported(); + expect(view.pair.value).toBe("5/5"); + expect(slow.kinds).toEqual(["state", "state", "state"]); + expect(seen.every((pair) => pair.split("/")[0] === pair.split("/")[1])).toBe(true); + expect(relay.errors.server).toEqual([]); + }); + }); + + describe("an upstream drop", () => { + fakeTimers(); + + it("sends loading downstream, fails relayed calls unavailable, holds again with the same id and forwards the new bound, another variant allowed (race 12)", async () => { + const { tab, workers, errors } = await rawRelay(); + const got = tab.get(PlacedNotes, { session: "S1" }, context); + const first = await workerAt(workers); + const held = await first.next("hold"); + bindPlaced(first, held, 1); + const notes = await got; + expect(notes.variant).toBe("editor"); + const call = notes.wait(context); + await first.next("call"); + first.close(); + expect(await rejection(call)).toMatchObject({ code: "unavailable" }); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + await vi.advanceTimersByTimeAsync(1000); + const second = await workerAt(workers, 1); + const again = await second.next("hold"); + expect(again).toMatchObject({ hold: held.hold, origin: held.origin }); + bindPlaced(second, again, 4, READER); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + expect(notes.variant).toBe("reader"); + expect(notes.count.value).toBe(4); + expect(errors.tab).toEqual([]); + }); + + it("answers a call that crossed the loading notice unavailable (race 13)", async () => { + const { raw, workers } = await rawRelay(); + holdPlaced(raw, 0); + const worker = await workerAt(workers); + bindPlaced(worker, await worker.next("hold")); + await raw.next("bound"); + worker.close(); + await transported(); + raw.send({ kind: "call", call: 0, hold: 0, method: wait }); + expect(await raw.next("loading")).toEqual({ kind: "loading", hold: 0 }); + expect((await raw.next("error")).error).toMatchObject({ error: { code: "unavailable" } }); + }); + }); + + describe("a downstream release", () => { + it("before the upstream hold was sent drops it; after, releases it upstream; messages crossing it are dropped (race 14)", async () => { + let finish: () => void = () => {}; + const workers: RawPeer[] = []; + const make = rawWorkers(workers); + const { server } = await relayed({ + resolve: async (address) => { + await new Promise((resolve) => { + finish = resolve; + }); + return make(address); + }, + }); + const raw = rawTab(server); + holdPlaced(raw, 0); + await transported(); + raw.send({ kind: "release", hold: 0 }); + holdPlaced(raw, 1); + await transported(); + finish(); + const worker = await workerAt(workers); + const sent = await worker.next("hold"); + expect(sent.hold).toBe(1); + await transported(); + expect(worker.kinds()).toEqual(["hello"]); + raw.send({ kind: "release", hold: 1 }); + expect(await worker.next("release")).toEqual({ kind: "release", origin: sent.origin, hold: 1 }); + bindPlaced(worker, sent); + worker.send(countState([sent], 1)); + await transported(); + // The relay no longer relays the hold: dropped there (Q1), each message acknowledged to the worker at once. + expect(raw.kinds()).toEqual(["hello"]); + expect(worker.received.filter(({ kind }) => kind === "ack")).toHaveLength(2); + }); + }); + + describe("a downstream drop", () => { + it("closes its origin upstream: one closed per worker, no release or cancel (race 15)", async () => { + const { raw, workers } = await rawRelay(); + holdPlaced(raw, 0, "S1"); + holdPlaced(raw, 1, "S2"); + const one = await workerAt(workers, 0); + const held = await one.next("hold"); + const two = await workerAt(workers, 1); + await two.next("hold"); + bindPlaced(one, held); + await raw.next("bound"); + raw.send({ kind: "call", call: 0, hold: 0, method: wait }); + await one.next("call"); + raw.close(); + await transported(); + for (const worker of [one, two]) { + const ends = worker.received.filter(({ kind }) => ["release", "abort", "closed"].includes(kind)); + expect(ends).toEqual([{ kind: "closed", origin: held.origin }]); + } + }); + }); + + describe("gone from upstream", () => { + it("is forwarded and the relayed hold forgotten (race 16)", async () => { + const { tab, workers } = await rawRelay(); + const got = tab.get(PlacedNotes, { session: "S1" }, context); + const worker = await workerAt(workers); + const held = await worker.next("hold"); + bindPlaced(worker, held); + const notes = await got; + const watch = await notes.count.watch(context); + worker.send({ + kind: "gone", + ...originOf(held), + hold: held.hold, + error: { kind: "chord", error: { code: "unavailable", message: "revoked" } }, + }); + expect(await watch.closed).toEqual({ kind: "ended" }); + expect(() => notes.count.value).toThrow("revoked"); + // A state crossing the gone reaches the tab, which ignores it; nothing closes. + worker.send(countState([held], 9)); + await transported(); + expect(worker.closed).toBe(false); + }); + }); + + describe("GC at a relay", () => { + it("a released relayed hold and a closed downstream link keep no relay state", async () => { + const { refs, server } = await (async () => { + const { raw, workers, server } = await rawRelay(); + const made = relayedHolds.length; + holdPlaced(raw, 0); + holdPlaced(raw, 1); + const worker = await workerAt(workers); + const one = await worker.next("hold"); + bindPlaced(worker, one); + await raw.next("bound"); + raw.send({ kind: "call", call: 0, hold: 0, method: wait }); + await worker.next("call"); + raw.send({ kind: "release", hold: 0 }); + raw.close(); + await transported(); + return { refs: relayedHolds.slice(made), server }; + })(); + expect(refs).toHaveLength(2); + await expectCollected(...refs); + expect(server).toBeDefined(); + }); + }); +}); + +describe("received buffers at a relay", () => { + it("are not kept by the parts of a relayed hold after its message was handled", async () => { + const workers: RawPeer[] = []; + const { server } = await relayed({ resolve: rawWorkers(workers) }); + const [a, b] = memoryLinkPair(); + const buffers: WeakRef[] = []; + const listening: Link = { + send: (message) => a.send(message), + close: () => a.close(), + onClose: (listener) => a.onClose(listener), + onMessage: (listener) => + a.onMessage((message) => { + buffers.push(new WeakRef(message.buffer)); + listener(message); + }), + }; + server.accept(listening, { principal: { kind: "owner" }, codec: binaryCodec }, context); + const tab = (() => { + const peer = new RawPeer(b, { codec: binaryCodec }); + holdPlaced(peer, 0, "S1"); + return peer; + })(); + const worker = await workerAt(workers); + bindPlaced(worker, await worker.next("hold")); + await tab.next("bound"); + expect(buffers.length).toBeGreaterThan(1); + await expectCollected(...buffers); + expect(relayedHolds.at(-1)?.deref()).toBeDefined(); + }); +}); + +describe("mixed codecs", () => { + it("relay between a binary tab link and a text route link", async () => { + const { tab, workers } = await relayed({ codecs: { tab: binaryCodec, route: textCodec } }); + const notes = await tab.get(PlacedNotes, { session: "S1" }, context); + await addNote(workers.get("session|S1")!, "S1", "é"); + await transported(); + expect(notes.notes.value).toEqual([{ text: "é" }]); + if (notes.variant !== "editor") throw new Error("expected the editor"); + expect(await notes.add({ text: "b" }, context)).toBe(2); + const { context: cancellable, cancel } = withCancel(context); + const waiting = notes.wait(cancellable); + cancel(new Error("stop")); + await expect(waiting).rejects.toThrow("stop"); + }); +}); + +/** + * A link end whose incoming messages can be held back (a slow tab): paused, they wait; resumed, they arrive in order. + * `kinds` lists what arrived, decoded with the test codec. + */ +function pausable(link: Link): { readonly link: Link; readonly kinds: string[]; pause(): void; resume(): void } { + let paused = false; + const held: Uint8Array[] = []; + const kinds: string[] = []; + let listener: (message: Uint8Array) => void = () => {}; + link.onMessage((message) => { + kinds.push(testCodec().decode(message).kind); + if (paused) held.push(message); + else listener(message); + }); + return { + kinds, + link: { + send: (message) => link.send(message), + close: () => link.close(), + onClose: (close) => link.onClose(close), + onMessage: (next) => { + listener = next; + }, + }, + pause: () => { + paused = true; + }, + resume: () => { + paused = false; + for (const message of held.splice(0)) listener(message); + }, + }; +} + +/** A compressor that counts its inflates (`nodeCompression` underneath). */ +function countingInflates(): Compression & { inflates: number } { + const counts = { + inflates: 0, + deflate: (bytes: Uint8Array) => nodeCompression.deflate(bytes), + inflate: (bytes: Uint8Array, limit: number) => { + counts.inflates++; + return nodeCompression.inflate(bytes, limit); + }, + }; + return counts; +} + +forEachCodec(() => { + describe("limits and compression through a relay", () => { + it("check an argument's size on the admitting link at the relay: too_large/part, nothing forwarded", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers) }); + const [a, b] = memoryLinkPair(); + relay.server.accept(a, { principal: { kind: "owner" }, codec: testCodec(), limits: { part: 100 } }, context); + const raw = new RawPeer(b); + holdPlaced(raw, 0); + const worker = await workerAt(workers); + bindPlaced(worker, await worker.next("hold")); + await raw.next("bound"); + raw.send({ kind: "call", call: 0, hold: 0, method: add, args: jsonPart({ text: "x".repeat(200) }) }); + expect(await raw.next("error")).toMatchObject({ + call: 0, + error: { error: { code: "too_large", reason: "part" } }, + }); + await transported(); + expect(worker.kinds()).not.toContain("call"); + }); + + it("stamp the admitting link's decoded limit on the upstream hold; a tab's own is ignored (race 12)", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers) }); + const [a, b] = memoryLinkPair(); + relay.server.accept( + a, + { principal: { kind: "owner" }, codec: testCodec(), limits: { decoded: 4096 } }, + context, + ); + const raw = new RawPeer(b); + raw.send({ + kind: "hold", + hold: 0, + type: PlacedNotes.name, + version: PlacedNotes.version, + fingerprint: PlacedNotes.fingerprint, + params: jsonPart({ session: "S1" }), + address: { kind: "session", params: { id: "S1" } }, + limit: 1_000_000_000, + }); + const worker = await workerAt(workers); + expect((await worker.next("hold")).limit).toBe(4096); + }); + + it("enforce the stamped limit where the part is decoded: the worker refuses the tab's large argument", async () => { + const relay = await relayed({ trusted: true, limits: { decoded: 1000 } }); + const notes = await relay.tab.get(PlacedNotes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("bound as the editor"); + // Within the tab link's wire limit, past its decoded limit, which the trusted worker link does not have. + expect(await rejection(notes.add({ text: "x".repeat(2000) }, context))).toMatchObject({ + code: "too_large", + reason: "part", + }); + expect(await notes.add({ text: "small" }, context)).toBe(1); + }); + + it("forward deflated parts as they came: the relay inflates nothing (a counter)", async () => { + const [server, tab] = [countingInflates(), countingInflates()]; + const relay = await relayed({ compression: { server, worker: nodeCompression, tab } }); + const notes = await relay.tab.get(PlacedNotes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("bound as the editor"); + const text = "lorem ipsum ".repeat(3000); + await notes.add({ text }, context); + await transported(); + expect(notes.notes.value).toEqual([{ text }]); + // The worker's state deflated once at its origin, inflated by its owner, the tab. + expect([server.inflates, tab.inflates]).toEqual([0, 1]); + }); + + it("forward deflated parts as they came at a relay without a compressor: nothing inflated there", async () => { + const inflated = countInflated(); + try { + const tab = countingInflates(); + const relay = await relayed({ compression: { worker: nodeCompression, tab } }); + const notes = await relay.tab.get(PlacedNotes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("bound as the editor"); + await notes.add({ text: "lorem ipsum ".repeat(3000) }, context); + await transported(); + expect(tab.inflates).toBe(1); + expect(inflated.bytes).toBe(0); + } finally { + inflated.restore(); + } + }); + + it("forward answers from upstream unchecked: the end that parses a part checks it against its own link", async () => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers, { limits: { part: 50 } }) }); + const getting = relay.tab.get(PlacedNotes, { session: "S1" }, context); + const worker = await workerAt(workers); + const held = await worker.next("hold"); + bindPlaced(worker, held); + const notes = await getting; + if (notes.variant !== "editor") throw new Error("bound as the editor"); + // A state part and a result past the route link's wire limit reach the tab, whose link has none that small. + const ops = jsonPart([["r", 1234567890], ...Array(10).fill(["r", 7])]); + worker.send({ + kind: "state", + ...originOf(held), + holds: [{ hold: held.hold, members: [{ member: EDITOR.state.count!, ops }], failed: [], retired: [] }], + }); + const added = notes.add({ text: "x" }, context); + const call = await worker.next("call"); + worker.send({ kind: "result", ...originOf(held), call: call.call, value: jsonPart("y".repeat(60)) }); + expect(await added).toBe("y".repeat(60)); + await transported(); + expect(notes.count.value).toBe(7); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it.each([0, 1200])( + "detach a reference failed too_large from its route; retry %i ms after the drop places it anew, a new hold", + async (wait) => { + const workers: RawPeer[] = []; + const relay = await relayed({ resolve: rawWorkers(workers, { limits: { part: 50 } }) }); + const notes = relay.server.hold(PlacedNotes, { session: "S1" }, context); + const first = await workerAt(workers); + const held = await first.next("hold"); + const members = [ + { member: EDITOR.state.notes!, value: jsonPart([{ text: "z".repeat(60) }]) }, + { member: EDITOR.state.count!, value: jsonPart(1) }, + ]; + first.send({ kind: "bound", holds: [{ hold: held.hold, variant: EDITOR.variant, members, failed: [] }] }); + expect(await first.next("release")).toMatchObject({ hold: held.hold }); + expect(notes.state.value).toMatchObject({ kind: "failed", cause: { error: { code: "too_large" } } }); + // The worker's link drops: nothing holds the address, so nothing resolves it again for this reference. + first.close(); + await new Promise((resolve) => setTimeout(resolve, wait)); + await transported(); + expect(workers).toHaveLength(1); + // The route forgot the address: the retry places the reference anew, resolves it and holds it. + notes.retry(); + expect(notes.state.value.kind).toBe("loading"); + for (let tries = 0; workers.length < 2 && tries < 30; tries++) await new Promise((r) => setTimeout(r, 100)); + const second = await workerAt(workers, 1); + bindPlaced(second, await second.next("hold"), 2); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + expect(notes.count.value).toBe(2); + }, + ); + + it("let a tab without a compressor inflate the worker's deflated parts asynchronously, in order", async () => { + const relay = await relayed({ compression: { worker: nodeCompression, server: nodeCompression } }); + const notes = await relay.tab.get(PlacedNotes, { session: "S1" }, context); + if (notes.variant !== "editor") throw new Error("bound as the editor"); + const text = "lorem ipsum ".repeat(3000); + for (let i = 0; i < 3; i++) await notes.add({ text }, context); + for (let i = 0; i < 100 && notes.count.value < 3; i++) await new Promise((resolve) => setTimeout(resolve, 5)); + expect(notes.notes.value).toEqual([{ text }, { text }, { text }]); + }); + }); +}); diff --git a/packages/chord/test/next/remote/remote-object.test.ts b/packages/chord/test/next/remote/remote-object.test.ts new file mode 100644 index 000000000..e5e3d69b7 --- /dev/null +++ b/packages/chord/test/next/remote/remote-object.test.ts @@ -0,0 +1,768 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local, method, obj, remote, str } from "../../../src/next/contracts/index.ts"; +import { PluginError } from "../../../src/next/data/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { + createHost, + defineFacet, + type Host, + type HostError, + type Reference, + type RefIn, +} from "../../../src/next/host/index.ts"; +import { resolveWatchable } from "../../../src/next/host/reference.ts"; +import { type Link, memoryLinkPair } from "../../../src/next/link/index.ts"; +import { jsonPart, wireError } from "../../../src/next/link/internal.ts"; +import { RemoteObject } from "../../../src/next/remote/remote-object.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { + bindHold, + bindNext, + changesOf, + delivered, + EDITOR, + expectCollected, + failures, + fakeTimers, + forEachCodec, + framesOf, + instancesOf, + linked, + Notes, + partValue, + RawPeer, + rawParent, + rawProvider, + rejection, + serveNotes, + type TestPrincipal, + testCodec, + transported, + wireNames, +} from "../support.ts"; + +/** A counter document of the remote types defined in tests. */ +const Counter = defineState("counter", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + +forEachCodec(() => { + describe("a hold over the parent link", () => { + it("reject a broken failure at once", async () => { + const Broken = remote("app.broken", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const { client } = await linked((h) => + h.serve(Broken, () => { + throw new Error("factory threw"); + }), + ); + expect(await rejection(client.get(Broken, {}, context, { readyTimeout: 5_000 }))).toEqual({ + code: "unavailable", + message: "app.broken: factory threw", + }); + }); + + it("gone: the reference ends, its pending calls reject with the error", async () => { + const { client, peer } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + const hold = await bindNext(peer); + const notes = await got; + const watch = await notes.notes.watch(context); + const call = notes.wait(context); + await peer.next("call"); + peer.send({ + kind: "gone", + hold, + error: { kind: "chord", error: { code: "released", reason: "ended", message: "app.notes(S1): ended" } }, + }); + expect(await rejection(call)).toEqual({ code: "released", reason: "ended", message: "app.notes(S1): ended" }); + expect(await watch.closed).toEqual({ kind: "ended" }); + expect(() => notes.count.value).toThrow("app.notes(S1): ended"); + }); + + it("two references from one host are two holds", async () => { + const { client, peer } = await rawProvider(); + const first = client.get(Notes, { session: "S1" }, context); + const second = client.get(Notes, { session: "S1" }, context); + expect(await bindNext(peer)).toBe(0); + expect(await bindNext(peer)).toBe(1); + await Promise.all([first, second]); + }); + + it("a member failure fails only that replica: reads throw it, its watch gets a failure call and stays open; the others update", async () => { + const { client, peer } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + const hold = await bindNext(peer); + const notes = await got; + const count = changesOf(await notes.count.watch(context)); + const list = framesOf(await notes.notes.watch(context)); + const failed = [{ member: EDITOR.state.count!, error: wireError(new PluginError("derive", "count threw")) }]; + const members = [{ member: EDITOR.state.notes!, ops: jsonPart([["p", [], 0, 0, [{ text: "a" }]]]) }]; + peer.send({ kind: "state", holds: [{ hold, members, failed, retired: [] }] }); + await delivered(); + expect(count).toEqual([{ kind: "failed", error: expect.objectContaining({ code: "derive" }) }]); + expect(list).toEqual([[[{ text: "a" }], [["p", [], 0, 0, [{ text: "a" }]]]]]); + expect(() => notes.count.value).toThrow("count threw"); + // The provider's derive healed: its next value is a root replacement, in the same hold. + const healed = [{ member: EDITOR.state.count!, ops: jsonPart([["r", 1]]) }]; + peer.send({ kind: "state", holds: [{ hold, members: healed, failed: [], retired: [] }] }); + await delivered(); + expect(notes.count.value).toBe(1); + expect(count.at(-1)).toEqual({ kind: "value", value: 1, ops: [["r", 1]], retired: false }); + }); + }); + + describe("retired members", () => { + it("reach the replicas' watches with the flag: with the member's ops, or alone", async () => { + const { client, peer } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + const hold = await bindNext(peer, [{ text: "a" }]); + const notes = await got; + const list = changesOf(await notes.notes.watch(context)); + const count = changesOf(await notes.count.watch(context)); + const members = [{ member: EDITOR.state.notes!, ops: jsonPart([["r", []]]) }]; + const retired = [EDITOR.state.notes!, EDITOR.state.count!]; + peer.send({ kind: "state", holds: [{ hold, members, failed: [], retired }] }); + await delivered(); + expect(list).toEqual([{ kind: "value", value: [], ops: [["r", []]], retired: true }]); + expect(count).toEqual([{ kind: "value", value: 1, ops: [], retired: true }]); + }); + + it("of a failed member wait for its next value: never read from the failed replica; the link stays open", async () => { + const { client, peer, errors } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + const hold = await bindNext(peer, [{ text: "a" }]); + const notes = await got; + const count = changesOf(await notes.count.watch(context)); + const member = EDITOR.state.count!; + const failed = [{ member, error: wireError(new PluginError("derive", "count threw")) }]; + peer.send({ kind: "state", holds: [{ hold, members: [], failed, retired: [] }] }); + // A retirement alone (no ops) of the failed member, then its healed value. + peer.send({ kind: "state", holds: [{ hold, members: [], failed: [], retired: [member] }] }); + await delivered(); + const value = [{ member, ops: jsonPart([["r", 5]]) }]; + peer.send({ kind: "state", holds: [{ hold, members: value, failed: [], retired: [] }] }); + await transported(); + expect([peer.closed, notes.state.value, errors]).toEqual([false, { kind: "live" }, []]); + expect(count).toEqual([ + { kind: "failed", error: expect.objectContaining({ message: "count threw" }) }, + { kind: "value", value: 5, ops: [["r", 5]], retired: true }, + ]); + }); + + it.each([ + ["names a member outside the variant", [7]], + ["names a member twice", [EDITOR.state.count!, EDITOR.state.count!]], + ])("that %s closes the link", async (_, retired) => { + const { client, peer, errors } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context); + const hold = await bindNext(peer); + await got; + peer.send({ kind: "state", holds: [{ hold, members: [], failed: [], retired }] }); + await transported(); + expect(peer.closed).toBe(true); + expect(errors.map(({ kind }) => kind)).toEqual(["protocol_error"]); + }); + }); + + describe("a snapshot", () => { + it.each([ + ["misses a member", [{ member: EDITOR.state.notes!, value: "[]" }], EDITOR.variant], + [ + "names a member twice", + [ + { member: EDITOR.state.notes!, value: "[]" }, + { member: EDITOR.state.notes!, value: "[]" }, + { member: EDITOR.state.count!, value: "0" }, + ], + EDITOR.variant, + ], + ["names a member outside the variant", [{ member: 7, value: "0" }], EDITOR.variant], + ["names a variant outside the version", [], 5], + ])("that %s closes the link; the reference stays loading", async (_, members, variant) => { + const { client, peer, errors } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + peer.send({ kind: "bound", holds: [{ hold, variant, members, failed: [] }] }); + await transported(); + expect(peer.closed).toBe(true); + expect(errors.map(({ kind }) => kind)).toEqual(["protocol_error"]); + expect(notes.state.value).toEqual({ kind: "loading" }); + expect(notes.variant).toBeUndefined(); + }); + }); + + describe("transient failures over a link", () => { + fakeTimers(); + + it("are waited through; the deadline rejects with the last cause and releases the hold", async () => { + const Flaky = remote("app.flaky", { + version: 1, + params: { id: int() }, + variants: { v: { state: { n: int() } } }, + }); + const opens: string[] = []; + const { client } = await linked((h) => + h.serve(Flaky, ({ id }, _principal, self) => { + self.open( + async (ctx) => { + opens.push(`open ${opens.length + 1}`); + if (opens.length === 1) throw new Error("offline"); + await new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", resolve)); + opens.push("aborted"); + }, + () => {}, + ); + return { variant: "v", members: { n: constant(id) } }; + }), + ); + const got = rejection(client.get(Flaky, { id: 1 }, context, { readyTimeout: 1_500 })); + await transported(); + await vi.advanceTimersByTimeAsync(1_000); + await transported(); + expect(opens).toEqual(["open 1", "open 2"]); + await vi.advanceTimersByTimeAsync(500); + expect(await got).toEqual({ code: "unavailable", message: "app.flaky(1): offline" }); + await transported(); + // The release ended the object: its pending attempt was abandoned. + expect(opens).toEqual(["open 1", "open 2", "aborted"]); + }); + }); + + /** Every remote object bound while this file runs, weakly. */ + const remoteObjects = instancesOf(RemoteObject, "hold"); + + describe("GC of a released remote reference", () => { + it("a released reference retained by the test keeps no hold, replica or object", async () => { + const { server, client } = await linked((h) => void serveNotes(h)); + const { notes, refs } = await (async () => { + const made = remoteObjects.length; + const notes = await client.get(Notes, { session: "S1" }, context); + const refs: WeakRef[] = [new WeakRef(resolveWatchable(notes.notes)!), ...remoteObjects.slice(made)]; + expect(refs).toHaveLength(2); + notes.release(); + await transported(); + return { notes, refs }; + })(); + await expectCollected(...refs); + expect([notes, server, client]).toBeDefined(); + }); + + it("a failure call's error kept by the app keeps neither the replica nor the remote object", async () => { + const Halves = remote("app.halves", { + version: 1, + params: {}, + variants: { v: { state: { half: int() }, methods: { set: method(int()) } } }, + }); + const { server, client } = await linked((h) => + h.serve(Halves, (_, _principal, self) => { + const doc = self.state(Counter); + return { + variant: "v", + members: { + half: self.derive([doc], ([{ n }]) => { + // A fresh error: its stack's frames would keep what threw it. + if (n === 2) throw new Error("half threw"); + return n / 2; + }), + set: async (value, ctx) => { + await doc.change((draft) => { + draft.n = value; + }, ctx); + return null; + }, + }, + }; + }), + ); + const kept: unknown[] = []; + const refs = await (async () => { + const made = remoteObjects.length; + const halves = await client.get(Halves, {}, context); + if (halves.variant !== "v") throw new Error("expected variant v"); + const refs: WeakRef[] = [new WeakRef(resolveWatchable(halves.half)!), ...remoteObjects.slice(made)]; + (await halves.half.watch(context)).start(async (call) => { + if (call.kind === "failed") kept.push(call.error); + }); + await halves.set(2, context); + await transported(); + halves.release(); + await transported(); + return refs; + })(); + expect(kept).toHaveLength(1); + await expectCollected(...refs); + expect([server, client]).toBeDefined(); + }); + }); + + describe("self.get of a type over the link (R18)", () => { + it("a failed hold fails the dependent transiently; bound retries it at once with a new hold", async () => { + const View = local("app.view", { version: 1, params: {}, members: { count: int() } }); + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const [a, b] = memoryLinkPair(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ count: self.get(Notes, { session: "S1" }).count })), + ); + const loading = client.load([{ name: "@test/view", facet }], context); + const { hold } = await peer.next("hold"); + peer.send({ + kind: "failed", + hold, + cause: { kind: "transient", error: { kind: "chord", error: { code: "unavailable", message: "warming" } } }, + }); + await loading; + expect(errors).toMatchObject([ + { + kind: "activation_failed", + object: "app.view", + failure: { kind: "transient", error: { code: "unavailable" } }, + }, + ]); + const members = [ + { member: EDITOR.state.notes!, value: "[]" }, + { member: EDITOR.state.count!, value: "0" }, + ]; + peer.send({ kind: "bound", holds: [{ hold, variant: EDITOR.variant, members, failed: [] }] }); + // The retry binds its own hold before the failed attempt's is released. + const retried = await bindNext(peer, [{ text: "a" }]); + expect(retried).toBe(1); + expect(await peer.next("release")).toMatchObject({ hold }); + const view = await client.get(View, {}, context); + expect(view.count.value).toBe(1); + }); + }); + + describe("self.get through a relay whose upstream dropped (R18)", () => { + it("a loading notice while loading fails the dependent at once, transient; bound retries it", async () => { + const View = local("app.view", { version: 1, params: {}, members: { count: int() } }); + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const [a, b] = memoryLinkPair(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const peer = new RawPeer(b); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ count: self.get(Notes, { session: "S1" }).count })), + ); + const loading = client.load([{ name: "@test/view", facet }], context); + const { hold } = await peer.next("hold"); + // The relay's upstream link dropped before the hold was bound: no wait for the readyTimeout. + peer.send({ kind: "loading", hold }); + await loading; + expect(errors).toMatchObject([ + { + kind: "activation_failed", + object: "app.view", + failure: { kind: "transient", error: { message: expect.stringContaining("app.notes") } }, + }, + ]); + bindHold(peer, hold); + const retried = await bindNext(peer, [{ text: "a" }]); + expect(retried).toBe(1); + const view = await client.get(View, {}, context); + expect(view.count.value).toBe(1); + }); + }); + + describe("a dependent of an object failed transiently over the link", () => { + // Timers stand still: neither the backoff window nor a readyTimeout ends while the test runs. + fakeTimers(); + + it("fails at once with the real cause when its load holds it (no wait for a timeout)", async () => { + const Flaky = remote("app.flaky", { version: 1, params: {}, variants: { v: { state: { n: int() } } } }); + const View = local("app.view", { version: 1, params: {}, members: { n: int() } }); + const { server, client: first } = await linked((h) => + h.serve(Flaky, (_, _principal, self) => { + self.open( + async () => { + throw new Error("offline"); + }, + () => {}, + ); + return { variant: "v", members: { n: constant(1) } }; + }), + ); + // The first client's get makes the object fail; it stays failed for its backoff window. + const failing = first.get(Flaky, {}, context).catch(() => {}); + await transported(); + const errors: HostError[] = []; + const second = createHost({ onError: (error) => errors.push(error) }); + const [a, b] = memoryLinkPair(); + server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + second.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); + const facet = defineFacet((h) => h.serve(View, (_, self) => ({ n: self.get(Flaky, {}).n }))); + await second.load([{ name: "@test/view", facet }], context); + expect(errors).toMatchObject([ + { + kind: "activation_failed", + object: "app.view", + failure: { + kind: "transient", + error: { message: "app.view: dependency app.flaky failed", cause: { message: "app.flaky: offline" } }, + }, + }, + ]); + await first.close(context); + await failing; + }); + }); +}); + +/** A remote type whose variants share `y`: `a` has `x` and a method, `b` has `z`. */ +const Shifting = remote("app.shifting", { + version: 1, + params: {}, + variants: { + a: { state: { x: int(), y: int() }, methods: { m: method() } }, + b: { state: { y: int(), z: int() } }, + }, +}); + +/** A client whose `View` derives `x`+`y` of a `Shifting` reference, connected to a raw provider; `View` is bound. */ +async function shifting() { + const View = local("app.view", { version: 1, params: {}, members: { sum: int(), label: str() } }); + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const peer = await rawParent(client); + let held: Reference | undefined; + const facet = defineFacet((h) => + h.serve(View, (_, self) => { + const ref = self.get(Shifting, {}); + held = ref; + const { x, y } = ref as RefIn; + return { + sum: self.derive([x, y], ([x, y]) => x + y), + // Reads the reference's variant when `y` changes. + label: self.derive([y], ([y]) => `${ref.variant}:${y}`), + }; + }), + ); + const loading = client.load([{ name: "@test/view", facet }], context); + const A = wireNames(Shifting, "a"); + const { hold } = await peer.next("hold"); + const members = [ + { member: A.state.x!, value: "1" }, + { member: A.state.y!, value: "2" }, + ]; + peer.send({ kind: "bound", holds: [{ hold, variant: A.variant, members, failed: [] }] }); + await loading; + const view = await client.get(View, {}, context); + return { client, peer, view, ref: held!, errors }; +} + +forEachCodec(() => { + describe("re-holds after a drop", () => { + it("hold every reference not released again on the next link, with that link's fresh ids, in order", async () => { + const { client, peer } = await rawProvider(); + const refs = ["S1", "S2", "S3"].map((session) => client.hold(Notes, { session }, context)); + for (let i = 0; i < 3; i++) await bindNext(peer); + refs[1]!.release(); + peer.close(); + await transported(); + const next = await rawParent(client); + const holds = [await next.next("hold"), await next.next("hold")]; + expect(holds.map(({ hold, params }) => [hold, partValue(params)])).toEqual([ + [0, { session: "S1" }], + [1, { session: "S3" }], + ]); + }); + + it("apply bound in one batch: each member gets one frame [['r', value]] and a fresh decoder", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + await bindNext(peer, [{ text: "a" }]); + await transported(); + const frames = framesOf(await notes.notes.watch(context)); + const counts = framesOf(await notes.count.watch(context)); + peer.close(); + await transported(); + const next = await rawParent(client); + const hold = await bindNext(next, [{ text: "a" }, { text: "b" }]); + await delivered(); + const list = [{ text: "a" }, { text: "b" }]; + expect(frames).toEqual([[list, [["r", list]]]]); + expect(counts).toEqual([[2, [["r", 2]]]]); + // The fresh decoder: the provider's new encoder starts over, interning paths anew. + next.send({ + kind: "state", + holds: [{ hold, members: [{ member: EDITOR.state.count!, ops: '[["r",3]]' }], failed: [], retired: [] }], + }); + await delivered(); + expect(counts.at(-1)).toEqual([3, [["r", 3]]]); + }); + + it("a derive over two members of a re-held reference sees the snapshot whole: one batch", async () => { + const { client, peer, view } = await shifting(); + const sums = framesOf(await view.sum.watch(context)); + peer.close(); + await transported(); + const next = await rawParent(client); + const { hold } = await next.next("hold"); + const A = wireNames(Shifting, "a"); + const members = [ + { member: A.state.x!, value: "10" }, + { member: A.state.y!, value: "20" }, + ]; + next.send({ kind: "bound", holds: [{ hold, variant: A.variant, members, failed: [] }] }); + await transported(); + expect(sums.map(([sum]) => sum)).toEqual([30]); + }); + + it("another variant: vanished members fail (watches stay open), derives over them fail, new members get values, parked calls are checked (race 14)", async () => { + const { client, peer, view, ref } = await shifting(); + expect(view.sum.value).toBe(3); + const a = ref as RefIn; + const x = changesOf(await a.x.watch(context)); + const y = framesOf(await a.y.watch(context)); + const sum = changesOf(await view.sum.watch(context)); + const labels = framesOf(await view.label.watch(context)); + peer.close(); + await transported(); + const parked = rejection(a.m(context)); + const next = await rawParent(client); + const { hold } = await next.next("hold"); + const B = wireNames(Shifting, "b"); + const members = [ + { member: B.state.y!, value: "20" }, + { member: B.state.z!, value: "30" }, + ]; + next.send({ kind: "bound", holds: [{ hold, variant: B.variant, members, failed: [] }] }); + await transported(); + expect(ref.variant).toBe("b"); + const vanished = { kind: "failed", error: expect.objectContaining({ code: "invalid", reason: "member" }) }; + expect(x).toEqual([vanished]); + expect(sum).toEqual([vanished]); + expect(y).toEqual([[20, [["r", 20]]]]); + // The variant changed in the same batch: the derive recomputed with the new one. + expect(labels.map(([label]) => label)).toEqual(["b:20"]); + expect((ref as RefIn).z.value).toBe(30); + expect(() => a.x.value).toThrow('app.shifting (variant b) has no state member "x"'); + expect(await parked).toEqual({ + code: "invalid", + reason: "member", + message: 'app.shifting (variant b) has no method "m"', + }); + // A later binding of the variant that has `x` brings it back in the same watches; the derive heals. + next.close(); + await transported(); + const third = await rawParent(client); + const again = (await third.next("hold")).hold; + const A = wireNames(Shifting, "a"); + const back = [ + { member: A.state.x!, value: "5" }, + { member: A.state.y!, value: "6" }, + ]; + third.send({ kind: "bound", holds: [{ hold: again, variant: A.variant, members: back, failed: [] }] }); + await transported(); + expect(x.at(-1)).toEqual({ kind: "value", value: 5, ops: [["r", 5]], retired: false }); + expect(view.sum.value).toBe(11); + expect(sum.at(-1)).toEqual({ kind: "value", value: 11, ops: [["r", 11]], retired: false }); + }); + + it("calls sent before a drop fail unavailable, never resent; an old link's leftovers never reach the next (race 1)", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + await bindNext(peer); + const { context: cancellable, cancel } = withCancel(context); + const sent = notes.wait(cancellable); + await peer.next("call"); + peer.close(); + expect(await rejection(sent)).toEqual({ code: "unavailable", message: "the link closed" }); + const next = await rawParent(client); + const hold = await bindNext(next); + await transported(); + cancel(new Error("late")); + notes.release(); + await transported(); + expect(next.received).toEqual([{ kind: "ack" }, { kind: "release", hold }]); + }); + + it("a drop before bound, or during the next link's handshake, leaves it loading; the next connect holds again (race 5)", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + await peer.next("hold"); + peer.close(); + await transported(); + const [a, b] = memoryLinkPair(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + // The peer never answers the handshake; the hold waits in the channel, then the link drops. + b.close(); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + await bindNext(await rawParent(client)); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it("a re-hold refused after a provider upgrade stays failed; the next connect holds it again", async () => { + const NotesV2 = remote("app.notes", { + version: 2, + params: { session: str() }, + variants: { editor: { state: { count: int() } } }, + }); + const { server, client, links } = await linked((h) => void serveNotes(h)); + const upgraded = createHost({ principal: { kind: "server" }, onError: () => {} }); + const facet = defineFacet((h) => + h.serve(NotesV2, () => ({ variant: "editor", members: { count: constant(0) } })), + ); + await upgraded.load([{ name: "@test/app", facet }], context); + const notes = await client.get(Notes, { session: "S1" }, context); + /** Drops the client's link and connects it to `to`. */ + const relink = async (to: Host, from: Link) => { + from.close(); + await transported(); + const [a, b] = memoryLinkPair(); + to.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context); + client.connect(b, { principal: { kind: "server" }, codec: testCodec() }, context); + await transported(); + return a; + }; + const toUpgraded = await relink(upgraded, links[0]); + expect(notes.state.value).toMatchObject({ + kind: "failed", + cause: { kind: "broken", error: { code: "contract_mismatch" } }, + retryAt: undefined, + }); + notes.retry(); + await transported(); + expect(notes.state.value.kind).toBe("failed"); + await relink(server, toUpgraded); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it("a drop while failed: loading, held again on the next link (race 19)", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + const { hold } = await peer.next("hold"); + const error = { code: "unavailable", message: "no configuration" } as const; + peer.send({ kind: "failed", hold, cause: { kind: "broken", error: { kind: "chord", error } } }); + await transported(); + expect(notes.state.value.kind).toBe("failed"); + peer.close(); + await transported(); + expect(notes.state.value).toEqual({ kind: "loading" }); + await bindNext(await rawParent(client)); + await transported(); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + + it("a member watch before the first binding waits for it, and rejects when it fails or is released (race 16)", async () => { + const { client, peer } = await rawProvider(); + const bound = client.hold(Notes, { session: "S1" }, context); + const failing = client.hold(Notes, { session: "S2" }, context); + const released = client.hold(Notes, { session: "S3" }, context); + const watches = [bound.count.watch(context), failing.count.watch(context), released.count.watch(context)]; + await bindNext(peer, [{ text: "a" }]); + const { hold } = await peer.next("hold"); + const error = { code: "unavailable", message: "offline" } as const; + peer.send({ + kind: "failed", + hold, + cause: { kind: "transient", error: { kind: "chord", error } }, + retryIn: 1_000, + }); + released.release(); + expect((await watches[0]!).value).toBe(1); + expect(await rejection(watches[1]!)).toEqual(error); + expect(await rejection(watches[2]!)).toEqual({ code: "released", message: "app.notes: released" }); + }); + }); + + describe("a get across a drop", () => { + fakeTimers(); + + it("keeps waiting within its one deadline, from the moment it was called", async () => { + const { client, peer } = await rawProvider(); + const got = client.get(Notes, { session: "S1" }, context, { readyTimeout: 1_000 }); + const late = rejection(client.get(Notes, { session: "S2" }, context, { readyTimeout: 1_000 })); + await peer.next("hold"); + await peer.next("hold"); + await vi.advanceTimersByTimeAsync(400); + peer.close(); + await transported(); + const next = await rawParent(client); + await bindNext(next, [{ text: "a" }]); + expect((await got).count.value).toBe(1); + await vi.advanceTimersByTimeAsync(600); + expect(await late).toEqual({ + code: "unavailable", + message: "app.notes(S2): did not become live within 1000 ms", + }); + }); + }); + + describe("an activating dependent whose remote dependency's link drops (race 18)", () => { + fakeTimers(); + + const View = local("app.view", { version: 1, params: {}, members: { count: int() } }); + const facet = defineFacet((h) => + h.serve(View, (_, self) => ({ count: self.get(Notes, { session: "S1" }).count })), + ); + + /** Binds the kept hold (0) on `next`; the retry's own hold (1) is bound and the kept one released (R18). */ + async function retried(client: Host, next: RawPeer): Promise { + expect(await bindNext(next, [{ text: "a" }])).toBe(0); + // Retried at once, long before its backoff window (1 s) ends: its new attempt holds anew. + expect(await bindNext(next, [{ text: "a" }])).toBe(1); + expect(await next.next("release")).toEqual({ kind: "release", hold: 0 }); + expect((await client.get(View, {}, context, { readyTimeout: 1 })).count.value).toBe(1); + } + + it("fails transiently at once with the drop, keeps the reference, and is retried at once when it is bound", async () => { + const errors: HostError[] = []; + const client = createHost({ onError: (error) => errors.push(error) }); + const peer = await rawParent(client); + const loading = client.load([{ name: "@test/view", facet }], context); + await peer.next("hold"); + peer.close(); + await loading; + expect(errors).toMatchObject([ + { + kind: "activation_failed", + failure: { + kind: "transient", + error: { + message: "app.view: dependency app.notes(S1) failed", + cause: { message: "the link closed" }, + }, + }, + }, + ]); + await retried(client, await rawParent(client)); + }); + + it("without any link fails at its readyTimeout, keeps the reference, and is retried when the first link binds it", async () => { + const errors: HostError[] = []; + const client = createHost({ readyTimeout: 1_000, onError: (error) => errors.push(error) }); + const loading = client.load([{ name: "@test/view", facet }], context); + await vi.advanceTimersByTimeAsync(1_000); + await loading; + expect(failures(errors)).toEqual(["app.view: transient"]); + await retried(client, await rawParent(client)); + }); + }); + + describe("GC across a drop", () => { + it("a reference that lived through a drop keeps nothing of the closed link", async () => { + const { client, peer } = await rawProvider(); + const notes = client.hold(Notes, { session: "S1" }, context); + await bindNext(peer); + const closed = await (async () => { + const [a, b] = memoryLinkPair(); + peer.close(); + await transported(); + client.connect(a, { principal: { kind: "server" }, codec: testCodec() }, context); + const second = new RawPeer(b); + await second.next("hello"); + await bindNext(second); + second.close(); + await transported(); + return [new WeakRef(a), new WeakRef(second)]; + })(); + await bindNext(await rawParent(client)); + await expectCollected(...closed); + expect(notes.state.value).toEqual({ kind: "live" }); + }); + }); +}); diff --git a/packages/chord/test/next/remote/replica.test.ts b/packages/chord/test/next/remote/replica.test.ts new file mode 100644 index 000000000..e42bb477b --- /dev/null +++ b/packages/chord/test/next/remote/replica.test.ts @@ -0,0 +1,89 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { ChordError } from "../../../src/next/data/index.ts"; +import { Replica } from "../../../src/next/remote/replica.ts"; +import { Batch, Derive } from "../../../src/next/state/internal.ts"; +import { changesOf, delivered, framesOf } from "../support.ts"; + +/** A replica of `{ n }` on a fresh batch, with a derive over it attached. */ +function replica() { + const batch = new Batch(() => {}, undefined); + const source = new Replica(batch, { kind: "value", value: { n: 1 } }); + const double = new Derive([source], ([value]) => (value as { n: number }).n * 2, batch); + double.attach( + (declared) => declared as Replica, + () => {}, + ); + return { batch, source, double }; +} + +describe("replicas", () => { + it("adopt a value and its ops inside a batch: one frame, derives recomputed", async () => { + const { batch, source, double } = replica(); + const frames = framesOf(source.openWatch(context)); + const doubles = framesOf(double.openWatch(context)); + batch.commit(() => source.adopt({ n: 2 }, [["s", ["n"], 2]], false), context); + await delivered(); + expect(frames).toEqual([[{ n: 2 }, [["s", ["n"], 2]]]]); + expect(doubles).toEqual([[4, [["r", 4]]]]); + }); + + it("fail: reads throw the error, watches and derives over it get one failure call and recover with a root replacement", async () => { + const { batch, source, double } = replica(); + const calls = changesOf(source.openWatch(context)); + const derived = changesOf(double.openWatch(context)); + const error = new Error("member failed"); + batch.commit(() => source.fail(error), context); + await delivered(); + expect(calls).toEqual([{ kind: "failed", error }]); + expect(derived).toEqual([{ kind: "failed", error }]); + expect(() => source.value).toThrow("member failed"); + // A non-root op cannot apply to a failed member; its recovery is a root replacement. + expect(() => source.decode([["s", ["n"], 2]])).toThrow("root replacement"); + const { value, ops } = source.decode([["r", { n: 3 }]]); + batch.commit(() => source.adopt(value, ops, false), context); + await delivered(); + expect(calls.at(-1)).toEqual({ kind: "value", value: { n: 3 }, ops: [["r", { n: 3 }]], retired: false }); + expect(derived.at(-1)).toEqual({ kind: "value", value: 6, ops: [["r", 6]], retired: false }); + }); + + it("fail once per episode: a second failure while failed gives no call; a retirement meanwhile waits for the value", async () => { + const { batch, source } = replica(); + const calls = changesOf(source.openWatch(context)); + const first = new Error("first"); + batch.commit(() => source.fail(first), context); + batch.commit(() => source.fail(new Error("second")), context); + await delivered(); + // A retirement alone while failed: the value is not read, the flag waits. + batch.commit(() => source.retireAlone(), context); + const { value, ops } = source.decode([["r", { n: 5 }]]); + batch.commit(() => source.adopt(value, ops, false), context); + await delivered(); + expect(calls).toEqual([ + { kind: "failed", error: first }, + { kind: "value", value: { n: 5 }, ops: [["r", { n: 5 }]], retired: true }, + ]); + }); + + it("open a new watch while failed at the last value, the failure first; with no value yet, reject it", async () => { + const { batch, source } = replica(); + const error = new Error("member failed"); + batch.commit(() => source.fail(error), context); + const watch = source.openWatch(context); + expect(watch.value).toEqual({ n: 1 }); + const calls = changesOf(watch); + await delivered(); + expect(calls).toEqual([{ kind: "failed", error }]); + // Failed since its binding: there is no value to start at. + const unborn = new Replica(batch, { kind: "failed", error }); + expect(() => unborn.openWatch(context)).toThrow(error); + }); + + it("end with their hold: reads throw its error, watches end", async () => { + const { source } = replica(); + const watch = source.openWatch(context); + source.end(new ChordError({ code: "released", message: "app.notes: ended" })); + expect(await watch.closed).toEqual({ kind: "ended" }); + expect(() => source.value).toThrow("app.notes: ended"); + }); +}); diff --git a/packages/chord/test/next/state/batch.test.ts b/packages/chord/test/next/state/batch.test.ts new file mode 100644 index 000000000..c1fd58801 --- /dev/null +++ b/packages/chord/test/next/state/batch.test.ts @@ -0,0 +1,154 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, local, method, obj } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import type { Host, Reference } from "../../../src/next/host/index.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import { Batch, type BatchError } from "../../../src/next/state/internal.ts"; +import { delivered, framesOf, loaded, rejection } from "../support.ts"; + +const Num = defineState("num", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const Probe = local("app.probe", { + version: 1, + params: {}, + members: { probe: int(), fixed: int(), set: method(int()) }, +}); + +/** Every Chord call a derive compute may not make, each refused or allowed. */ +type Attempts = { [operation: string]: unknown }; + +/** + * A host whose `probe` derive runs `attempt` on every compute with n > 0. `attempt` gets the host and a reference to + * the probe itself; what each operation threw is recorded. + */ +async function probe(attempt: (host: Host, self: Reference) => Attempts) { + const seen: Attempts[] = []; + const holder: { host?: Host; reference?: Reference } = {}; + const { host } = await loaded((h) => + h.serve(Probe, (_, self) => { + const num = self.state(Num); + return { + probe: self.derive([num], ([v]) => { + if (v.n > 0) seen.push(attempt(holder.host!, holder.reference!)); + if (v.n > 1) throw new Error("compute threw"); + return v.n; + }), + fixed: constant(1), + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + ); + holder.host = host; + holder.reference = await host.get(Probe, {}, context); + return { host, reference: holder.reference, seen }; +} + +/** What a synchronous throw or a rejection carries, as JSON for ChordErrors. */ +function outcome(run: () => unknown): unknown { + try { + const result = run(); + return result instanceof Promise ? rejection(result) : result; + } catch (error) { + return rejection(Promise.reject(error)); + } +} + +const reentrant = { code: "invalid", reason: "reentrant" }; + +describe("reentrancy", () => { + const attempts = (host: Host, self: Reference): Attempts => ({ + watch: outcome(() => self.probe.watch(context)), + "watch a constant member": outcome(() => self.fixed.watch(context)), + "watch ref.state": outcome(() => self.state.watch(context)), + release: outcome(() => self.release()), + get: outcome(() => host.get(Probe, {}, context)), + load: outcome(() => host.load([], context)), + close: outcome(() => host.close(context)), + set: outcome(() => self.set(5, context)), + }); + + it("refuses watch (ref.state too), release, get, load, close and a transaction in a commit's derive computes", async () => { + const { reference, seen } = await probe(attempts); + await reference.probe.watch(context); + await reference.set(1, context); + for (const result of await Promise.all(Object.values(seen[0]!))) expect(result).toMatchObject(reentrant); + }); + + it("refuses them in a lazy compute outside a commit", async () => { + const { reference, seen } = await probe(attempts); + await reference.set(1, context); + expect(seen).toEqual([]); + expect(reference.probe.value).toBe(1); + for (const result of await Promise.all(Object.values(seen[0]!))) expect(result).toMatchObject(reentrant); + }); + + it("clears the flag after a throwing compute", async () => { + const { host, reference } = await probe(() => ({})); + await reference.set(2, context); + expect(() => reference.probe.value).toThrow("compute threw"); + expect((await host.get(Probe, {}, context)).probe).toBeDefined(); + await expect(reference.set(3, context)).resolves.toBeNull(); + }); +}); + +describe("delivery", () => { + it("reports a delivery that throws and still delivers the later frames of the commit", async () => { + const Other = defineState("other", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); + const Two = local("app.two", { + version: 1, + params: {}, + members: { bad: obj({ a: int() }), other: obj({ n: int() }), set: method(int()) }, + }); + const { host, errors } = await loaded((h) => + h.serve(Two, (_, self) => { + const num = self.state(Num); + const other = self.state(Other); + return { + // `undefined` is no JSON: diffing it throws. + bad: self.derive([num], ([v]) => (v.n === 0 ? { a: 0 } : ({ a: undefined } as never))), + other, + set: async (n, ctx) => { + // `num` is adopted first, so the derive's delivery comes before `other`'s. + await self.tx(async (tx) => { + (await tx.draft(num)).n = n; + (await tx.draft(other)).n = n; + }, ctx); + return null; + }, + }; + }), + ); + const two = await host.get(Two, {}, context); + await two.bad.watch(context); + const frames = framesOf(await two.other.watch(context)); + await two.set(1, context); + await delivered(); + expect(errors).toMatchObject([{ kind: "delivery_failed" }]); + expect(frames.map(([value]) => value)).toEqual([{ n: 1 }]); + }); +}); + +describe("flushes", () => { + it("run once per listing at the end of the commit, after its deliveries; a throwing one is reported", () => { + const reported: BatchError[] = []; + const batch = new Batch((error) => reported.push(error), undefined); + const order: string[] = []; + batch.commit(() => { + batch.flushing(() => order.push("first")); + batch.flushing(() => { + throw new Error("flush threw"); + }); + batch.flushing(() => order.push("third")); + order.push("adopted"); + }, context); + expect(order).toEqual(["adopted", "first", "third"]); + expect(reported).toMatchObject([{ kind: "delivery_failed", error: { message: "flush threw" } }]); + batch.commit(() => {}, context); + expect(order).toEqual(["adopted", "first", "third"]); + }); +}); diff --git a/packages/chord/test/next/state/derive.test.ts b/packages/chord/test/next/state/derive.test.ts new file mode 100644 index 000000000..a6434688f --- /dev/null +++ b/packages/chord/test/next/state/derive.test.ts @@ -0,0 +1,484 @@ +import { describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { arr, int, local, method, obj, str } from "../../../src/next/contracts/index.ts"; +import { defineState, type SingletonHandle } from "../../../src/next/documents/index.ts"; +import type { Self } from "../../../src/next/host/index.ts"; +import { ChordObject } from "../../../src/next/host/object.ts"; +import { mapShared } from "../../../src/next/state/index.ts"; +import { Derive } from "../../../src/next/state/internal.ts"; +import type { Watchable } from "../../../src/watch.ts"; +import { + changesOf, + delivered, + expectCollected, + framesOf, + instancesOf, + loaded, + metered, + restartable, +} from "../support.ts"; + +const objects = instancesOf(ChordObject, "attempt"); + +const Num = defineState("num", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const Store = local("app.store", { + version: 1, + params: {}, + members: { n: int(), double: int(), set: method(int()) }, +}); + +/** A store with a number, its double, and a setter. */ +function serveStore(self: Self) { + const num = self.state(Num); + return { + n: self.derive([num], ([v]) => v.n), + double: self.derive([num], ([v]) => v.n * 2), + set: async (n: number, ctx: typeof context) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; +} + +/** Counts derive computes on a host's adapter. */ +async function computes() { + const telemetry = await metered(); + return { telemetry: telemetry.telemetry, count: () => telemetry.counts()["chord.derive.computes"] ?? 0 }; +} + +describe("derives across objects", () => { + it("update in the batch of the commit that changed their source's object", async () => { + const Viewer = local("app.viewer", { version: 1, params: {}, members: { label: str() } }); + const { host } = await loaded((h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Viewer, (_, self) => ({ + label: self.derive([self.get(Store, {}).double], ([d]) => `double ${d}`), + })); + }); + const store = await host.get(Store, {}, context); + const viewer = await host.get(Viewer, {}, context); + const frames = framesOf(await viewer.label.watch(context)); + await store.set(2, context); + await delivered(); + expect(frames).toEqual([["double 4", [["r", "double 4"]]]]); + }); +}); + +describe("the pull model", () => { + it("computes a diamond's derives once each per batch", async () => { + const { telemetry, count } = await computes(); + const Diamond = local("app.diamond", { version: 1, params: {}, members: { d: int(), set: method(int()) } }); + const { host } = await loaded( + (h) => + h.serve(Diamond, (_, self) => { + const num = self.state(Num); + const b = self.derive([num], ([v]) => v.n + 1); + const c = self.derive([num], ([v]) => v.n + 2); + return { + d: self.derive([b, c], ([x, y]) => x + y), + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + { telemetry }, + ); + const diamond = await host.get(Diamond, {}, context); + const frames = framesOf(await diamond.d.watch(context)); + const before = count(); + await diamond.set(1, context); + expect(count() - before).toBe(3); + await delivered(); + expect(frames.map(([value]) => value)).toEqual([5]); + }); + + it("leaves unwatched derives dirty until read; a one-off read does not make them eager", async () => { + const { telemetry, count } = await computes(); + const { host } = await loaded((h) => h.serve(Store, (_, self) => serveStore(self)), { telemetry }); + const store = await host.get(Store, {}, context); + const before = count(); + await store.set(1, context); + expect(count()).toBe(before); + expect(store.double.value).toBe(2); + expect(store.double.value).toBe(2); + expect(count()).toBe(before + 1); + await store.set(2, context); + expect(count()).toBe(before + 1); + }); + + it("diffs a watched derive's old and new value (counted), and sends no frame when they are equal", async () => { + const telemetry = await metered(); + const Parity = local("app.parity", { + version: 1, + params: {}, + members: { even: obj({ even: int() }), set: method(int()) }, + }); + const { host } = await loaded( + (h) => + h.serve(Parity, (_, self) => { + const num = self.state(Num); + return { + even: self.derive([num], ([v]) => ({ even: v.n % 2 })), + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + { telemetry: telemetry.telemetry }, + ); + const parity = await host.get(Parity, {}, context); + const frames = framesOf(await parity.even.watch(context)); + await parity.set(2, context); + await parity.set(3, context); + await delivered(); + expect(frames).toEqual([[{ even: 1 }, [["s", ["even"], 1]]]]); + expect(telemetry.counts()["chord.derive.diffs"]).toBe(2); + }); +}); + +describe("failures", () => { + it("give the throwing derive's watches and those of the derives over it one failure call; they heal on a later change", async () => { + const { failing, errors } = await failingDerives(); + const [t, u, v] = await Promise.all([ + failing.t.watch(context), + failing.u.watch(context), + failing.v.watch(context), + ]); + const [ts, us] = [changesOf(t), changesOf(u)]; + const frames = framesOf(v); + await failing.set(1, context); + await delivered(); + expect(ts).toEqual([{ kind: "failed", error: THREW }]); + expect(us).toEqual([{ kind: "failed", error: THREW }]); + expect(() => failing.t.value).toThrow(THREW); + expect(() => failing.u.value).toThrow(THREW); + expect(failing.v.value).toBe(1); + await failing.set(2, context); + await delivered(); + // Open all along; the next value replaces the root. + expect(ts.at(-1)).toEqual({ kind: "value", value: 2, ops: [["r", 2]], retired: false }); + expect(us.at(-1)).toEqual({ kind: "value", value: 3, ops: [["r", 3]], retired: false }); + expect(frames.map(([value]) => value)).toEqual([1, 2]); + // The throwing derive's owner reports it, once; the derive over it (its source failed) does not. + expect(errors).toEqual([{ kind: "derive_failed", object: "app.failing", member: "t", error: THREW }]); + }); + + it("give one failure call per episode: a recompute that throws again is quiet; a heal ends the episode", async () => { + const { failing } = await failingDerives(); + const [t, u] = [changesOf(await failing.t.watch(context)), changesOf(await failing.u.watch(context))]; + for (const n of [1, -1, -2, 3, -3]) { + await failing.set(n, context); + await delivered(); + } + const failed = { kind: "failed", error: THREW }; + expect(t).toEqual([failed, { kind: "value", value: 3, ops: [["r", 3]], retired: false }, failed]); + expect(u).toEqual([failed, { kind: "value", value: 4, ops: [["r", 4]], retired: false }, failed]); + }); + + it("open a new watch on a failed derive at its last good value, the failure its first call, then the heal", async () => { + const { failing } = await failingDerives(); + await failing.set(5, context); + // Lazy: the last good value is the last one computed. + expect(failing.u.value).toBe(6); + await failing.set(1, context); + const [t, u] = [await failing.t.watch(context), await failing.u.watch(context)]; + expect([t.value, u.value]).toEqual([5, 6]); + const [ts, us] = [changesOf(t), changesOf(u)]; + await delivered(); + await failing.set(-1, context); + await failing.set(7, context); + await delivered(); + const failed = { kind: "failed", error: THREW }; + expect(ts).toEqual([failed, { kind: "value", value: 7, ops: [["r", 7]], retired: false }]); + expect(us).toEqual([failed, { kind: "value", value: 8, ops: [["r", 8]], retired: false }]); + }); + + it("report once per failure episode, also when nothing watches (a lazy derive fails when read)", async () => { + const { failing, errors } = await failingDerives(); + await failing.set(1, context); + expect(() => failing.t.value).toThrow(THREW); + expect(() => failing.t.value).toThrow(THREW); + await failing.set(-1, context); + expect(() => failing.t.value).toThrow(THREW); + expect(errors).toHaveLength(1); + await failing.set(2, context); + expect(failing.t.value).toBe(2); + await failing.set(1, context); + expect(() => failing.t.value).toThrow(THREW); + expect(errors).toHaveLength(2); + }); + + it("fold a failure and a value for a busy listener: the value replaces the failure, as a root replacement", async () => { + const { failing } = await failingDerives(); + const watch = await failing.t.watch(context); + const calls: unknown[] = []; + const gate = Promise.withResolvers(); + watch.start(async (call) => { + calls.push(call); + if (calls.length === 1) await gate.promise; + }); + await failing.set(5, context); + await delivered(); + await failing.set(1, context); + await failing.set(6, context); + gate.resolve(); + await delivered(); + expect(calls).toEqual([ + { kind: "value", value: 5, ops: [["r", 5]], retired: false }, + { kind: "value", value: 6, ops: [["r", 6]], retired: false }, + ]); + }); +}); + +const THREW = new Error("t threw"); + +/** `app.failing`: `t` throws while `n` is 1 or negative, `u` is `t + 1`, `v` is `n`; the host's reports collected. */ +async function failingDerives() { + const Failing = local("app.failing", { + version: 1, + params: {}, + members: { t: int(), u: int(), v: int(), set: method(int()) }, + }); + const { host, errors } = await loaded((h) => + h.serve(Failing, (_, self) => { + const num = self.state(Num); + const t = self.derive([num], ([v]) => { + if (v.n === 1 || v.n < 0) throw THREW; + return v.n; + }); + return { + t, + u: self.derive([t], ([x]) => x + 1), + v: self.derive([num], ([x]) => x.n), + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + ); + const failing = await host.get(Failing, {}, context); + return { host, failing, errors }; +} + +describe("GC", () => { + it("a detached derive is not held by its former sources", async () => { + const Viewer = local("app.viewer", { version: 1, params: { id: int() }, members: { label: int() } }); + const refs: WeakRef[] = []; + const { host } = await loaded((h) => { + h.serve(Store, (_, self) => serveStore(self)); + h.serve(Viewer, (_, self) => { + const label = self.derive([self.get(Store, {}).n], ([n]) => n); + refs.push(new WeakRef(label)); + return { label }; + }); + }); + const store = await host.get(Store, {}, context); + (await host.get(Viewer, { id: 1 }, context)).release(); + await store.set(1, context); + await expectCollected(...refs); + }); + + it("a watch open on a failed derive keeps nothing more: it ends with its reference, and a retained one holds no source", async () => { + const Paged = local("app.paged", { + version: 1, + params: { id: int() }, + members: { t: int(), set: method(int()) }, + }); + const refs: WeakRef[] = []; + const { host } = await loaded((h) => + h.serve(Paged, (_, self) => { + const num = self.state(Num); + const t = self.derive([num], ([v]) => { + if (v.n === 1) throw THREW; + return v.n; + }); + refs.push(new WeakRef(t), new WeakRef(num)); + return { + t, + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + ); + const paged = await host.get(Paged, { id: 1 }, context); + const watch = await paged.t.watch(context); + const calls = changesOf(watch); + await paged.set(1, context); + await delivered(); + expect(calls).toEqual([{ kind: "failed", error: THREW }]); + paged.release(); + expect(await watch.closed).toEqual({ kind: "ended" }); + // `watch` stays referenced here. + await expectCollected(...refs); + void watch; + await host.close(context); + }); +}); + +describe("GC of handed-out errors", () => { + it("a failure call's fresh error kept by its listener keeps nothing of its host once the host closes", async () => { + const Fresh = local("app.fresh", { version: 1, params: {}, members: { t: int(), set: method(int()) } }); + const refs: WeakRef[] = []; + const kept: unknown[] = []; + await (async () => { + const { host } = await loaded((h) => + h.serve(Fresh, (_, self) => { + const num = self.state(Num); + const t = self.derive([num], ([v]) => { + // A fresh error: its stack's frames hold the derive, the document and the attempt. + if (v.n === 1) throw new Error("fresh"); + return v.n; + }); + refs.push(new WeakRef(t), new WeakRef(num), new WeakRef(self)); + return { + t, + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + ); + refs.push(new WeakRef(host)); + const fresh = await host.get(Fresh, {}, context); + (await fresh.t.watch(context)).start(async (call) => { + if (call.kind === "failed") kept.push(call.error); + }); + await fresh.set(1, context); + await delivered(); + await host.close(context); + })(); + expect(kept).toHaveLength(1); + await expectCollected(...refs); + }); +}); + +describe("mapShared", () => { + it("maps an object element once while it stays the same object; primitives every time", () => { + const f = vi.fn((x: { n: number } | number) => (typeof x === "number" ? x : x.n)); + const map = mapShared(f); + const a = { n: 1 }; + const b = { n: 2 }; + expect(map([a, b, 3])).toEqual([1, 2, 3]); + expect(map([a, { n: 2 }, 3])).toEqual([1, 2, 3]); + expect(f.mock.calls.map(([x]) => x)).toEqual([a, b, 3, { n: 2 }, 3]); + }); + + it("keeps no element alive in its cache", async () => { + const map = mapShared((note: { text: string }) => ({ text: note.text })); + const refs = (() => { + const note = { text: "t" }; + const [result] = map([note]); + return [new WeakRef(note), new WeakRef(result!)]; + })(); + await expectCollected(...refs); + expect(map([{ text: "u" }])).toEqual([{ text: "u" }]); + }); + + it("returns the previous result object for an unchanged element", () => { + const strip = mapShared((note: { text: string; at: number }) => ({ text: note.text })); + const note = { text: "t", at: 1 }; + expect(strip([note])[0]).toBe(strip([note, { text: "u", at: 2 }])[0]); + }); +}); + +const ListV1 = defineState("list", { schema: obj({ items: arr(str()) }), version: 1, initial: () => ({ items: [] }) }); + +describe("the retired flag", () => { + it("a derive over a retired source publishes retired: true although its value and ops are unchanged (rule 21)", async () => { + let list: SingletonHandle<{ items: string[] }> | undefined; + let many: Watchable | undefined; + let twice: Watchable | undefined; + await restartable(objects, (_, self) => { + list = self.state(ListV1); + many = self.derive([list], ([value]) => value.items.length > 1); + twice = self.derive([many], ([value]) => !value); + }); + await list!.change((draft) => void draft.items.push("a"), context); + const manys = changesOf(await many!.watch(context)); + const twices = changesOf(await twice!.watch(context)); + await list!.retire(context); + await delivered(); + expect(manys).toEqual([{ kind: "value", value: false, ops: [], retired: true }]); + expect(twices).toEqual([{ kind: "value", value: true, ops: [], retired: true }]); + await list!.change((draft) => void draft.items.push("a", "b"), context); + await delivered(); + expect(manys.at(-1)).toEqual({ kind: "value", value: true, ops: [["r", true]], retired: false }); + }); + + it("a retirement while the derive is failed travels with its next good value", async () => { + const Gate = defineState("gate", { schema: obj({ on: int() }), version: 1, initial: () => ({ on: 0 }) }); + let num: SingletonHandle<{ n: number }> | undefined; + let gate: SingletonHandle<{ on: number }> | undefined; + let n: Watchable | undefined; + await restartable(objects, (_, self) => { + num = self.state(Num); + gate = self.state(Gate); + n = self.derive([num, gate], ([v, g]) => { + if (g.on === 1 && v.n === 0) throw THREW; + return v.n; + }); + }); + await num!.change((draft) => { + draft.n = 1; + }, context); + await gate!.change((draft) => { + draft.on = 1; + }, context); + const calls = changesOf(await n!.watch(context)); + // The retirement makes `n` 0 again, which throws. + await num!.retire(context); + await num!.change((draft) => { + draft.n = 2; + }, context); + await delivered(); + expect(calls).toEqual([ + { kind: "failed", error: THREW }, + { kind: "value", value: 2, ops: [["r", 2]], retired: true }, + ]); + }); + + it("a retirement visits each derive of a diamond once, not once per path (40 derives)", async () => { + let list: SingletonHandle<{ items: string[] }> | undefined; + let bottom: Watchable | undefined; + await restartable(objects, (_, self) => { + list = self.state(ListV1); + let previous: Watchable = self.derive([list], ([value]) => value.items.length); + // 13 layers of two derives joined by a third: 2^13 paths from the top to the bottom. + for (let layer = 0; layer < 13; layer++) { + const left = self.derive([previous], ([n]) => n + 1); + const right = self.derive([previous], ([n]) => n + 2); + previous = self.derive([left, right], ([a, b]) => a + b); + } + bottom = previous; + }); + await list!.change((draft) => void draft.items.push("a"), context); + const frames = changesOf(await bottom!.watch(context)); + const visits = vi.spyOn(Derive.prototype, "sourceRetired"); + try { + await list!.retire(context); + await delivered(); + expect(visits.mock.calls.length).toBeLessThan(100); + } finally { + visits.mockRestore(); + } + expect(frames.at(-1)).toMatchObject({ retired: true }); + }); +}); diff --git a/packages/chord/test/next/state/watchable.test.ts b/packages/chord/test/next/state/watchable.test.ts new file mode 100644 index 000000000..f59f113c1 --- /dev/null +++ b/packages/chord/test/next/state/watchable.test.ts @@ -0,0 +1,133 @@ +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, local, method, obj } from "../../../src/next/contracts/index.ts"; +import { defineState } from "../../../src/next/documents/index.ts"; +import { resolveWatchable } from "../../../src/next/host/reference.ts"; +import { constant } from "../../../src/next/state/index.ts"; +import type { Source, Subscriber } from "../../../src/next/state/internal.ts"; +import { delivered, expectCollected, framesOf, loaded } from "../support.ts"; + +const Num = defineState("num", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }); +const Counter = local("app.counter", { + version: 1, + params: {}, + members: { num: obj({ n: int() }), set: method(int()) }, +}); + +/** A counter exposing its document, with background errors collected. */ +async function counter() { + const { host, errors } = await loaded((h) => + h.serve(Counter, (_, self) => { + const num = self.state(Num); + return { + num, + set: async (n, ctx) => { + await num.change((draft) => { + draft.n = n; + }, ctx); + return null; + }, + }; + }), + ); + return { counter: await host.get(Counter, {}, context), errors }; +} + +describe("watches", () => { + it("start from the value at acquisition; a listener that keeps up gets each commit", async () => { + const { counter: c } = await counter(); + const watch = await c.num.watch(context); + expect(watch.value).toEqual({ n: 0 }); + const frames = framesOf(watch); + await c.set(1, context); + await delivered(); + await c.set(2, context); + await delivered(); + expect(frames).toEqual([ + [{ n: 1 }, [["s", ["n"], 1]]], + [{ n: 2 }, [["s", ["n"], 2]]], + ]); + }); + + it("give a busy listener one call with the newest value and the folded ops", async () => { + const { counter: c } = await counter(); + const watch = await c.num.watch(context); + const frames: [unknown, unknown][] = []; + let release: () => void = () => {}; + watch.start(async (call) => { + if (call.kind !== "value") return; + frames.push([call.value, call.ops]); + // The first call stays busy until the test lets it return. + if (frames.length === 1) + await new Promise((resolve) => { + release = resolve; + }); + }); + await c.set(1, context); + await delivered(); + await c.set(2, context); + await c.set(3, context); + release(); + await delivered(); + expect(frames).toEqual([ + [{ n: 1 }, [["s", ["n"], 1]]], + // Two commits while busy: one call, the second `s` on the same path replaced the first. + [{ n: 3 }, [["s", ["n"], 3]]], + ]); + }); + + it("end cancelled when their context aborts", async () => { + const { counter: c } = await counter(); + const { context: cancellable, cancel } = withCancel(context); + const watch = await c.num.watch(cancellable); + cancel(); + expect(await watch.closed).toEqual({ kind: "aborted" }); + }); + + it("end with listener_error when the listener throws, reported to onError", async () => { + const { counter: c, errors } = await counter(); + const watch = await c.num.watch(context); + const error = new Error("listener threw"); + watch.start(async () => { + throw error; + }); + await c.set(1, context); + expect(await watch.closed).toEqual({ kind: "listener_error", error }); + expect(errors).toEqual([{ kind: "listener_error", error }]); + }); +}); + +describe("constant", () => { + it("never changes; its watches end only by stop or cancellation and hold nothing after", async () => { + const start = constant({ id: "S1" }); + expect(start.value).toEqual({ id: "S1" }); + const refs = await (async () => { + const watch = await start.watch(context); + expect(watch.value).toBe(start.value); + expect(await watch.stop()).toEqual({ kind: "stopped" }); + return [new WeakRef(watch)]; + })(); + await expectCollected(...refs); + }); +}); + +describe("subscribers", () => { + it("get each batch's frame synchronously, inside the commit, and count as watching", async () => { + const { counter: c } = await counter(); + const source = resolveWatchable(c.num) as Source; + const frames: unknown[] = []; + const subscriber: Subscriber = { + advance: (call) => void (call.kind === "value" && frames.push([call.value, call.ops])), + end: () => {}, + }; + source.subscribe(subscriber); + expect(source.watched).toBe(true); + const setting = c.set(1, context); + await setting; + expect(frames).toEqual([[{ n: 1 }, [["s", ["n"], 1]]]]); + source.unsubscribe(subscriber); + expect(source.watched).toBe(false); + await c.set(2, context); + expect(frames).toHaveLength(1); + }); +}); diff --git a/packages/chord/test/next/store/applier.test.ts b/packages/chord/test/next/store/applier.test.ts new file mode 100644 index 000000000..00d8e61f5 --- /dev/null +++ b/packages/chord/test/next/store/applier.test.ts @@ -0,0 +1,114 @@ +// Tests of the applier (slice 6a §13): recorded writes applied as recorded, checked against the store's own records: +// table names and columns by the name rules, indexes, rows against their recorded columns (NULL cells, defaults kept +// as stored, booleans, json(...) cells as JSON text) and documents with their ids kept. + +import { describe, expect, it } from "vitest"; +import { type AppliedWrite, applyWrite } from "../../../src/next/store/applier.ts"; +import { Step } from "../../../src/next/store/step.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { rejection } from "../support.ts"; + +const TABLE = "client:app.store:edits"; + +const table: AppliedWrite = { + kind: "table", + table: TABLE, + version: 2, + state: "ready", + columns: { + id: { type: "str" }, + done: { type: "bool", default: false }, + note: { type: "str", optional: true }, + meta: { type: "json", optional: true, version: 1, fingerprint: "f" }, + }, + key: ["id"], + fingerprint: "t", +}; + +/** Applies `writes` in one transaction of a fresh database; resolves with the rows of the table. */ +async function applied(...writes: AppliedWrite[]): Promise { + const db = await openNodeSqliteDatabase(":memory:"); + try { + return await db.transaction(async (executor) => { + const step = new Step(executor, 1, 1, new Map()); + for (const write of writes) await applyWrite(step, write, "test"); + return executor.all(`SELECT * FROM "${TABLE}"`); + }); + } finally { + await db.close(); + } +} + +describe("applied writes", () => { + it("a table and its rows as recorded: NULL for absent opt cells, booleans as 0 and 1, json(...) as text", async () => { + const rows = await applied(table, { + kind: "put", + table: TABLE, + row: { id: "a", done: true, meta: { x: [1] } }, + version: 1, + columns: { meta: 1 }, + }); + expect(rows).toEqual([{ id: "a", done: 1, note: null, meta: '{"x":[1]}', meta$v: 1, $v: 1 }]); + }); + + it.each<[string, AppliedWrite[], string]>([ + [ + "a put for a table not recorded", + [{ kind: "put", table: TABLE, row: {}, version: 1, columns: {} }], + "is not recorded", + ], + [ + "a key over an opt column", + [{ ...table, key: ["note"] } as AppliedWrite], + '"note" is not a non-opt str or int column', + ], + [ + "a json column without a fingerprint", + [{ ...table, columns: { id: { type: "str" }, meta: { type: "json", version: 1 } } } as AppliedWrite], + "a json(...) column has a version and a fingerprint", + ], + [ + "a default of another type", + [{ ...table, columns: { id: { type: "str" }, n: { type: "int", default: "x" } } } as AppliedWrite], + "is not a int value", + ], + [ + "an index over an unknown column", + [table, { kind: "index", table: TABLE, name: "by_x", columns: ["x"] }], + '"x" is not a non-opt str or int column', + ], + [ + "an index twice", + [ + table, + { kind: "index", table: TABLE, name: "by_id", columns: ["id"] }, + { kind: "index", table: TABLE, name: "by_id", columns: ["id"] }, + ], + "not a new index name", + ], + [ + "a version for a cell the row lacks", + [table, { kind: "put", table: TABLE, row: { id: "a", done: false }, version: 1, columns: { meta: 1 } }], + "no json(...) cell of the row", + ], + [ + "a json cell that is no JSON", + [ + table, + { + kind: "put", + table: TABLE, + row: { id: "a", done: false, meta: 1n as never }, + version: 1, + columns: { meta: 1 }, + }, + ], + "not JSON", + ], + ])("refuses %s", async (_, writes, message) => { + expect(await rejection(applied(...writes))).toMatchObject({ + code: "invalid", + message: expect.stringContaining(message), + }); + }); +}); diff --git a/packages/chord/test/next/store/archive.test.ts b/packages/chord/test/next/store/archive.test.ts new file mode 100644 index 000000000..3df415558 --- /dev/null +++ b/packages/chord/test/next/store/archive.test.ts @@ -0,0 +1,42 @@ +// Tests of the archive part schema (slice 6a §13, rule 25): every part an export yields passes; a part with an +// unknown kind, a missing or extra field or a value of another type is refused, naming its position. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { checkPart } from "../../../src/next/store/archive.ts"; +import { archiveStore, exported } from "../support.ts"; + +describe("the archive part schema", () => { + it("takes every part an export yields", async () => { + const store = await archiveStore(); + const parts = await exported(store); + for (const [position, part] of parts.entries()) expect(checkPart(part, position, context)).toBe(part); + await store.close(context); + }); + + it.each<[string, unknown]>([ + ["an unknown kind", { kind: "blob" }], + ["a missing field", { kind: "end" }], + ["an extra field", { kind: "end", writes: 0, more: 1 }], + ["a negative count", { kind: "end", writes: -1 }], + [ + "a column of an unknown type", + { + kind: "table", + table: "a:b:c", + version: 1, + state: "ready", + columns: { a: { type: "date" } }, + key: ["a"], + fingerprint: "f", + }, + ], + ["a put at version 0", { kind: "put", table: "a:b:c", row: {}, version: 0, columns: {} }], + [ + "a record without an id", + { kind: "document.create", record: { namespace: "n" }, base: { version: 1, fingerprint: "f", value: {} } }, + ], + ])("refuses %s", (_, part) => { + expect(() => checkPart(part, 4, context)).toThrow("import: part 4: not an archive part"); + }); +}); diff --git a/packages/chord/test/next/store/catalog.test.ts b/packages/chord/test/next/store/catalog.test.ts new file mode 100644 index 000000000..451c45cb2 --- /dev/null +++ b/packages/chord/test/next/store/catalog.test.ts @@ -0,0 +1,65 @@ +// The catalog (slice 6a §8): the store's records of its tables, read once at open, changed after COMMIT returned. + +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, str } from "../../../src/next/contracts/index.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import { type ChordStore, openStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { stageTable } from "../../../src/next/store/table-writes.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { memoryStore, tableIn } from "../support.ts"; + +const Edits = defineTable("edits", { + version: 1, + columns: { id: str(), at: int() }, + key: ["id"], + indexes: { by_at: ["at"] }, +}); +const TABLE = "session:app.store:edits"; + +describe("the catalog", () => { + it("is read once at open: a reopened store knows its tables", async () => { + const directory = await mkdtemp(join(tmpdir(), "chord-catalog-")); + const path = join(directory, "store.sqlite"); + const first = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context)) as ChordStore; + await tableIn(first, Edits); + await first.close(context); + const second = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context)) as ChordStore; + expect(second.table(TABLE)).toEqual({ + table: TABLE, + version: 1, + state: "ready", + columns: Edits.record, + key: ["id"], + fingerprint: Edits.fingerprint, + indexes: { by_at: ["at"] }, + }); + await second.close(context); + await rm(directory, { recursive: true, force: true }); + }); + + it("changes in memory only after COMMIT returned", async () => { + const store = await memoryStore(); + const { version, record: columns, key, fingerprint } = Edits; + const failing: StagedWrite = { + order: "write", + apply: (step) => + step + .run("INSERT INTO no_such_table VALUES (1)") + .then(() => ({ kind: "dropIndex", table: TABLE, name: "x" })), + }; + const table = stageTable({ kind: "table", table: TABLE, version, state: "ready", columns, key, fingerprint }); + await expect( + store.commit([table, failing], { admit: () => true, adopt: () => {} }, context), + ).rejects.toMatchObject({ + reason: "storage_failed", + }); + expect(store.table(TABLE)).toBeUndefined(); + await store.close(context); + }); +}); diff --git a/packages/chord/test/next/store/columns.test.ts b/packages/chord/test/next/store/columns.test.ts new file mode 100644 index 000000000..48e11981c --- /dev/null +++ b/packages/chord/test/next/store/columns.test.ts @@ -0,0 +1,73 @@ +// Table columns (decisions "Flat columns", "Nullable columns and the `set` form"): descriptors and row types. + +import { describe, expect, it } from "vitest"; +import { arr, bool, int, num, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { + type Columns, + columnRecord, + describeColumns, + type PutRowOf, + type RowOf, +} from "../../../src/next/store/columns.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import type { Equal } from "../support.ts"; + +describe("column descriptors", () => { + it("describes the helpers in declaration order", () => { + const columns = describeColumns( + { + id: str(), + n: int({ default: 0 }), + x: num(), + b: bool(), + note: opt(str()), + meta: opt(json(obj({}), { version: 2 })), + }, + "t", + ); + expect(columns.map(({ name, type, optional }) => [name, type, optional])).toEqual([ + ["id", "str", false], + ["n", "int", false], + ["x", "num", false], + ["b", "bool", false], + ["note", "str", true], + ["meta", "json", true], + ]); + expect(columns.map(columnRecord)).toEqual([ + { type: "str" }, + { type: "int", default: 0 }, + { type: "num" }, + { type: "bool" }, + { type: "str", optional: true }, + { type: "json", optional: true, version: 2, fingerprint: expect.any(String) }, + ]); + }); + + it("checks names: the pattern, no two equal ignoring case, no $", () => { + expect(() => describeColumns({ "a-b": str() }, "t")).toThrow("[A-Za-z_][A-Za-z0-9_]*"); + expect(() => describeColumns({ a$v: str() }, "t")).toThrow('column "a$v"'); + expect(() => describeColumns({ path: str(), Path: str() }, "t")).toThrow("only in case"); + expect(describeColumns({ camelCase: str(), _x1: int() }, "t")).toHaveLength(2); + }); + + it("refuses other schemas, bounds, patterns, opt defaults and defaults of another type", () => { + const columns = (value: unknown) => ({ c: value }) as unknown as Columns; + expect(() => describeColumns(columns(arr(str())), "t")).toThrow("expected str()"); + expect(() => describeColumns(columns(obj({})), "t")).toThrow("expected str()"); + expect(() => describeColumns(columns(str({ max: 3 })), "t")).toThrow('"maxLength" is not supported'); + expect(() => describeColumns(columns(str({ pattern: "a" })), "t")).toThrow('"pattern" is not supported'); + expect(() => describeColumns(columns(opt(int({ default: 1 }))), "t")).toThrow("no default"); + expect(() => describeColumns(columns(int({ default: 1.5 })), "t")).toThrow("not a int value"); + expect(() => describeColumns(columns(str({ default: "a\0" })), "t")).toThrow("NUL"); + }); +}); + +describe("row types", () => { + it("types rows with optional opt(...) fields and puts with optional defaulted fields", () => { + const shape = { id: str(), hits: int({ default: 0 }), note: opt(str()) }; + type S = typeof shape; + const row: Equal, { id: string; hits: number } & { note?: string }> = true; + const put: Equal, { id: string } & { hits?: number; note?: string }> = true; + expect([row, put]).toEqual([true, true]); + }); +}); diff --git a/packages/chord/test/next/store/engine.test.ts b/packages/chord/test/next/store/engine.test.ts new file mode 100644 index 000000000..78a41dab0 --- /dev/null +++ b/packages/chord/test/next/store/engine.test.ts @@ -0,0 +1,142 @@ +// The engine (slice 6a §10, §11): persisted documents as bases and deltas, their keys in code point order, retirement. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import type { Op } from "../../../src/delta/index.ts"; +import type { ArchivePart } from "../../../src/next/store/archive.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import { + changeDocument, + createDocument, + type DocumentAddress, + findDocument, + listDocumentKeys, + loadDocument, + retireDocument, +} from "../../../src/next/store/engine.ts"; +import { importStore } from "../../../src/next/store/import.ts"; +import { TABLES } from "../../../src/next/store/schema.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import { type ChordStore, openStore } from "../../../src/next/store/store.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { exported, rejection } from "../support.ts"; + +const OWNER = { namespace: "chord", owner: "session:app.store", kind: "notes" }; + +/** A store on `:memory:` (or `on`) and a commit helper resolving with the created ids. */ +async function engine(on?: ChordStore) { + const store = + on ?? ((await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore); + const commit = async (...writes: StagedWrite[]): Promise => { + let created: readonly number[] = []; + await store.commit( + writes, + { + admit: () => true, + adopt: (_seq, ids) => { + created = ids; + }, + }, + context, + ); + return created; + }; + const at = (key: string): DocumentAddress => ({ ...OWNER, key }); + const base = (value: object, version = 1) => ({ + version, + fingerprint: `f${version}`.padEnd(16, "0"), + value: value as never, + }); + const create = (key: string, value: object) => createDocument(at(key), "latest", base(value), JSON.stringify(value)); + const delta = (id: number, revision: number, ops: Op[]) => + changeDocument(id, revision, { kind: "delta", version: 1, ops }, JSON.stringify(ops)); + const read = (fn: Parameters[0]) => store.read(fn, context) as Promise; + return { store, commit, at, base, create, delta, read }; +} + +/** `store`'s archive with `edit` applied to its parts, imported into a fresh `:memory:` store. */ +async function reimported(store: ChordStore, edit: (parts: ArchivePart[]) => unknown[]): Promise { + const parts = edit(await exported(store)); + return (await importStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), parts, context)) as ChordStore; +} + +const LIMIT = Number.MAX_SAFE_INTEGER; + +describe("documents", () => { + it("load as their base with the deltas after it applied; a new base prunes the older revisions", async () => { + const { commit, at, base, create, delta, read } = await engine(); + const [id] = await commit(create("a", { n: 0 })); + await commit(delta(id!, 1, [["s", ["n"], 1]])); + await commit(delta(id!, 2, [["s", ["n"], 2]])); + const record = (await read>>((e) => findDocument(e, at("a"))))!; + expect(record).toMatchObject({ id, revision: 3, version: 1, history: "latest", createdAt: 1 }); + expect(await read((e) => loadDocument(e, record))).toMatchObject({ value: { n: 2 }, deltasSinceBase: 2 }); + const next = base({ m: 9 }, 2); + await commit(changeDocument(id!, 3, { kind: "base", ...next }, JSON.stringify(next.value))); + const rebased = (await read>>((e) => findDocument(e, at("a"))))!; + expect(rebased).toMatchObject({ revision: 4, version: 2, fingerprint: next.fingerprint }); + expect(await read((e) => loadDocument(e, rebased))).toMatchObject({ value: { m: 9 }, deltasSinceBase: 0 }); + expect(await read((e) => e.all(`SELECT seq FROM ${TABLES.revisions}`))).toEqual([{ seq: 4 }]); + }); + + it("list live keys in code point order, paging after a key", async () => { + const { commit, create, read } = await engine(); + await commit(create("\u{1F600}", {}), create("\uFF5E", {}), create("b", {}), create("a", {})); + expect(await read((e) => listDocumentKeys(e, OWNER, undefined, 10))).toEqual(["a", "b", "\uFF5E", "\u{1F600}"]); + expect(await read((e) => listDocumentKeys(e, OWNER, "b", 1))).toEqual(["\uFF5E"]); + }); + + it("retire a latest record with its revisions (rule 19); the address takes a new incarnation", async () => { + const { commit, at, create, read } = await engine(); + const [id] = await commit(create("a", { n: 0 })); + const [next] = await commit(retireDocument(id!, 1), create("a", { n: 1 })); + expect(next).toBe(2); + expect(await read((e) => e.all(`SELECT id FROM ${TABLES.documents}`))).toEqual([{ id: 2 }]); + expect(await read((e) => e.all(`SELECT document_id FROM ${TABLES.revisions}`))).toEqual([{ document_id: 2 }]); + expect(await read((e) => findDocument(e, at("a")))).toMatchObject({ id: 2 }); + }); + + it("refuse two live records at one address and a delta on another version: storage_failed", async () => { + const first = await engine(); + await first.commit(first.create("a", {})); + expect(await rejection(first.commit(first.create("a", {})))).toMatchObject({ reason: "storage_failed" }); + const second = await engine(); + const [id] = await second.commit(second.create("a", {})); + const crossing = changeDocument(id!, 1, { kind: "delta", version: 2, ops: [] }, "[]"); + expect(await rejection(second.commit(crossing))).toMatchObject({ reason: "storage_failed" }); + }); +}); + +describe("counters at the safe-integer limit", () => { + it("allocate document ids up to the safe range after an import near it; past it a commit fails storage_failed", async () => { + const { store: source } = await engine(); + const near = (next: number) => + reimported(source, ([manifest, ...rest]) => [{ ...manifest, nextDocumentId: next }, ...rest]); + const { store, commit, create } = await engine(await near(LIMIT - 1)); + expect(await commit(create("a", { n: 0 }))).toEqual([LIMIT - 1]); + expect(store.nextDocumentId).toBe(LIMIT); + expect(await rejection(commit(create("b", { n: 0 })))).toMatchObject({ reason: "storage_failed" }); + // Two creations in one commit where only one id is left: nothing of it is written. + const both = await engine(await near(LIMIT - 1)); + expect(await rejection(both.commit(both.create("a", { n: 0 }), both.create("b", { n: 0 })))).toMatchObject({ + reason: "storage_failed", + }); + expect([both.store.lastSeq, both.store.nextDocumentId]).toEqual([0, LIMIT - 1]); + }); + + it("count revisions up to the safe range after an import near it; a change past it fails storage_failed", async () => { + const { store: source, commit: seed, create } = await engine(); + await seed(create("a", { n: 0 })); + const store = await reimported(source, (parts) => + parts.map((part) => + part.kind === "document.create" ? { ...part, record: { ...part.record, revision: LIMIT - 1 } } : part, + ), + ); + const { commit, delta, read, at } = await engine(store); + const record = (await read>>((e) => findDocument(e, at("a"))))!; + await commit(delta(record.id, LIMIT - 1, [["s", ["n"], 1]])); + expect(await rejection(commit(delta(record.id, LIMIT, [["s", ["n"], 2]])))).toMatchObject({ + reason: "storage_failed", + }); + }); +}); diff --git a/packages/chord/test/next/store/evaluate.test.ts b/packages/chord/test/next/store/evaluate.test.ts new file mode 100644 index 000000000..9f94e2b94 --- /dev/null +++ b/packages/chord/test/next/store/evaluate.test.ts @@ -0,0 +1,211 @@ +// Tests of the JavaScript evaluator of the table vocabulary, and that it agrees with SQLite on the same nodes. +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { bool, int, num, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { applySet, type EvaluatedRow, matches } from "../../../src/next/store/evaluate.ts"; +import { + add, + and, + coalesce, + compileSet, + compileWhere, + concat, + eq, + gt, + gte, + inArray, + isNull, + lt, + lte, + max, + min, + ne, + not, + or, + prefix, + replacePrefix, + sub, + type WhereNode, + when, +} from "../../../src/next/store/expression.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import type { ColumnRefs } from "../../../src/next/store/query.ts"; +import { createTableSql, TableSql, whereSql } from "../../../src/next/store/sql.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import type { SqliteValue } from "../../../src/sqlite/index.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const T = defineTable("t", { + version: 1, + columns: { + id: str(), + path: str(), + n: int(), + x: num(), + b: bool(), + note: opt(str()), + count: opt(int()), + meta: opt(json(obj({ a: int() }), { version: 1 })), + }, + key: ["id"], +}); +type C = typeof T extends { "~columns"?: infer X } ? NonNullable : never; +const refs = T.refs as ColumnRefs; +const args = expect.objectContaining({ code: "invalid", reason: "args" }); +describe("the evaluator", () => { + const row: EvaluatedRow = { id: "r", path: "a/x", n: 2, x: 1, b: true, count: 3 }; + + it("follows SQL's NULL rules: a comparison with NULL never matches, not and ne included", () => { + const where = (condition: unknown) => compileWhere(T, condition, "where"); + expect(matches(where(eq(refs.note, "a")), row)).toBe(false); + expect(matches(where(ne(refs.note, "a")), row)).toBe(false); + expect(matches(where(not(eq(refs.note, "a"))), row)).toBe(false); + expect(matches(where(or(isNull(refs.note), ne(refs.note, "a"))), row)).toBe(true); + expect(matches(where(and()), row)).toBe(true); + expect(matches(where(or()), row)).toBe(false); + expect(matches(where(inArray(refs.n, [])), row)).toBe(false); + }); + + it("evaluates every right side against the row before (a swap)", () => { + const Swap = defineTable("s", { version: 1, columns: { id: str(), a: str(), b: str() }, key: ["id"] }); + const set = compileSet( + Swap, + { a: (Swap.refs as { b: unknown }).b, b: (Swap.refs as { a: unknown }).a }, + "update", + ); + expect(applySet(set, { id: "1", a: "x", b: "y" }, "s")).toEqual({ id: "1", a: "y", b: "x" }); + }); + + it("leaves a value without the prefix unchanged and refuses integers outside the safe range (rule 10)", () => { + expect(applySet(compileSet(T, { path: replacePrefix(refs.path, "b/", "c/") }, "u"), row, "t").path).toBe("a/x"); + expect(applySet(compileSet(T, { path: replacePrefix(refs.path, "a/", "") }, "u"), row, "t").path).toBe("x"); + const big = { ...row, n: Number.MAX_SAFE_INTEGER }; + expect(() => applySet(compileSet(T, { n: add(refs.n, 1) }, "u"), big, "t")).toThrow(args); + }); +}); + +describe("one semantics: the evaluator agrees with SQLite", () => { + const ROWS: EvaluatedRow[] = [ + { id: "1", path: "a/x", n: 1, x: 0.5, b: true, note: "z", count: 1 }, + { id: "2", path: "a/\u{1F600}", n: 2, x: 2, b: false }, + { id: "3", path: "a\uFFFF", n: -3, x: -1, b: true, note: "" }, + { id: "4", path: "b", n: 0, x: 0, b: false, count: 0 }, + { id: "5", path: "", n: 9, x: 9.5, b: true, note: "a/" }, + { id: "6", path: "\u{10FFFF}", n: 5, x: 1, b: false, count: -2 }, + ]; + const CONDITIONS = [ + eq(refs.note, "z"), + ne(refs.note, "z"), + not(ne(refs.count, 0)), + lt(refs.path, "a\uFFFF"), + gte(refs.path, "a/\u{1F600}"), + prefix(refs.path, "a"), + prefix(refs.path, ""), + prefix(refs.path, "\u{10FFFF}"), + inArray(refs.path, ["b", "", "x"]), + inArray(refs.b, [true]), + or(isNull(refs.count), gt(refs.x, 1)), + and(not(isNull(refs.note)), lte(refs.n, 1)), + not(or(eq(refs.b, true), lt(refs.count, 0))), + ]; + + async function table() { + const db = await openNodeSqliteDatabase(":memory:"); + await db.run(createTableSql("t", T.record, T.key)); + const sql = new TableSql("t", T.columns, T.rows.physical, T.key); + for (const row of ROWS) await db.run(sql.put, ...T.rows.put(row, 1).params); + return { db, sql }; + } + + it("selects the rows the evaluator matches", async () => { + const { db } = await table(); + for (const condition of CONDITIONS) { + const node = compileWhere(T, condition, "where"); + const params: SqliteValue[] = []; + const text = whereSql(node, params); + const rows = await db.all<{ id: string }>(`SELECT id FROM t WHERE ${text} ORDER BY id`, ...params); + const expected = ROWS.filter((row) => matches(node, row)).map(({ id }) => id); + expect([JSON.stringify(node), rows.map(({ id }) => id)]).toEqual([JSON.stringify(node), expected]); + } + await db.close(); + }); + + it("agrees on every operator's NULL truth under not, and, or and when", async () => { + const PREDICATES = [ + eq(refs.note, "z"), + ne(refs.note, "z"), + lt(refs.count, 1), + lte(refs.count, 0), + gt(refs.note, "a"), + gte(refs.count, 0), + inArray(refs.note, ["z", ""]), + inArray(refs.note, []), + prefix(refs.note, "a"), + prefix(refs.note, ""), + isNull(refs.count), + ]; + const WRAPPED = PREDICATES.flatMap((p) => [ + p, + not(p), + and(p, eq(refs.b, true)), + or(p, eq(refs.b, true)), + not(and(p, eq(refs.b, true))), + not(or(p, eq(refs.b, false))), + ]); + const { db } = await table(); + for (const condition of WRAPPED) { + const node = compileWhere(T, condition, "where"); + const params: SqliteValue[] = []; + const text = whereSql(node, params); + const rows = await db.all<{ id: string }>(`SELECT id FROM t WHERE ${text} ORDER BY id`, ...params); + const expected = ROWS.filter((row) => matches(node, row)).map(({ id }) => id); + expect([JSON.stringify(node), rows.map(({ id }) => id)]).toEqual([JSON.stringify(node), expected]); + } + await db.close(); + for (const condition of WRAPPED) { + const { db, sql } = await table(); + const node = compileSet(T, { n: when(condition, 1, 2) }, "updateWhere"); + const statement = sql.updateWhere({ and: [] } as WhereNode, node, new Map()); + await db.run(statement.sql, ...statement.params); + const stored = await db.all<{ [column: string]: SqliteValue }>(`SELECT * FROM t ORDER BY id`); + const read = stored.map((row) => T.rows.decode(row, context)); + expect([JSON.stringify(node), read]).toEqual([ + JSON.stringify(node), + ROWS.map((row) => applySet(node, row, "t")), + ]); + await db.close(); + } + }); + + it("binds a long inArray list as one parameter (beyond SQLite's 32,766 variables)", async () => { + const { db } = await table(); + const values = Array.from({ length: 40_000 }, (_, i) => `v${i}`); + values.push("b"); + const node = compileWhere(T, inArray(refs.path, values), "where"); + const params: SqliteValue[] = []; + const text = whereSql(node, params); + expect(params).toHaveLength(1); + const rows = await db.all<{ id: string }>(`SELECT id FROM t WHERE ${text}`, ...params); + expect(rows.map(({ id }) => id)).toEqual(["4"]); + await db.close(); + }); + + it("updates rows as the evaluator does", async () => { + const SETS = [ + { n: add(refs.n, 10), count: sub(coalesce(refs.count, 5), 1) }, + { path: replacePrefix(refs.path, "a/", "\u{1F600}/"), note: concat(refs.path, "!") }, + { x: min(refs.x, 1), n: max(refs.n, 1), note: when(prefix(refs.path, "a"), refs.path, refs.note) }, + { note: refs.path, path: coalesce(refs.note, "none"), count: undefined }, + ]; + for (const set of SETS) { + const { db, sql } = await table(); + const node = compileSet(T, set, "updateWhere"); + const statement = sql.updateWhere({ and: [] } as WhereNode, node, new Map()); + await db.run(statement.sql, ...statement.params); + const stored = await db.all<{ [column: string]: SqliteValue }>(`SELECT * FROM t ORDER BY id`); + const read = stored.map((row) => T.rows.decode(row, context)); + expect(read).toEqual(ROWS.map((row) => applySet(node, row, "t"))); + await db.close(); + } + }); +}); diff --git a/packages/chord/test/next/store/export.test.ts b/packages/chord/test/next/store/export.test.ts new file mode 100644 index 000000000..e0e743023 --- /dev/null +++ b/packages/chord/test/next/store/export.test.ts @@ -0,0 +1,230 @@ +// Tests of `store.export(ctx)` (slice 6a §13, rule 26): the parts in archive order, lossless; the commit queue held +// from the first `next()` until the export ends, is closed or aborts (also while the consumer is suspended); GC of +// closed and aborted iterators and their abort listeners. + +import { getEventListeners } from "node:events"; +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { int, str } from "../../../src/next/contracts/index.ts"; +import { ARCHIVE_FORMAT } from "../../../src/next/store/archive.ts"; +import { EXPORT_PAGE } from "../../../src/next/store/export.ts"; +import { STORE_SCHEMA } from "../../../src/next/store/schema.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import { + archiveStore, + expectCollected, + exported, + flushed, + memoryStore, + rejection, + settledHeap, + tableIn, +} from "../support.ts"; + +const OPEN = { admit: () => true, adopt: () => {} }; + +let store: ChordStore; +afterEach(async () => { + await store?.close(context); +}); + +describe("parts", () => { + it("come in archive order: manifest, per table its table write, puts and indexes, the live documents, end", async () => { + store = await archiveStore(); + const parts = await exported(store); + expect(parts[0]).toEqual({ + kind: "manifest", + format: ARCHIVE_FORMAT, + schema: STORE_SCHEMA, + lastSeq: store.lastSeq, + nextDocumentId: store.nextDocumentId, + }); + const kinds = parts.map((part) => + part.kind === "put" || part.kind === "table" || part.kind === "index" + ? `${part.kind} ${part.table.split(":")[2]}` + : part.kind, + ); + const runs: [string, number][] = []; + for (const kind of kinds) { + const last = runs.at(-1); + if (last?.[0] === kind) last[1]++; + else runs.push([kind, 1]); + } + expect(runs).toEqual([ + ["manifest", 1], + ["table edits", 1], + ["put edits", 150], + ["index edits", 1], + ["table notes", 1], + ["put notes", 150], + ["document.create", 3], + ["end", 1], + ]); + expect(parts.at(-1)).toEqual({ kind: "end", writes: parts.length - 2 }); + }); + + it("are lossless: stored values and versions, NULL cells absent, a migrating table's state, deltas folded into the base", async () => { + store = await archiveStore(); + const parts = await exported(store); + const edits = parts.filter((part) => part.kind === "put" && part.table.endsWith(":edits")); + expect(edits.slice(0, 2)).toEqual([ + { + kind: "put", + table: "client:app.store:edits", + row: { id: "e000", path: "p/0", at: 0, note: "n0", meta: { a: 0 }, done: false }, + version: 1, + columns: { meta: 1 }, + }, + { + kind: "put", + table: "client:app.store:edits", + row: { id: "e001", path: "p/1", at: 1, done: false }, + version: 1, + columns: {}, + }, + ]); + expect(parts.find((part) => part.kind === "table" && part.table.endsWith(":notes"))).toMatchObject({ + version: 2, + state: "migrating", + }); + const notes = parts.filter((part) => part.kind === "put" && part.table.endsWith(":notes")); + expect(notes.map((part) => (part.kind === "put" ? part.version : 0)).filter((v) => v === 2)).toHaveLength(100); + const documents = parts.filter((part) => part.kind === "document.create"); + expect( + documents.map((part) => + part.kind === "document.create" ? [part.record.key, part.record.revision, part.base.value] : [], + ), + ).toEqual([ + // A singleton's record has no key. + [undefined, 1, { title: "" }], + ["a", 3, { title: "a3" }], + ["b", 2, { title: "b2" }], + ]); + }); +}); + +describe("the commit queue", () => { + it("is held from the first next() until the export ends: a commit waits", async () => { + store = await archiveStore(); + const iterator = store.export(context); + await iterator.next(); + let committed = false; + const commit = store.commit([], OPEN, context).then(() => { + committed = true; + }); + await flushed(); + expect(committed).toBe(false); + while (!(await iterator.next()).done) {} + await commit; + expect(committed).toBe(true); + }); + + it("is given back by return(); the closed iterator keeps nothing", async () => { + store = await archiveStore(); + let iterator: AsyncIterableIterator | undefined = store.export(context); + await iterator.next(); + await iterator.next(); + await iterator.return!(); + await store.commit([], OPEN, context); + const collected = new WeakRef(iterator); + iterator = undefined; + await expectCollected(collected); + }); + + it("is given back at once when ctx aborts while the consumer is suspended; a later next() rejects aborted", async () => { + store = await archiveStore(); + const { context: cancellable, cancel } = withCancel(context); + const iterator = store.export(cancellable); + await iterator.next(); + const signal = cancellable.abortSignal!; + expect(getEventListeners(signal, "abort")).toHaveLength(1); + cancel(new Error("stop")); + await store.commit([], OPEN, context); + expect(getEventListeners(signal, "abort")).toHaveLength(0); + expect(await rejection(iterator.next())).toMatchObject({ code: "aborted" }); + }); + + it("a retained aborted iterator keeps no page", async () => { + store = await memoryStore(); + const Big = defineTable("big", { version: 1, columns: { id: int(), text: str() }, key: ["id"] }); + const staged: StagedWrite[] = []; + const changes = new TableChanges(await tableIn(store, Big), staged); + for (let i = 0; i < EXPORT_PAGE; i++) changes.put({ id: i, text: `${i}`.padEnd(8_000, "x") }); + await store.commit(staged, OPEN, context); + const { context: cancellable, cancel } = withCancel(context); + const before = await settledHeap(); + const iterator = store.export(cancellable); + // The manifest, the table write, and the first put of a page of 100 rows of 8 KB each. + for (let i = 0; i < 3; i++) await iterator.next(); + cancel(new Error("stop")); + const after = await settledHeap(); + expect(after - before).toBeLessThan(200 * 1024); + expect(await rejection(iterator.next())).toMatchObject({ code: "aborted" }); + }); + + it("code that commits while consuming its own export waits until ctx aborts", async () => { + store = await archiveStore(); + const { context: cancellable, cancel } = withCancel(context); + const iterator = store.export(cancellable); + await iterator.next(); + const commit = store.commit([], OPEN, context); + let committed = false; + void commit.then(() => { + committed = true; + }); + await flushed(); + expect(committed).toBe(false); + cancel(new Error("stop")); + await commit; + expect(committed).toBe(true); + }); + + it("closing the store ends a held export: its next() rejects released, also for a part that needs no read", async () => { + store = await archiveStore(); + const iterator = store.export(context); + await iterator.next(); + await store.close(context); + // The table write would need no read. + expect(await rejection(iterator.next())).toMatchObject({ code: "released" }); + expect(await rejection(iterator.next())).toMatchObject({ code: "released" }); + }); + + it("an abort while next() waits for the queue rejects at once; no abort listener is left", async () => { + store = await archiveStore(); + const first = store.export(context); + await first.next(); + const { context: cancellable, cancel } = withCancel(context); + const second = store.export(cancellable); + const waiting = rejection(second.next()); + await flushed(); + cancel(new Error("stop")); + expect(await waiting).toMatchObject({ code: "aborted" }); + expect(getEventListeners(cancellable.abortSignal!, "abort")).toHaveLength(0); + // The first export still holds the queue; once it ends, the aborted one gives the queue back at once. + await first.return!(); + await store.commit([], OPEN, context); + expect(await rejection(second.next())).toMatchObject({ code: "aborted" }); + }); + + it("closing the store before an export's first next() rejects it released: not even a manifest", async () => { + store = await archiveStore(); + const iterator = store.export(context); + await store.close(context); + expect(await rejection(iterator.next())).toMatchObject({ code: "released" }); + expect(await rejection(iterator.next())).toMatchObject({ code: "released" }); + }); + + it("closing the store while an export waits for the queue rejects its next() released", async () => { + store = await archiveStore(); + const first = store.export(context); + await first.next(); + const second = store.export(context); + const waiting = rejection(second.next()); + await flushed(); + await store.close(context); + expect(await waiting).toMatchObject({ code: "released" }); + }); +}); diff --git a/packages/chord/test/next/store/expression.test.ts b/packages/chord/test/next/store/expression.test.ts new file mode 100644 index 000000000..c39c844ec --- /dev/null +++ b/packages/chord/test/next/store/expression.test.ts @@ -0,0 +1,152 @@ +// Tests of the table expression language: the builders and their compilation into the data vocabulary. +import { describe, expect, it } from "vitest"; +import { bool, int, nul, num, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { + add, + and, + coalesce, + compileSet, + compileWhere, + concat, + eq, + gt, + gte, + inArray, + isNull, + lt, + lte, + max, + ne, + not, + or, + prefix, + replacePrefix, + sub, + when, +} from "../../../src/next/store/expression.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import type { ColumnRefs } from "../../../src/next/store/query.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; + +const T = defineTable("t", { + version: 1, + columns: { + id: str(), + path: str(), + n: int(), + x: num(), + b: bool(), + note: opt(str()), + count: opt(int()), + meta: opt(json(obj({ a: int() }), { version: 1 })), + }, + key: ["id"], +}); +type C = typeof T extends { "~columns"?: infer X } ? NonNullable : never; +const refs = T.refs as ColumnRefs; +const args = expect.objectContaining({ code: "invalid", reason: "args" }); +describe("compiling conditions", () => { + it("compiles every operator into data", () => { + const where = and( + eq(refs.path, "a"), + ne(refs.n, 1), + lt(refs.n, 2), + lte(refs.x, 2.5), + gt(refs.n, 0), + gte(refs.path, "a"), + inArray(refs.n, [1, 2]), + isNull(refs.note), + prefix(refs.path, "a/"), + or(not(eq(refs.b, true))), + ); + expect(compileWhere(T, where, "where")).toEqual({ + and: [ + { eq: ["path", "a"] }, + { ne: ["n", 1] }, + { lt: ["n", 2] }, + { lte: ["x", 2.5] }, + { gt: ["n", 0] }, + { gte: ["path", "a"] }, + { inArray: ["n", [1, 2]] }, + { isNull: "note" }, + { prefix: ["path", "a/"] }, + { or: [{ not: { eq: ["b", true] } }] }, + ], + }); + }); + + it("refuses undefined and null operands (rule 16), wrong types, json and unknown columns", () => { + expect(() => compileWhere(T, eq(refs.note, undefined as never), "where")).toThrow("ask with isNull"); + expect(() => compileWhere(T, eq(refs.note, null as never), "where")).toThrow(args); + expect(() => compileWhere(T, eq(refs.n, "1" as never), "where")).toThrow(args); + expect(() => compileWhere(T, eq(refs.n, 1.5), "where")).toThrow(args); + expect(() => compileWhere(T, eq(refs.meta as never, 1), "where")).toThrow("not queryable"); + expect(() => compileWhere(T, eq({ column: "nope" } as never, 1), "where")).toThrow("expected a column reference"); + expect(() => compileWhere(T, "x", "where")).toThrow("expected a condition"); + expect(() => compileWhere(T, prefix(refs.n as never, "a"), "where")).toThrow("str() column"); + }); + + it("makes text operands storable", () => { + expect(compileWhere(T, eq(refs.path, "a\uD800"), "where")).toEqual({ eq: ["path", "a\uFFFD"] }); + expect(() => compileWhere(T, prefix(refs.path, "a\0"), "where")).toThrow("NUL"); + }); +}); + +describe("compiling sets", () => { + it("compiles plain values and expressions into one vocabulary, a clear as null (rule 14)", () => { + expect(compileSet(T, { path: "p", note: undefined, b: true, meta: { a: 1 } }, "update")).toEqual({ + path: "p", + note: null, + b: true, + meta: { json: { a: 1 } }, + }); + const set = compileSet( + T, + { + n: add(refs.n, 1), + count: sub(coalesce(refs.count, 0), refs.n), + x: max(refs.x, 1.5), + path: replacePrefix(concat(refs.path, "/"), "a/", "b/"), + note: when(gt(refs.n, 1), refs.path, refs.note), + }, + "update", + ); + expect(JSON.parse(JSON.stringify(set))).toEqual({ + n: { add: [{ col: "n" }, 1] }, + count: { sub: [{ coalesce: [{ col: "count" }, 0] }, { col: "n" }] }, + x: { max: [{ col: "x" }, 1.5] }, + path: { replacePrefix: [{ concat: [{ col: "path" }, "/"] }, "a/", "b/"] }, + note: { when: [{ gt: ["n", 1] }, { col: "path" }, { col: "note" }] }, + }); + }); + + it("refuses an empty set, key columns (rule 13), unknown columns and clears of non-opt columns", () => { + expect(() => compileSet(T, {}, "update")).toThrow("empty"); + expect(() => compileSet(T, { id: "x" }, "update")).toThrow("cannot change"); + expect(() => compileSet(T, { nope: 1 }, "update")).toThrow('no column "nope"'); + expect(() => compileSet(T, { path: undefined }, "update")).toThrow("cannot be cleared"); + }); + + it("undefined clears an opt column; null is a json(...) cell's JSON null and refused for scalar columns", () => { + expect(compileSet(T, { note: undefined, meta: undefined }, "update")).toEqual({ note: null, meta: null }); + const Nullable = defineTable("n", { + version: 1, + columns: { id: str(), data: json(nul(), { version: 1 }) }, + key: ["id"], + }); + expect(compileSet(Nullable, { data: null }, "update")).toEqual({ data: { json: null } }); + expect(() => compileSet(T, { note: null as never }, "update")).toThrow("null is no str value"); + }); + + it("refuses integer arithmetic outside int columns (rule 10), type mismatches and nullable results into non-opt columns", () => { + expect(() => compileSet(T, { x: add(refs.n, 1) }, "update")).toThrow("add gives int"); + expect(() => compileSet(T, { path: max(refs.n, 1) as never }, "update")).toThrow("gives int or num"); + expect(() => compileSet(T, { path: refs.n as never }, "update")).toThrow('column "n" is int, not str'); + expect(() => compileSet(T, { n: add(refs.count, 1) }, "update")).toThrow("may be NULL"); + expect(() => compileSet(T, { path: refs.note }, "update")).toThrow("may be NULL"); + expect(compileSet(T, { path: coalesce(refs.note, "d") }, "update")).toEqual({ + path: { coalesce: [{ col: "note" }, "d"] }, + }); + expect(() => compileSet(T, { meta: refs.meta as never }, "update")).toThrow("plain values"); + }); +}); diff --git a/packages/chord/test/next/store/import.test.ts b/packages/chord/test/next/store/import.test.ts new file mode 100644 index 000000000..d440a901e --- /dev/null +++ b/packages/chord/test/next/store/import.test.ts @@ -0,0 +1,436 @@ +// Tests of `importStore` (slice 6a §13, rules 12, 24, 25): round trips (old row versions, a migrating table, +// uncheckpointed deltas), the store it returns, every refusal of the archive's order and content, the rollback, and an +// input that stalls cancelled by `ctx`. + +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import type { JsonValue } from "../../../src/json.ts"; +import { nul, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import type { ArchivePart } from "../../../src/next/store/archive.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import { findDocument, loadDocument } from "../../../src/next/store/engine.ts"; +import { importStore } from "../../../src/next/store/import.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import { evolveTable } from "../../../src/next/store/table-evolve.ts"; +import { TableHandle } from "../../../src/next/store/table-handle.ts"; +import { prepareTables } from "../../../src/next/store/table-migration.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { ArchiveTables, archiveStore, exported, memoryStore, rejection, tableIn, tableOwner } from "../support.ts"; + +const stores: ChordStore[] = []; +afterEach(async () => { + for (const store of stores.splice(0)) await store.close(context); +}); + +/** `parts` imported into a fresh `:memory:` backend. */ +async function imported(parts: Iterable | AsyncIterable, outbox?: boolean): Promise { + const backend = sqliteBackend(await openNodeSqliteDatabase(":memory:")); + const store = (await importStore(backend, parts, context, { outbox })) as ChordStore; + stores.push(store); + return store; +} + +/** The archive of `archiveStore`. */ +async function archive(): Promise { + const source = await archiveStore(); + stores.push(source); + return exported(source); +} + +describe("round trips", () => { + it("an imported archive exports the same parts: lossless, at the manifest's lastSeq and next document id", async () => { + const parts = await archive(); + const store = await imported(parts); + expect(await exported(store)).toEqual(parts); + const manifest = parts[0] as Extract; + expect([store.lastSeq, store.nextDocumentId]).toEqual([manifest.lastSeq, manifest.nextDocumentId]); + }); + + it("the store keeps its ids, seqs and folded documents; the outbox, when asked, starts after lastSeq", async () => { + const parts = await archive(); + const store = await imported(parts, true); + expect(await store.outbox!.read(store.lastSeq, 10, context)).toEqual([]); + expect(await rejection(store.outbox!.read(0, 10, context))).toMatchObject({ code: "invalid" }); + const record = await store.read( + (executor) => + findDocument(executor, { namespace: "chord", owner: "client:app.store", kind: "memo", key: "a" }), + context, + ); + const original = parts.find((part) => part.kind === "document.create" && part.record.key === "a"); + expect(record).toEqual((original as Extract).record); + expect((await store.read((executor) => loadDocument(executor, record!), context)).value).toEqual({ title: "a3" }); + }); + + it("a migrating table resumes on the imported store; rows written at version 1 read as the table's", async () => { + const store = await imported(await archive()); + const { EditsV2, NotesV2 } = ArchiveTables; + const notes = "client:app.store:notes"; + const fence = { admit: () => true, adopt: () => {} }; + await expect( + prepareTables([{ definition: NotesV2, table: notes }], store, fence, "app", context), + ).rejects.toThrow("crash"); + const edits = new TableHandle(EditsV2, "client:app.store:edits", store, tableOwner()); + expect(await edits.get({ id: "e003" }, context)).toEqual({ + id: "e003", + path: "p/3", + at: 3, + meta: { a: 3 }, + done: false, + }); + }); +}); + +describe("json(...) cells holding null", () => { + it("a JSON null round-trips, required and optional, put and updated; an absent optional cell stays absent", async () => { + const source = await memoryStore(); + stores.push(source); + const Cells = defineTable("cells", { + version: 1, + columns: { id: str(), data: json(nul(), { version: 1 }), more: opt(json(nul(), { version: 1 })) }, + key: ["id"], + }); + const handle = await tableIn(source, Cells); + const staged: StagedWrite[] = []; + const changes = new TableChanges(handle, staged); + changes.put({ id: "a", data: null, more: null }); + changes.put({ id: "b", data: null }); + changes.put({ id: "c", data: null, more: null }); + changes.update({ id: "c" }, { more: undefined }); + await source.commit(staged, { admit: () => true, adopt: () => {} }, context); + const rows = [ + { id: "a", data: null, more: null }, + { id: "b", data: null }, + { id: "c", data: null }, + ]; + for (const row of rows) expect(await handle.get({ id: row.id }, context)).toEqual(row); + const parts = await exported(source); + const store = await imported(parts); + expect(await exported(store)).toEqual(parts); + const copy = new TableHandle(Cells, handle.sql.table, store, tableOwner()); + for (const row of rows) expect(await copy.get({ id: row.id }, context)).toEqual(row); + }); + + it("a JSON null at an older cell version round-trips with its version and migrates on access in the copy", async () => { + const source = await memoryStore(); + stores.push(source); + const V1 = defineTable("cells", { + version: 1, + columns: { id: str(), data: json(nul(), { version: 1 }) }, + key: ["id"], + }); + const old = await tableIn(source, V1); + const staged: StagedWrite[] = []; + new TableChanges(old, staged).put({ id: "a", data: null }); + await source.commit(staged, { admit: () => true, adopt: () => {} }, context); + const V2 = evolveTable(V1, (t) => + t.json("data", json(obj({ was: nul() }), { version: 2, migrate: (value) => ({ was: value as null }) })), + ); + const handle = await tableIn(source, V2); + expect(await handle.get({ id: "a" }, context)).toEqual({ id: "a", data: { was: null } }); + const parts = await exported(source); + expect(parts.find((part) => (part as ArchivePart).kind === "put")).toMatchObject({ + row: { id: "a", data: null }, + columns: { data: 1 }, + }); + const store = await imported(parts); + expect(await exported(store)).toEqual(parts); + const copy = new TableHandle(V2, handle.sql.table, store, tableOwner()); + expect(await copy.get({ id: "a" }, context)).toEqual({ id: "a", data: { was: null } }); + }); +}); + +/** The archive with `edit` applied to a copy of its parts. */ +async function edited(edit: (parts: ArchivePart[]) => unknown[]): Promise { + return edit(structuredClone(await archive())); +} + +/** The index of the first part of `kind` (and table suffix). */ +function first(parts: readonly ArchivePart[], kind: ArchivePart["kind"], table?: string): number { + return parts.findIndex( + (part) => part.kind === kind && (table === undefined || ("table" in part && part.table.endsWith(table))), + ); +} + +interface Mutable { + [field: string]: JsonValue | Mutable; +} + +/** Part `index` as a writable object. */ +function at(parts: unknown[], index: number): Mutable { + return parts[index] as Mutable; +} + +describe("refusals", () => { + it.each<[string, (parts: ArchivePart[]) => unknown[], string]>([ + [ + "a part outside the schema", + (parts) => [...parts.slice(0, 1), { kind: "put" }, ...parts.slice(1)], + "not an archive part", + ], + ["no manifest first", (parts) => parts.slice(1), "the manifest comes first"], + ["a second manifest", (parts) => [parts[0], ...parts], "the manifest comes first, once"], + [ + "another format", + (parts) => { + at(parts, 0).format = 2; + return parts; + }, + "format 2 is not 1", + ], + [ + "a newer store schema", + (parts) => { + at(parts, 0).schema = 9; + return parts; + }, + "schema 9 is newer", + ], + [ + "a lastSeq of 2^53 (a later seq would repeat)", + (parts) => { + at(parts, 0).lastSeq = 2 ** 53; + return parts; + }, + "part 0: not an archive part", + ], + [ + "a document revision beyond 2^53 - 1", + (parts) => { + const record = at(parts, first(parts, "document.create")).record as Mutable; + record.revision = 2 ** 53; + return parts; + }, + "not an archive part", + ], + ["a part after the end", (parts) => [...parts, { kind: "end", writes: 0 }], "after the end"], + ["no end", (parts) => parts.slice(0, -1), "has no end part"], + [ + "a wrong end count", + (parts) => { + at(parts, parts.length - 1).writes = 1; + return parts; + }, + "counts 1 writes", + ], + [ + "a put before its table", + (parts) => { + const put = first(parts, "put", ":notes"); + return [...parts.slice(0, first(parts, "table")), parts[put], ...parts.slice(first(parts, "table"))]; + }, + "a put follows its table's table write", + ], + [ + "a put after its table's indexes", + (parts) => { + const index = first(parts, "index"); + return [...parts.slice(0, index + 1), parts[index - 1], ...parts.slice(index + 1)]; + }, + "a put follows its table's table write, before its indexes", + ], + [ + "a table twice", + (parts) => { + const table = first(parts, "table", ":notes"); + return [...parts.slice(0, table), parts[first(parts, "table")], ...parts.slice(table)]; + }, + "occurs twice", + ], + [ + "a table after the documents", + (parts) => { + const table = first(parts, "table", ":notes"); + const documents = first(parts, "document.create"); + return [ + ...parts.slice(0, table), + ...parts.slice(documents, -1), + ...parts.slice(table, documents), + parts.at(-1), + ]; + }, + "tables come before the documents", + ], + [ + "a table name that is not kind:type:table", + (parts) => { + at(parts, first(parts, "table")).table = "edits"; + return parts; + }, + "not kind:type:table", + ], + [ + "a column name breaking the rules", + (parts) => { + const table = at(parts, first(parts, "table")); + table.columns = { ...(table.columns as Mutable), "bad name": { type: "str", optional: true } }; + return parts; + }, + "not a column name", + ], + [ + "a row with an unknown column", + (parts) => { + (at(parts, first(parts, "put")).row as Mutable).extra = 1; + return parts; + }, + 'unknown column "extra"', + ], + [ + "a row without a non-opt column", + (parts) => { + delete (at(parts, first(parts, "put")).row as Mutable).path; + return parts; + }, + 'column "path" is missing', + ], + [ + "a value of another type", + (parts) => { + (at(parts, first(parts, "put")).row as Mutable).at = "x"; + return parts; + }, + "is not a int value", + ], + [ + "a json(...) cell newer than its column", + (parts) => { + (at(parts, first(parts, "put")).columns as Mutable).meta = 5; + return parts; + }, + "no readable version", + ], + [ + "a row newer than its table", + (parts) => { + at(parts, first(parts, "put")).version = 7; + return parts; + }, + "version 7 is newer", + ], + [ + "an id not below the manifest's next id", + (parts) => { + (at(parts, first(parts, "document.create")).record as Mutable).id = 10_000; + return parts; + }, + "is not below", + ], + [ + "an id twice", + (parts) => { + const document = first(parts, "document.create"); + (at(parts, document + 1).record as Mutable).id = (at(parts, document).record as Mutable).id!; + return parts; + }, + "occurs twice", + ], + [ + "a record created after lastSeq", + (parts) => { + (at(parts, first(parts, "document.create")).record as Mutable).createdAt = 10_000; + return parts; + }, + "created after the manifest's lastSeq", + ], + [ + "a base at another version", + (parts) => { + (at(parts, first(parts, "document.create")).base as Mutable).version = 2; + return parts; + }, + "the base is not at the record's version", + ], + [ + "two live records at one address", + (parts) => { + const document = first(parts, "document.create"); + (at(parts, document + 1).record as Mutable).key = (at(parts, document).record as Mutable).key!; + return parts; + }, + "a second live record at one address", + ], + [ + "a record's meta with a NUL", + (parts) => { + (at(parts, first(parts, "document.create")).record as Mutable).meta = "a\u0000b"; + return parts; + }, + "text may not contain NUL", + ], + [ + "a record's meta with a lone surrogate", + (parts) => { + (at(parts, first(parts, "document.create")).record as Mutable).meta = "a\uD800"; + return parts; + }, + "the meta is not well-formed text", + ], + ])("%s", async (_, edit, message) => { + const parts = await edited(edit); + expect(await rejection(imported(parts))).toMatchObject({ + code: "invalid", + reason: "args", + message: expect.stringContaining(message), + }); + }); + + it("a non-empty backend is refused; a refused import closes its backend and leaves it as it was", async () => { + const directory = await mkdtemp(join(tmpdir(), "chord-import-")); + try { + const path = join(directory, "store.db"); + const parts = await archive(); + const broken = [...parts.slice(0, -1), { kind: "end", writes: 0 }]; + expect( + await rejection(importStore(sqliteBackend(await openNodeSqliteDatabase(path)), broken, context)), + ).toMatchObject({ + code: "invalid", + }); + const db = await openNodeSqliteDatabase(path); + expect(await db.all("SELECT name FROM sqlite_master")).toEqual([]); + await db.close(); + const store = await importStore(sqliteBackend(await openNodeSqliteDatabase(path)), parts, context); + await store.close(context); + expect( + await rejection(importStore(sqliteBackend(await openNodeSqliteDatabase(path)), parts, context)), + ).toMatchObject({ + message: "import: the backend holds a store already", + }); + } finally { + await rm(directory, { recursive: true, force: true }); + } + }); + + it("an input that stalls is cancelled by ctx: the import rejects aborted and the input is closed", async () => { + const parts = await archive(); + let closed = false; + const stalled: AsyncIterable = { + [Symbol.asyncIterator]: () => { + let index = 0; + return { + next: () => + index < 3 + ? Promise.resolve({ done: false, value: parts[index++] }) + : new Promise>(() => {}), + return: async () => { + closed = true; + return { done: true, value: undefined }; + }, + }; + }, + }; + const { context: cancellable, cancel } = withCancel(context); + const backend = sqliteBackend(await openNodeSqliteDatabase(":memory:")); + const importing = importStore(backend, stalled, cancellable); + await new Promise((resolve) => setTimeout(resolve, 10)); + cancel(new Error("stop")); + expect(await rejection(importing)).toMatchObject({ code: "aborted" }); + expect(closed).toBe(true); + }); +}); diff --git a/packages/chord/test/next/store/json-column.test.ts b/packages/chord/test/next/store/json-column.test.ts new file mode 100644 index 000000000..339aaf1f3 --- /dev/null +++ b/packages/chord/test/next/store/json-column.test.ts @@ -0,0 +1,58 @@ +// `json(...)` columns (rule 8): their declaration, and a cell read and migrated on access. + +import { describe, expect, it } from "vitest"; +import { int, obj, str } from "../../../src/next/contracts/index.ts"; +import { JsonColumn, json } from "../../../src/next/store/json-column.ts"; +import { counted } from "../support.ts"; + +const MetaV1 = obj({ tag: str() }); +const MetaV2 = obj({ tags: str(), n: int() }); + +describe("json(...)", () => { + it("is a schema carrying its column: version and the fingerprint of the schema's normal form", () => { + const column = (json(MetaV1, { version: 1 }) as unknown as { json: JsonColumn }).json; + expect(column).toBeInstanceOf(JsonColumn); + expect(column.version).toBe(1); + const again = (json(obj({ tag: str({}) }), { version: 1 }) as unknown as { json: JsonColumn }).json; + const other = (json(MetaV2, { version: 1 }) as unknown as { json: JsonColumn }).json; + expect(again.fingerprint).toBe(column.fingerprint); + expect(other.fingerprint).not.toBe(column.fingerprint); + }); + + it("refuses a version that is not a positive integer", () => { + expect(() => json(MetaV1, { version: 0 })).toThrow("version 0"); + }); +}); + +describe("reading a cell", () => { + const v2 = new JsonColumn(MetaV2, 2, (old, from) => ({ tags: (old as { tag: string }).tag, n: from })); + + it("parses a cell of the current version with no check", async () => { + const counts = await counted((context) => { + expect(v2.read('{"tags":"a","n":1}', 2, "t.meta", context)).toEqual({ tags: "a", n: 1 }); + }); + expect(counts.validations).toBe(0); + }); + + it("migrates an older cell and checks the result once (R8)", async () => { + const counts = await counted((context) => { + expect(v2.read('{"tag":"x"}', 1, "t.meta", context)).toEqual({ tags: "x", n: 1 }); + }); + expect(counts.validations).toBe(1); + }); + + it("refuses a newer cell, a missing or throwing migrate and an invalid result", async () => { + await counted((context) => { + expect(() => v2.read("{}", 3, "t.meta", context)).toThrow("newer than 2"); + expect(() => new JsonColumn(MetaV2, 2, undefined).read("{}", 1, "t.meta", context)).toThrow("no migrate"); + const throwing = new JsonColumn(MetaV2, 2, () => { + throw new Error("no"); + }); + expect(() => throwing.read("{}", 1, "t.meta", context)).toThrow("threw"); + const invalid = new JsonColumn(MetaV2, 2, () => ({ tags: 1, n: 1 }) as never); + expect(() => invalid.read("{}", 1, "t.meta", context)).toThrow( + expect.objectContaining({ code: "unavailable" }), + ); + }); + }); +}); diff --git a/packages/chord/test/next/store/jsonl.test.ts b/packages/chord/test/next/store/jsonl.test.ts new file mode 100644 index 000000000..3d36e2321 --- /dev/null +++ b/packages/chord/test/next/store/jsonl.test.ts @@ -0,0 +1,56 @@ +// Tests of the JSONL archive encoding (slice 6a §13): parts as lines, chunks split anywhere (inside a line, inside a +// UTF-8 sequence), blank lines, and the refusal of a line that is no JSON. + +import { describe, expect, it } from "vitest"; +import type { ArchivePart } from "../../../src/next/store/archive.ts"; +import { decodeJsonl, encodeJsonl } from "../../../src/next/store/jsonl.ts"; +import { rejection } from "../support.ts"; + +async function* from(values: readonly T[]): AsyncIterable { + yield* values; +} + +async function collect(values: AsyncIterable): Promise { + const all: T[] = []; + for await (const value of values) all.push(value); + return all; +} + +const PARTS: ArchivePart[] = [ + { kind: "manifest", format: 1, schema: 1, lastSeq: 3, nextDocumentId: 2 }, + { kind: "put", table: "client:app.store:edits", row: { id: "ü€😀", n: 1 }, version: 1, columns: {} }, + { kind: "end", writes: 1 }, +]; + +describe("jsonl", () => { + it("encodes one part per line and decodes it back, from chunks split anywhere", async () => { + const bytes = Buffer.concat(await collect(encodeJsonl(from(PARTS)))); + expect(bytes.toString("utf8").split("\n")).toHaveLength(PARTS.length + 1); + for (const size of [1, 2, 3, 7, bytes.length]) { + const chunks: Uint8Array[] = []; + for (let at = 0; at < bytes.length; at += size) chunks.push(bytes.subarray(at, at + size)); + expect(await collect(decodeJsonl(from(chunks)))).toEqual(PARTS); + } + }); + + it("skips blank lines and takes a last line without a newline", async () => { + const text = `\n${JSON.stringify(PARTS[0])}\n\n${JSON.stringify(PARTS[2])}`; + expect(await collect(decodeJsonl(from([new TextEncoder().encode(text)])))).toEqual([PARTS[0], PARTS[2]]); + }); + + it("refuses a line that is no JSON, naming it", async () => { + const text = `${JSON.stringify(PARTS[0])}\n{nope\n`; + expect(await rejection(collect(decodeJsonl(from([new TextEncoder().encode(text)]))))).toMatchObject({ + code: "invalid", + reason: "args", + message: "jsonl: line 2 is not JSON", + }); + }); + + it("refuses bytes that are no UTF-8", async () => { + expect(await rejection(collect(decodeJsonl(from([new Uint8Array([0x7b, 0xff, 0x0a])]))))).toMatchObject({ + code: "invalid", + message: "jsonl: line 1 is not UTF-8", + }); + }); +}); diff --git a/packages/chord/test/next/store/outbox.test.ts b/packages/chord/test/next/store/outbox.test.ts new file mode 100644 index 000000000..2a720c080 --- /dev/null +++ b/packages/chord/test/next/store/outbox.test.ts @@ -0,0 +1,165 @@ +// Tests of the outbox (slice 6a §2; decisions "The outbox is optional, internal and versioned", small items 3-4): on, +// off and kept across opens, its commits in their recorded form, `trim` and its floor, and the format check. + +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import { changeDocument, createDocument } from "../../../src/next/store/engine.ts"; +import { OUTBOX_FORMAT } from "../../../src/next/store/outbox.ts"; +import { TABLES } from "../../../src/next/store/schema.ts"; +import { type ChordStore, openStore } from "../../../src/next/store/store.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { memoryStore, rejection } from "../support.ts"; + +const ADDRESS = { namespace: "chord", owner: "session:app.store", kind: "notes", key: "" }; +const BASE = { version: 1, fingerprint: "0000000000000000", value: { n: 0 } }; +const OPEN = { admit: () => true, adopt: () => {} }; + +/** Creates the singleton at `ADDRESS` (with `key`) in one commit. */ +async function created(store: ChordStore, key = ""): Promise { + await store.commit([createDocument({ ...ADDRESS, key }, "latest", BASE, JSON.stringify(BASE.value))], OPEN, context); +} + +let directory: string | undefined; + +afterEach(async () => { + if (directory !== undefined) await rm(directory, { recursive: true, force: true }); + directory = undefined; +}); + +/** A store on a fresh file, reopened by `open`. */ +async function onFile() { + directory = await mkdtemp(join(tmpdir(), "chord-outbox-")); + const path = join(directory, "store.sqlite"); + return async (outbox?: boolean) => + (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { outbox })) as ChordStore; +} + +describe("the outbox", () => { + it("is off for a new store, on from the seq it is turned on at, kept when omitted, dropped when turned off", async () => { + const open = await onFile(); + let store = await open(); + await created(store); + await store.close(context); + store = await open(true); + expect(store.outbox?.floor).toBe(1); + const id = 1; + const ops = [["s", ["n"], 2]] as const; + await store.commit( + [changeDocument(id, 1, { kind: "delta", version: 1, ops }, JSON.stringify(ops))], + OPEN, + context, + ); + await store.close(context); + store = await open(); + expect(await store.outbox!.read(1, 10, context)).toEqual([ + { + seq: 2, + writes: [{ kind: "document.change", id, revision: 1, content: { kind: "delta", version: 1, ops } }], + }, + ]); + expect(await rejection(store.outbox!.read(0, 10, context))).toMatchObject({ code: "invalid", reason: "args" }); + expect(await rejection(store.outbox!.read(3, 10, context))).toMatchObject({ code: "invalid", reason: "args" }); + await store.close(context); + store = await open(false); + expect(store.outbox).toBeUndefined(); + await store.close(context); + }); + + it("records creations with their ids and seqs; trim is idempotent and refuses beyond lastSeq", async () => { + const store = await memoryStore({ outbox: true }); + await created(store); + const [commit] = await store.outbox!.read(0, 10, context); + expect(commit!.writes[0]).toMatchObject({ + kind: "document.create", + record: { id: 1, createdAt: 1, ...ADDRESS }, + base: BASE, + }); + await store.outbox!.trim(1, context); + await store.outbox!.trim(1, context); + expect(store.outbox!.floor).toBe(1); + expect(await store.outbox!.read(1, 10, context)).toEqual([]); + expect(await rejection(store.outbox!.trim(2, context))).toMatchObject({ code: "invalid", reason: "args" }); + }); + + it("overlapping trims run one at a time: the stored floor never moves back and equals the one in memory", async () => { + const open = await onFile(); + let store = await open(true); + for (let i = 0; i < 11; i++) await created(store, `k${i}`); + await Promise.all([store.outbox!.trim(9, context), store.outbox!.trim(3, context)]); + expect(store.outbox!.floor).toBe(9); + await store.close(context); + store = await open(); + expect(store.outbox!.floor).toBe(9); + expect(await rejection(store.outbox!.read(3, 10, context))).toMatchObject({ code: "invalid", reason: "args" }); + expect((await store.outbox!.read(9, 10, context)).map(({ seq }) => seq)).toEqual([10, 11]); + await store.close(context); + }); + + it("a read waits for a trim whose commit has not returned: the trimmed floor refuses it, never an empty page", async () => { + const db = await openNodeSqliteDatabase(":memory:"); + const transaction = db.transaction.bind(db); + const entered = Promise.withResolvers(); + const gate = Promise.withResolvers(); + const store = (await openStore(sqliteBackend(db), context, { outbox: true })) as ChordStore; + await created(store); + // The trim's DELETE committed; its transaction has not returned (the floor in memory is still 0). + db.transaction = async (fn) => { + const result = await transaction(fn); + entered.resolve(); + await gate.promise; + return result; + }; + const trim = store.outbox!.trim(1, context); + await entered.promise; + db.transaction = transaction; + const read = rejection(store.outbox!.read(0, 100, context)); + gate.resolve(); + await trim; + expect(await read).toMatchObject({ code: "invalid", reason: "args" }); + await store.close(context); + }); + + it("a trim waits for a read in flight: the read gets the commits it was admitted for", async () => { + const db = await openNodeSqliteDatabase(":memory:"); + const store = (await openStore(sqliteBackend(db), context, { outbox: true })) as ChordStore; + await created(store); + const all = db.all.bind(db); + const entered = Promise.withResolvers(); + const gate = Promise.withResolvers(); + db.all = (async (sql: string, ...params: never[]) => { + entered.resolve(); + await gate.promise; + return all(sql, ...params); + }) as typeof db.all; + const read = store.outbox!.read(0, 100, context); + await entered.promise; + db.all = all; + const trim = store.outbox!.trim(1, context); + gate.resolve(); + expect((await read).map(({ seq }) => seq)).toEqual([1]); + await trim; + expect(store.outbox!.floor).toBe(1); + await store.close(context); + }); + + it("a commit stored in a newer format is refused unavailable; the data stays", async () => { + const open = await onFile(); + let store = await open(true); + await created(store); + await store.close(context); + const db = await openNodeSqliteDatabase(join(directory!, "store.sqlite")); + await db.run(`UPDATE ${TABLES.outbox} SET format = ? WHERE seq = 1`, OUTBOX_FORMAT + 1); + await db.close(); + store = await open(); + expect(await rejection(store.outbox!.read(0, 10, context))).toMatchObject({ + code: "unavailable", + message: expect.stringContaining(`format ${OUTBOX_FORMAT + 1}`), + }); + expect(await rejection(store.outbox!.read(0, 10, context))).toMatchObject({ code: "unavailable" }); + await store.close(context); + }); +}); diff --git a/packages/chord/test/next/store/query.test.ts b/packages/chord/test/next/store/query.test.ts new file mode 100644 index 000000000..9dfb93473 --- /dev/null +++ b/packages/chord/test/next/store/query.test.ts @@ -0,0 +1,86 @@ +// Queries (slice 6a §7): `where`, `orderBy` and limits over a table, and cursors paging through them. + +import { describe, expect, it } from "vitest"; +import { int, opt, str } from "../../../src/next/contracts/index.ts"; +import { eq, gt } from "../../../src/next/store/expression.ts"; +import { decodeCursor, encodeCursor, query } from "../../../src/next/store/query.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; + +const T = defineTable("edits", { + version: 1, + columns: { id: str(), entry: int(), at: int(), note: opt(str()) }, + key: ["id", "entry"], +}); +const args = expect.objectContaining({ code: "invalid", reason: "args" }); + +describe("queries", () => { + it("are immutable; several wheres are and-ed", () => { + const base = query(T); + const one = base.where((e) => eq(e.id, "a")); + const two = one.where((e) => gt(e.at, 1)); + expect(base.condition).toBeUndefined(); + expect(one.condition).toEqual({ eq: ["id", "a"] }); + expect(two.condition).toEqual({ and: [{ eq: ["id", "a"] }, { gt: ["at", 1] }] }); + }); + + it("order by the key ascending by default and append the key as the tie-break, not repeated", () => { + expect(query(T).order).toEqual({ columns: ["id", "entry"], direction: "asc" }); + expect(query(T).orderBy((e) => e.at, "desc").order).toEqual({ + columns: ["at", "id", "entry"], + direction: "desc", + }); + expect(query(T).orderBy((e) => e.entry).order).toEqual({ columns: ["entry", "id"], direction: "asc" }); + }); + + it("refuse duplicate orderBy columns, two directions and opt columns (rule 17)", () => { + expect(() => + query(T) + .orderBy((e) => e.at) + .orderBy((e) => e.at), + ).toThrow("twice"); + expect(() => + query(T) + .orderBy((e) => e.at) + .orderBy((e) => e.id, "desc"), + ).toThrow("one direction"); + expect(() => query(T).orderBy((e) => e.note as never)).toThrow(args); + }); +}); + +describe("cursors", () => { + const order = query(T).orderBy((e) => e.at, "desc").order; + + it("round-trip the order values, also across conditions with the same order", () => { + const cursor = encodeCursor("client:app:edits", order, [5, "a\u{1F600}", 2]); + expect(cursor).toMatch(/^[A-Za-z0-9_-]+$/); + expect(decodeCursor(T, "client:app:edits", order, cursor)).toEqual([5, "a\u{1F600}", 2]); + const other = query(T) + .where((e) => eq(e.id, "x")) + .orderBy((e) => e.at, "desc").order; + expect(decodeCursor(T, "client:app:edits", other, cursor)).toEqual([5, "a\u{1F600}", 2]); + }); + + it("are a trust boundary: another table, order or direction, garbage and bad values are invalid/args", () => { + const cursor = encodeCursor("client:app:edits", order, [5, "a", 2]); + expect(() => decodeCursor(T, "client:app:edits", query(T).order, cursor)).toThrow(args); + expect(() => decodeCursor(T, "client:app:edits", query(T).orderBy((e) => e.at).order, cursor)).toThrow(args); + expect(() => + decodeCursor( + defineTable("other", { version: 1, columns: { id: str(), entry: int(), at: int() }, key: ["id", "entry"] }), + "client:app:other", + order, + cursor, + ), + ).toThrow(args); + // The same definition in another object's table. + expect(() => decodeCursor(T, "client:other:edits", order, cursor)).toThrow(args); + expect(() => decodeCursor(T, "client:app:edits", order, "%%%")).toThrow(args); + expect(() => decodeCursor(T, "client:app:edits", order, 42)).toThrow(args); + expect(() => + decodeCursor(T, "client:app:edits", order, encodeCursor("client:app:edits", order, ["x", "a", 2])), + ).toThrow(args); + expect(() => + decodeCursor(T, "client:app:edits", order, encodeCursor("client:app:edits", order, [5, "a"])), + ).toThrow(args); + }); +}); diff --git a/packages/chord/test/next/store/rows.test.ts b/packages/chord/test/next/store/rows.test.ts new file mode 100644 index 000000000..9f784ad90 --- /dev/null +++ b/packages/chord/test/next/store/rows.test.ts @@ -0,0 +1,77 @@ +// Rows (slice 6a §5, §9): `put` rows and keys checked and encoded, stored rows decoded. + +import { describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { bool, int, num, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; + +const T = defineTable("t", { + version: 3, + columns: { + id: str(), + n: int({ default: 7 }), + x: num(), + b: bool(), + note: opt(str()), + meta: json(obj({ a: int() }), { version: 2 }), + }, + key: ["id"], +}); +const args = expect.objectContaining({ code: "invalid", reason: "args" }); + +describe("put rows", () => { + it("encodes a row in physical order, defaults filled, booleans as 0 and 1, json serialized with its version", () => { + const encoded = T.rows.put({ id: "a", x: 1.5, b: true, meta: { a: 1 } }, 3); + expect(encoded.params).toEqual(["a", 7, 1.5, 1, null, '{"a":1}', 2, 3]); + expect(encoded.row).toEqual({ id: "a", n: 7, x: 1.5, b: true, meta: { a: 1 } }); + expect(encoded.columns).toEqual({ meta: 2 }); + expect(encoded.id).toBe(T.rows.key({ id: "a" }, "get").id); + }); + + it("is a closed check: missing and unknown columns, wrong types, unsafe integers, non-finite numbers, NUL", () => { + const row = { id: "a", x: 1, b: false, meta: {} }; + expect(() => T.rows.put({ ...row, id: undefined }, 3)).toThrow(args); + expect(() => T.rows.put({ ...row, extra: 1 }, 3)).toThrow('unknown column "extra"'); + expect(() => T.rows.put({ ...row, n: 2 ** 53 }, 3)).toThrow(args); + expect(() => T.rows.put({ ...row, n: 1.5 }, 3)).toThrow(args); + expect(() => T.rows.put({ ...row, x: Number.POSITIVE_INFINITY }, 3)).toThrow(args); + expect(() => T.rows.put({ ...row, b: 1 }, 3)).toThrow(args); + expect(() => T.rows.put({ ...row, id: "a\0" }, 3)).toThrow("NUL"); + expect(() => T.rows.put({ ...row, note: null }, 3)).toThrow(args); + expect(() => T.rows.put([], 3)).toThrow("expected an object"); + }); + + it("serializes json cells at staging: a cycle or a BigInt fails there (rule 8)", () => { + const cycle: { self?: unknown } = {}; + cycle.self = cycle; + expect(() => T.rows.put({ id: "a", x: 1, b: false, meta: cycle }, 3)).toThrow(args); + expect(() => T.rows.put({ id: "a", x: 1, b: false, meta: { a: 1n } }, 3)).toThrow(args); + }); + + it("replaces lone surrogates in text", () => { + expect(T.rows.put({ id: "a\uD800", x: 1, b: false, meta: {} }, 3).row.id).toBe("a\uFFFD"); + }); +}); + +describe("keys", () => { + it("takes exactly the key columns", () => { + expect(T.rows.key({ id: "a" }, "get").params).toEqual(["a"]); + expect(() => T.rows.key({}, "get")).toThrow("is missing"); + expect(() => T.rows.key({ id: "a", n: 1 }, "get")).toThrow('unknown column "n"'); + expect(() => T.rows.key({ id: 1 }, "get")).toThrow(args); + }); +}); + +describe("decoding", () => { + it("reads booleans, leaves NULL opt columns absent and parses json cells", () => { + const row = T.rows.decode({ id: "a", n: 1, x: 2, b: 0, note: null, meta: '{"a":1}', meta$v: 2 }, context); + expect(row).toEqual({ id: "a", n: 1, x: 2, b: false, meta: { a: 1 } }); + }); + + it("refuses a json cell of a newer version", () => { + expect(() => T.rows.decode({ id: "a", n: 1, x: 2, b: 0, note: null, meta: "{}", meta$v: 3 }, context)).toThrow( + "newer than 2", + ); + }); +}); diff --git a/packages/chord/test/next/store/sql.test.ts b/packages/chord/test/next/store/sql.test.ts new file mode 100644 index 000000000..e8adaa0eb --- /dev/null +++ b/packages/chord/test/next/store/sql.test.ts @@ -0,0 +1,106 @@ +// SQL text (slice 6a §8): quoted names, the DDL of tables and indexes, and the statements of a table. + +import { describe, expect, it } from "vitest"; +import { bool, int, num, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { + createIndexSql, + createTableSql, + dropIndexSql, + quote, + TableSql, + tableSqlName, +} from "../../../src/next/store/sql.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; + +const T = defineTable("edits", { + version: 1, + columns: { + id: str(), + at: int({ default: 0 }), + x: num(), + b: bool(), + note: opt(str()), + meta: json(obj({}), { version: 2 }), + }, + key: ["id"], +}); + +describe("names", () => { + it("renders ::

, quoted", () => { + expect(tableSqlName("session", "@pi/file-history.store", "edits")).toBe("session:@pi/file-history.store:edits"); + expect(quote("session:a:b")).toBe('"session:a:b"'); + }); +}); + +describe("DDL", () => { + it("renders a STRICT WITHOUT ROWID table with checks, defaults, hidden version columns and the key", () => { + expect(createTableSql("s:o:edits", T.record, T.key)).toBe( + 'CREATE TABLE "s:o:edits" ("id" TEXT NOT NULL, "at" INTEGER NOT NULL DEFAULT 0 CHECK ("at" BETWEEN -9007199254740991 AND 9007199254740991), ' + + '"x" REAL NOT NULL, "b" INTEGER NOT NULL CHECK ("b" IN (0, 1)), "note" TEXT, "meta" TEXT NOT NULL, "meta$v" INTEGER NOT NULL, ' + + '"$v" INTEGER NOT NULL, PRIMARY KEY ("id")) STRICT, WITHOUT ROWID', + ); + expect(createIndexSql("s:o:edits", "by_at", ["at"], ["id"])).toBe( + 'CREATE INDEX "s:o:edits:by_at" ON "s:o:edits" ("at", "id")', + ); + expect(createIndexSql("s:o:edits", "by_id", ["id", "at"], ["id"])).toBe( + 'CREATE INDEX "s:o:edits:by_id" ON "s:o:edits" ("id", "at")', + ); + expect(dropIndexSql("s:o:edits", "by_at")).toBe('DROP INDEX "s:o:edits:by_at"'); + }); + + it("keeps int columns within ±(2^53 - 1) and bool columns within 0 and 1 (rule 10)", async () => { + const db = await openNodeSqliteDatabase(":memory:"); + await db.run(createTableSql("t", T.record, T.key)); + const sql = new TableSql("t", T.columns, T.rows.physical, T.key); + const params = T.rows.put({ id: "a", x: 1, b: true, meta: {} }, 1).params; + await db.run(sql.put, ...params); + await expect(db.run(`UPDATE t SET at = ?`, 2 ** 53)).rejects.toThrow("CHECK"); + await expect(db.run(`UPDATE t SET b = 2`)).rejects.toThrow("CHECK"); + await expect(db.run(`UPDATE t SET at = 'x'`)).rejects.toThrow(); + await db.run(`INSERT INTO t (id, x, b, meta, meta$v, "$v") VALUES ('d', 1, 0, '{}', 2, 1)`); + expect(await db.get(`SELECT at FROM t WHERE id = 'd'`)).toEqual({ at: 0 }); + await db.close(); + }); +}); + +describe("statements", () => { + const sql = new TableSql("t", T.columns, T.rows.physical, T.key); + + it("renders the fixed statements once", () => { + expect(sql.put).toBe( + 'INSERT OR REPLACE INTO "t" ("id", "at", "x", "b", "note", "meta", "meta$v", "$v") VALUES (?, ?, ?, ?, ?, ?, ?, ?)', + ); + expect(sql.get).toBe('SELECT "id", "at", "x", "b", "note", "meta", "meta$v" FROM "t" WHERE "id" = ?'); + expect(sql.delete).toBe('DELETE FROM "t" WHERE "id" = ?'); + }); + + it("stamps a json column's version with its value, NULL with a clear", () => { + const versions = new Map([["meta", 2]]); + expect(sql.update({ meta: { json: { a: 1 } } }, ["k"], versions)).toEqual({ + sql: 'UPDATE "t" SET "meta" = ?, "meta$v" = ? WHERE "id" = ?', + params: ['{"a":1}', 2, "k"], + }); + expect(sql.update({ meta: null }, ["k"], versions).params).toEqual([null, "k"]); + }); + + it("binds every value: one text per query shape, whatever the values (rule 18)", () => { + const order = { columns: ["at", "id"], direction: "desc" as const }; + const a = sql.select({ inArray: ["id", ["a", "b"]] }, order, [1, "x"], 10); + const b = sql.select({ inArray: ["id", Array.from({ length: 40_000 }, (_, i) => `k${i}`)] }, order, [2, "y"], 99); + expect(a.sql).toBe(b.sql); + expect(a.sql).toBe( + 'SELECT "id", "at", "x", "b", "note", "meta", "meta$v" FROM "t" WHERE "id" IN (SELECT value FROM json_each(?)) AND ("at", "id") < (?, ?) ORDER BY "at" DESC, "id" DESC LIMIT ?', + ); + expect(a.params).toEqual(['["a","b"]', 1, "x", 10]); + expect(sql.count({ prefix: ["id", "\u{10FFFF}"] })).toEqual({ + sql: 'SELECT count(*) AS count FROM "t" WHERE "id" >= ?', + params: ["\u{10FFFF}"], + }); + expect(sql.deleteWhere({ prefix: ["id", "a"] })).toEqual({ + sql: 'DELETE FROM "t" WHERE ("id" >= ? AND "id" < ?)', + params: ["a", "b"], + }); + }); +}); diff --git a/packages/chord/test/next/store/store.test.ts b/packages/chord/test/next/store/store.test.ts new file mode 100644 index 000000000..03f1c0087 --- /dev/null +++ b/packages/chord/test/next/store/store.test.ts @@ -0,0 +1,250 @@ +// The store (slice 6a §2, §3): opening, the commit step, storage failures, one host per store (rule 7), close. + +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context, withCancel } from "../../../src/context/index.ts"; +import { sqliteBackend } from "../../../src/next/store/backend.ts"; +import { changeDocument, createDocument, retireDocument } from "../../../src/next/store/engine.ts"; +import { importStore } from "../../../src/next/store/import.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import { type ChordStore, openStore, type Store } from "../../../src/next/store/store.ts"; +import { openNodeSqliteDatabase } from "../../../src/sqlite/node.ts"; +import { exported, metered, rejection } from "../support.ts"; + +const ADDRESS = { namespace: "chord", owner: "session:app.store", kind: "notes", key: "" }; +const BASE = { version: 1, fingerprint: "0000000000000000", value: { n: 0 } }; + +/** A store on a fresh `:memory:` database, as Chord's layers see it. */ +async function memoryStore(options?: { readonly outbox?: boolean }): Promise { + return (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context, options)) as ChordStore; +} + +/** A fence that admits and records what it adopted. */ +function recording(adopted: [number, readonly number[]][] = []) { + return { + adopted, + fence: { admit: () => true, adopt: (seq: number, ids: readonly number[]) => void adopted.push([seq, ids]) }, + }; +} + +/** Creates the singleton at `ADDRESS` and resolves with its id. */ +async function created(store: ChordStore): Promise { + const { adopted, fence } = recording(); + await store.commit([createDocument(ADDRESS, "latest", BASE, JSON.stringify(BASE.value))], fence, context); + return adopted[0]![1][0]!; +} + +/** A staged write whose step fails in the backend. */ +const BROKEN: StagedWrite = { + order: "write", + apply: (step) => + step.run("INSERT INTO no_such_table VALUES (1)").then(() => ({ kind: "document.retire", id: 0, revision: 0 })), +}; + +let directory: string | undefined; + +afterEach(async () => { + if (directory !== undefined) await rm(directory, { recursive: true, force: true }); + directory = undefined; +}); + +describe("opening", () => { + it("creates a new store at seq 0 without an outbox, and reopens a file at its last seq and ids", async () => { + directory = await mkdtemp(join(tmpdir(), "chord-store-")); + const path = join(directory, "store.sqlite"); + const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context)) as ChordStore; + expect([store.lastSeq, store.outbox]).toEqual([0, undefined]); + expect(await created(store)).toBe(1); + await store.close(context); + const reopened = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context)) as ChordStore; + expect(reopened.lastSeq).toBe(1); + const { adopted, fence } = recording(); + await reopened.commit([createDocument({ ...ADDRESS, key: "b" }, "latest", BASE, "{}")], fence, context); + expect(adopted).toEqual([[2, [2]]]); + await reopened.close(context); + }); + + it("refuses a database that is not UTF-8, closing the backend (rule 12)", async () => { + const db = await openNodeSqliteDatabase(":memory:"); + await db.exec('PRAGMA encoding = "UTF-16le"'); + await expect(openStore(sqliteBackend(db), context)).rejects.toThrow("not UTF-8"); + await expect(db.get("SELECT 1")).rejects.toThrow(); + }); +}); + +describe("the commit step", () => { + it("assigns one seq per commit, applies retirements before creations, and adopts before the next step", async () => { + const store = await memoryStore(); + const id = await created(store); + const events: string[] = []; + const fence = (name: string) => ({ + admit: () => true, + adopt: (seq: number, ids: readonly number[]) => void events.push(`${name} ${seq} ${ids}`), + }); + const first = store.commit( + [createDocument(ADDRESS, "latest", BASE, "{}"), retireDocument(id, 1)], + fence("recreate"), + context, + ); + const second = store.commit( + [changeDocument(2, 1, { kind: "delta", version: 1, ops: [["s", ["n"], 1]] }, '[["s",["n"],1]]')], + fence("change"), + context, + ); + expect(await Promise.all([first, second])).toEqual([2, 3]); + expect(events).toEqual(["recreate 2 2", "change 3 "]); + expect(store.lastSeq).toBe(3); + }); + + it("rejects a commit its fence refuses at the entry released, writing nothing; an empty commit takes no seq", async () => { + const store = await memoryStore(); + const refused = store.commit( + [createDocument(ADDRESS, "latest", BASE, "{}")], + { admit: () => false, adopt: () => {} }, + context, + ); + expect(await rejection(refused)).toMatchObject({ code: "released" }); + const { adopted, fence } = recording(); + expect(await store.commit([], fence, context)).toBe(0); + expect(adopted).toEqual([[0, []]]); + expect(await created(store)).toBe(1); + }); + + it("numbers up to Number.MAX_SAFE_INTEGER after an import near it, then fails storage_failed: no seq repeats", async () => { + const [manifest, ...rest] = await exported(await memoryStore()); + const parts = [{ ...manifest!, lastSeq: Number.MAX_SAFE_INTEGER - 1 }, ...rest]; + const store = (await importStore( + sqliteBackend(await openNodeSqliteDatabase(":memory:")), + parts, + context, + )) as ChordStore; + expect(await created(store)).toBe(1); + expect(store.lastSeq).toBe(Number.MAX_SAFE_INTEGER); + const refused = store.commit( + [createDocument({ ...ADDRESS, key: "b" }, "latest", BASE, "{}")], + recording().fence, + context, + ); + expect(await rejection(refused)).toMatchObject({ code: "unavailable", reason: "storage_failed" }); + expect(store.lastSeq).toBe(Number.MAX_SAFE_INTEGER); + expect(await rejection(store.read(async () => null, context))).toMatchObject({ reason: "storage_failed" }); + }); + + it("counts one commit and three statements for a delta, four with an outbox (cost guard)", async () => { + for (const [outbox, statements] of [ + [false, 3], + [true, 4], + ] as const) { + const store = await memoryStore({ outbox }); + const id = await created(store); + const telemetry = await metered(); + const ops = [["s", ["n"], 1]] as const; + await store.commit( + [changeDocument(id, 1, { kind: "delta", version: 1, ops }, JSON.stringify(ops))], + recording().fence, + telemetry.context, + ); + expect(telemetry.counts()).toEqual({ "chord.store.commits": 1, "chord.store.statements": statements }); + } + }); +}); + +describe("storage failures", () => { + it("stop the store: the commit rejects storage_failed, later commits and reads too; the host is told once", async () => { + const store = await memoryStore(); + const told: unknown[] = []; + store.attach((error) => told.push(error)); + expect(await rejection(store.commit([BROKEN], recording().fence, context))).toMatchObject({ + code: "unavailable", + reason: "storage_failed", + }); + expect(await rejection(store.commit([BROKEN], recording().fence, context))).toMatchObject({ + reason: "storage_failed", + }); + expect(await rejection(store.read(async () => 1, context))).toMatchObject({ reason: "storage_failed" }); + expect(told.length).toBe(1); + expect(store.lastSeq).toBe(0); + await store.close(context); + await store.close(context); + }); + + it("include an adopt that throws and a conditional update that finds another revision", async () => { + const store = await memoryStore(); + const id = await created(store); + const throwing = { + admit: () => true, + adopt: () => { + throw new Error("adopt threw"); + }, + }; + expect(await rejection(store.commit([retireDocument(id, 1)], throwing, context))).toMatchObject({ + reason: "storage_failed", + }); + // An empty commit's adopt too. + const empty = await memoryStore(); + expect(await rejection(empty.commit([], throwing, context))).toMatchObject({ reason: "storage_failed" }); + expect(await rejection(empty.read(async () => 1, context))).toMatchObject({ reason: "storage_failed" }); + const other = await memoryStore(); + const otherId = await created(other); + const stale = changeDocument(otherId, 7, { kind: "delta", version: 1, ops: [] }, "[]"); + expect(await rejection(other.commit([stale], recording().fence, context))).toMatchObject({ + reason: "storage_failed", + }); + expect(other.lastSeq).toBe(1); + }); + + it("spare a read whose context was aborted (aborted), and reject a read that settles after the stop", async () => { + const store = await memoryStore(); + const cancel = withCancel(context); + cancel.cancel(); + const failing = store.read(() => Promise.reject(new Error("backend")), cancel.context); + expect(await rejection(failing)).toMatchObject({ code: "aborted" }); + const gate = Promise.withResolvers(); + const late = store.read(() => gate.promise, context); + await rejection(store.commit([BROKEN], recording().fence, context)); + gate.resolve(1); + expect(await rejection(late)).toMatchObject({ reason: "storage_failed" }); + }); +}); + +describe("one host per store (rule 7)", () => { + it("refuses a second listener until the first detaches", async () => { + const store = await memoryStore(); + store.attach(() => {}); + expect(() => store.attach(() => {})).toThrow("already attached"); + store.detach(); + store.attach(() => {}); + }); +}); + +describe("close", () => { + it("waits for the running step, rejects queued commits and later reads released, and closes the backend", async () => { + const store = await memoryStore(); + const gate = Promise.withResolvers(); + const entered = Promise.withResolvers(); + const create = createDocument(ADDRESS, "latest", BASE, "{}"); + const held: StagedWrite = { + order: "create", + apply: async (step) => { + entered.resolve(); + await gate.promise; + return create.apply(step); + }, + }; + const running = store.commit([held], recording().fence, context); + await entered.promise; + const queued = store.commit( + [createDocument({ ...ADDRESS, key: "q" }, "latest", BASE, "{}")], + recording().fence, + context, + ); + const closed = (store as Store).close(context); + gate.resolve(); + expect(await running).toBe(1); + expect(await rejection(queued)).toMatchObject({ code: "released" }); + await closed; + expect(await rejection(store.read(async () => 1, context))).toMatchObject({ code: "released" }); + }); +}); diff --git a/packages/chord/test/next/store/table-changes.test.ts b/packages/chord/test/next/store/table-changes.test.ts new file mode 100644 index 000000000..2f51e3e37 --- /dev/null +++ b/packages/chord/test/next/store/table-changes.test.ts @@ -0,0 +1,84 @@ +// A transaction's table writes (slice 6a §6, §7): staging, and the `tx.get` overlay over committed rows. + +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, opt, str } from "../../../src/next/contracts/index.ts"; +import { add, eq, prefix } from "../../../src/next/store/expression.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import { memoryStore, rejection, tableIn } from "../support.ts"; + +const Edits = defineTable("edits", { + version: 1, + columns: { id: str(), path: str(), hits: int({ default: 0 }), note: opt(str()) }, + key: ["id"], +}); +const FENCE = { admit: () => true, adopt: () => {} }; + +let store: ChordStore; +afterEach(async () => { + await store?.close(context); +}); + +/** Changes to a table holding `e1` (committed); the staged list they append to. */ +async function seeded(): Promise<{ readonly changes: TableChanges; readonly staged: StagedWrite[] }> { + store = await memoryStore(); + const edits = await tableIn(store, Edits); + const seed: StagedWrite[] = []; + new TableChanges(edits, seed).put({ id: "e1", path: "a/1", hits: 1 }); + await store.commit(seed, FENCE, context); + const staged: StagedWrite[] = []; + return { changes: new TableChanges(edits, staged), staged }; +} + +describe("staging", () => { + it("checks at the call, appends in order, and touches no backend before the commit", async () => { + const { changes, staged } = await seeded(); + changes.put({ id: "e2", path: "a/2" }); + expect(() => changes.put({ id: "e3" })).toThrow('column "path" is missing'); + expect(() => changes.update({ id: "e1" }, { id: "x" })).toThrow("cannot change"); + changes.update({ id: "e1" }, { note: "n" }); + changes.delete({ id: "e9" }); + expect(staged).toHaveLength(3); + expect( + await store.read((executor) => executor.get(`SELECT count(*) AS n FROM "session:app.store:edits"`), context), + ).toEqual({ n: 1 }); + }); +}); + +describe("the tx.get overlay", () => { + it("replays the transaction's own put, update and delete over the committed row", async () => { + const { changes } = await seeded(); + changes.update({ id: "e1" }, (e: { hits: never }) => ({ hits: add(e.hits, 2) })); + expect(await changes.get({ id: "e1" }, context)).toEqual({ id: "e1", path: "a/1", hits: 3 }); + changes.update({ id: "e1" }, { note: "n" }); + expect(await changes.get({ id: "e1" }, context)).toEqual({ id: "e1", path: "a/1", hits: 3, note: "n" }); + changes.delete({ id: "e1" }); + expect(await changes.get({ id: "e1" }, context)).toBeUndefined(); + changes.put({ id: "e1", path: "b" }); + changes.update({ id: "e1" }, (e: { hits: never }) => ({ hits: add(e.hits, 1) })); + expect(await changes.get({ id: "e1" }, context)).toEqual({ id: "e1", path: "b", hits: 1 }); + }); + + it("an update of an absent key changes nothing", async () => { + const { changes } = await seeded(); + changes.update({ id: "e9" }, { note: "n" }); + expect(await changes.get({ id: "e9" }, context)).toBeUndefined(); + }); + + it("is refused on a table after an updateWhere or deleteWhere (W10); other tables are unaffected", async () => { + const { changes } = await seeded(); + expect(await changes.get({ id: "e1" }, context)).toBeDefined(); + changes.updateWhere((e: { path: never }) => prefix(e.path, "a/"), { note: "bulk" }); + expect(await rejection(changes.get({ id: "e1" }, context))).toMatchObject({ + code: "invalid", + reason: "read_after_bulk", + }); + const other = new TableChanges(changes.handle, []); + other.deleteWhere((e: { id: never }) => eq(e.id, "x")); + expect(await rejection(other.get({ id: "e1" }, context))).toMatchObject({ reason: "read_after_bulk" }); + expect(await new TableChanges(changes.handle, []).get({ id: "e1" }, context)).toBeDefined(); + }); +}); diff --git a/packages/chord/test/next/store/table-evolve.test.ts b/packages/chord/test/next/store/table-evolve.test.ts new file mode 100644 index 000000000..8118de4bd --- /dev/null +++ b/packages/chord/test/next/store/table-evolve.test.ts @@ -0,0 +1,225 @@ +// Tests of `evolveTable` (slice 6a §9): versions appended and materialized at declaration, the edits recorded before +// and after the data step, their checks, the editor rule shared with contracts, and the types along the chain. + +import { describe, expect, expectTypeOf, it } from "vitest"; +import { int, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import type { RowOf } from "../../../src/next/store/columns.ts"; +import { add, eq } from "../../../src/next/store/expression.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { defineTable, type TableDefinition } from "../../../src/next/store/table.ts"; +import { evolveTable } from "../../../src/next/store/table-evolve.ts"; + +const EditsV1 = defineTable("edits", { + version: 1, + columns: { id: str(), file: str(), at: int(), meta: opt(json(obj({ a: int() }), { version: 1 })) }, + key: ["id"], + indexes: { by_file: ["file", "at"] }, +}); + +describe("versions", () => { + it("each evolveTable appends version + 1, materialized and fingerprinted, linked to the one before", () => { + const V2 = evolveTable(EditsV1, (t) => t.add("dir", str({ default: "" }))); + const V3 = evolveTable(V2, (t) => t.index("by_dir", ["dir"])); + expect([V2.version, V3.version]).toEqual([2, 3]); + expect(V3.evolution?.previous).toBe(V2); + expect(V2.evolution?.previous).toBe(EditsV1); + expect(new Set([EditsV1.fingerprint, V2.fingerprint, V3.fingerprint]).size).toBe(3); + expect(V3.indexes).toEqual( + new Map([ + ["by_file", ["file", "at"]], + ["by_dir", ["dir"]], + ]), + ); + expectTypeOf(V3.version).toEqualTypeOf<3>(); + }); + + it("the edit must return the editor it was given (contracts' rule)", () => { + expect(() => evolveTable(EditsV1, () => ({}) as never)).toThrow( + "edits: evolveTable must return the editor it was given", + ); + }); +}); + +describe("edits", () => { + it("are recorded before and after the data step; the step reads the columns as they stand there", () => { + const V2 = evolveTable(EditsV1, (t) => + t + .rename("file", "path") + .add("dir", str({ default: "" })) + .rows((row) => ({ dir: row.path })) + .drop("at") + .index("by_dir", ["dir"]), + ); + const migration = V2.evolution!.migration; + expect(migration.before).toEqual([ + { kind: "rename", from: "file", to: "path" }, + { kind: "add", name: "dir", column: { type: "str", default: "" } }, + ]); + expect(migration.after).toEqual([ + { kind: "dropIndex", name: "by_file" }, + { kind: "drop", name: "at" }, + { kind: "index", name: "by_dir", columns: ["dir"] }, + ]); + const step = migration.step!; + expect(step.kind).toBe("rows"); + expect(step.table.columns.map(({ name }) => name)).toEqual(["id", "path", "at", "meta", "dir"]); + expect(V2.columns.map(({ name }) => name)).toEqual(["id", "path", "meta", "dir"]); + expect(V2.indexes).toEqual(new Map([["by_dir", ["dir"]]])); + }); + + it("a rename carries the key and the indexes", () => { + const V2 = evolveTable(EditsV1, (t) => t.rename("id", "uid").rename("file", "path")); + expect(V2.key).toEqual(["uid"]); + expect(V2.indexes).toEqual(new Map([["by_file", ["path", "at"]]])); + expectTypeOf ? C : never>>().toEqualTypeOf< + { uid: string; path: string; at: number } & { meta?: { a: number } } + >(); + }); + + it("an update step is compiled at declaration against the columns there", () => { + const V2 = evolveTable(EditsV1, (t) => + t.update({ where: (e) => eq(e.file, "x"), set: (e) => ({ at: add(e.at, 1) }) }), + ); + expect(V2.evolution!.migration.step).toMatchObject({ + kind: "update", + where: { eq: ["file", "x"] }, + set: { at: { add: [{ col: "at" }, 1] } }, + }); + expect(() => + evolveTable(EditsV1, (t) => + // @ts-expect-error: key columns cannot change (rule 13) + t.update({ set: { id: "x" } }), + ), + ).toThrow('key column "id" cannot change'); + }); + + it("a json edit changes the column's version and keeps its nullability", () => { + const V2 = evolveTable(EditsV1, (t) => + t.json( + "meta", + json(obj({ a: int(), b: int() }), { version: 2, migrate: (old) => ({ ...(old as { a: number }), b: 0 }) }), + ), + ); + expect(V2.byName.get("meta")).toMatchObject({ type: "json", optional: true }); + expect(V2.evolution!.migration.before).toEqual([ + { kind: "json", name: "meta", column: expect.objectContaining({ type: "json", optional: true, version: 2 }) }, + ]); + expectTypeOf ? C : never>>().toEqualTypeOf< + { id: string; file: string; at: number } & { meta?: { a: number; b: number } } + >(); + }); +}); + +describe("checks", () => { + it("a rename may change only the case of its own column", () => { + expect(evolveTable(EditsV1, (t) => t.rename("at", "AT")).byName.has("AT")).toBe(true); + }); + + it.each<[string, () => unknown, string]>([ + ["rename to an existing column", () => evolveTable(EditsV1, (t) => t.rename("file", "at")), 'column "at" exists'], + [ + "rename an unknown column", + () => evolveTable(EditsV1, (t) => t.rename("nope" as "file", "x")), + 'no column "nope"', + ], + [ + "add a column without opt(...) or a default", + // @ts-expect-error: an added column is opt(...) or has a default + () => evolveTable(EditsV1, (t) => t.add("dir", str())), + "needs opt(...) or a default", + ], + [ + "drop a key column", + // @ts-expect-error: a key column cannot be dropped + () => evolveTable(EditsV1, (t) => t.drop("id")), + '"id" is a key column', + ], + [ + "rename after a drop", + // @ts-expect-error: rename comes before drop, index and dropIndex + () => evolveTable(EditsV1, (t) => t.drop("at").rename("file", "path")), + "rename comes before the data step", + ], + [ + "a second data step", + // @ts-expect-error: at most one data step + () => evolveTable(EditsV1, (t) => t.rows(() => ({})).rows(() => ({}))), + "rows comes before the data step", + ], + ["an existing index", () => evolveTable(EditsV1, (t) => t.index("by_file", ["at"])), 'index "by_file" exists'], + ["dropIndex of an unknown index", () => evolveTable(EditsV1, (t) => t.dropIndex("nope")), 'no index "nope"'], + [ + "an index over an opt(...) column", + // @ts-expect-error: index columns are non-opt str() or int() columns + () => evolveTable(EditsV1, (t) => t.index("by_meta", ["meta"])), + "is not a non-opt str() or int() column", + ], + [ + "a json edit of a scalar column", + // @ts-expect-error: only json(...) columns take a json edit + () => evolveTable(EditsV1, (t) => t.json("file", json(obj({}), { version: 2 }))), + "is not a json(...) column", + ], + [ + "a json edit changing the schema at the cell's version (D9)", + () => evolveTable(EditsV1, (t) => t.json("meta", json(obj({ b: str() }), { version: 1 }))), + "changes its schema at version 1", + ], + [ + "a json edit going back a version", + () => + evolveTable( + evolveTable(EditsV1, (t) => t.json("meta", json(obj({ a: int() }), { version: 2 }))), + (t) => t.json("meta", json(obj({ a: int() }), { version: 1 })), + ), + "goes back from version 2", + ], + [ + "an add colliding by case with a column a later drop removes", + () => evolveTable(EditsV1, (t) => t.add("File", str({ default: "" })).drop("file")), + 'column "file" exists', + ], + [ + "a rename colliding by case with another column", + () => evolveTable(EditsV1, (t) => t.rename("at", "FILE")), + 'column "file" exists', + ], + [ + "a name folding to a column's only beyond ASCII (SQLite folds ASCII): the name rule, not a collision", + () => { + const Kinds = defineTable("kinds", { version: 1, columns: { id: str(), kind: str() }, key: ["id"] }); + return evolveTable(Kinds, (t) => t.add("\u212Aind", opt(str()))); + }, + "expected [A-Za-z_][A-Za-z0-9_]*", + ], + [ + "an index on a column missing at that edit, though a later dropIndex removes the index", + // @ts-expect-error: index columns are the table's columns at that edit + () => evolveTable(EditsV1, (t) => t.index("bad", ["missing"]).dropIndex("bad")), + 'index "bad": no column "missing"', + ], + [ + "an index on a column dropped before it", + // @ts-expect-error: the dropped column is gone at this edit + () => evolveTable(EditsV1, (t) => t.drop("at").index("by_at", ["at"])), + 'index "by_at": no column "at"', + ], + [ + "a rename to a name breaking the column name rule, though a later rename fixes it", + () => evolveTable(EditsV1, (t) => t.rename("at", "bad\u0000name").rename("bad\u0000name", "when")), + 'rename: column "bad\u0000name": expected [A-Za-z_][A-Za-z0-9_]*', + ], + [ + "a rename to a name breaking the column name rule, though a later drop removes it", + () => evolveTable(EditsV1, (t) => t.rename("at", "1st").drop("1st")), + 'rename: column "1st": expected', + ], + [ + "an index name outside [a-z0-9_]+", + () => evolveTable(EditsV1, (t) => t.index("By", ["file"])), + "expected [a-z0-9_]+", + ], + ])("refuses %s at declaration", (_, declare, message) => { + expect(declare).toThrow(message); + }); +}); diff --git a/packages/chord/test/next/store/table-handle.test.ts b/packages/chord/test/next/store/table-handle.test.ts new file mode 100644 index 000000000..7a51ef960 --- /dev/null +++ b/packages/chord/test/next/store/table-handle.test.ts @@ -0,0 +1,136 @@ +// Table handles (slice 6a §7): reads outside a transaction, and cells the code cannot read. + +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { bool, int, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { eq, gte, isNull, prefix } from "../../../src/next/store/expression.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { query } from "../../../src/next/store/query.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import type { TableHandle } from "../../../src/next/store/table-handle.ts"; +import { memoryStore, rejection, tableIn, tableOwner } from "../support.ts"; + +const Edits = defineTable("edits", { + version: 1, + columns: { + id: str(), + path: str(), + at: int(), + done: bool({ default: false }), + note: opt(str()), + meta: json(obj({ a: int() }), { version: 1 }), + }, + key: ["id"], + indexes: { by_path: ["path", "at"] }, +}); + +let store: ChordStore; +afterEach(async () => { + await store?.close(context); +}); + +/** A table with `count` rows e0..e(count-1): path `p/`, at `i`. */ +async function filled(count: number, owner = tableOwner()): Promise { + store = await memoryStore(); + const edits = await tableIn(store, Edits, owner); + const staged: StagedWrite[] = []; + const changes = new TableChanges(edits, staged); + for (let i = 0; i < count; i++) + changes.put({ id: `e${String(i).padStart(3, "0")}`, path: `p/${i % 3}`, at: i, meta: { a: i } }); + await store.commit(staged, { admit: () => true, adopt: () => {} }, context); + return edits; +} + +describe("reads outside a transaction", () => { + it("get takes a key object and reads committed rows as stored", async () => { + const edits = await filled(3); + expect(await edits.get({ id: "e001" }, context)).toEqual({ + id: "e001", + path: "p/1", + at: 1, + done: false, + meta: { a: 1 }, + }); + expect(await edits.get({ id: "nope" }, context)).toBeUndefined(); + expect(await rejection(edits.get({ id: "e001", path: "x" }, context))).toMatchObject({ + code: "invalid", + reason: "args", + }); + }); + + it("page fetches limit + 1 for next, and the cursor continues in order with the key tie-break", async () => { + const edits = await filled(10); + const q = query(Edits) + .where((e) => prefix(e.path, "p/")) + .orderBy((e) => e.path, "desc"); + const seen: string[] = []; + let after: string | undefined; + do { + const page = await edits.page(q, 3, context, after === undefined ? {} : { after }); + expect(page.items.length).toBeLessThanOrEqual(3); + seen.push(...page.items.map((row) => `${row.path}:${row.id}`)); + after = page.next; + } while (after !== undefined); + const all = (await edits.page(q, 100, context)).items.map((row) => `${row.path}:${row.id}`); + expect(seen).toEqual(all); + expect(all.slice(0, 4)).toEqual(["p/2:e008", "p/2:e005", "p/2:e002", "p/1:e007"]); + expect((await edits.page(q, 4, context)).next).toBeDefined(); + expect((await edits.page(query(Edits), 10, context)).next).toBeUndefined(); + }); + + it("rows reads every page; count counts", async () => { + const edits = await filled(250); + const ids: string[] = []; + for await (const row of edits.rows( + query(Edits).where((e) => gte(e.at, 10)), + context, + )) + ids.push(row.id as string); + expect(ids).toHaveLength(240); + expect(ids[0]).toBe("e010"); + expect( + await edits.count( + query(Edits).where((e) => eq(e.path, "p/0")), + context, + ), + ).toBe(84); + expect( + await edits.count( + query(Edits).where((e) => isNull(e.note)), + context, + ), + ).toBe(250); + }); + + it("refuses a query of another table, a bad limit and a foreign cursor before the backend", async () => { + const edits = await filled(1); + const Other = defineTable("other", { version: 1, columns: { id: str() }, key: ["id"] }); + expect(await rejection(edits.page(query(Other) as never, 1, context))).toMatchObject({ reason: "args" }); + expect(await rejection(edits.page(query(Edits), 0, context))).toMatchObject({ reason: "args" }); + expect(await rejection(edits.page(query(Edits), 1.5, context))).toMatchObject({ reason: "args" }); + expect(await rejection(edits.page(query(Edits), 1, context, { after: "x" }))).toMatchObject({ reason: "args" }); + }); + + it("reads go through the store's guard: refused once the store closed", async () => { + const edits = await filled(1); + await store.close(context); + expect(await rejection(edits.get({ id: "e000" }, context))).toMatchObject({ code: "released" }); + }); +}); + +describe("unreadable cells", () => { + it("a json cell of a newer version fails the owner and the read, the data intact", async () => { + const owner = tableOwner(); + const edits = await filled(1, owner); + await store.read((executor) => executor.run(`UPDATE "session:app.store:edits" SET "meta$v" = 2`), context); + const error = await rejection(edits.get({ id: "e000" }, context)); + expect(error).toMatchObject({ code: "unavailable" }); + expect(owner.failures).toHaveLength(1); + expect( + await store.read((executor) => executor.get(`SELECT meta FROM "session:app.store:edits"`), context), + ).toEqual({ meta: '{"a":0}' }); + }); +}); diff --git a/packages/chord/test/next/store/table-history.test.ts b/packages/chord/test/next/store/table-history.test.ts new file mode 100644 index 000000000..957e679f9 --- /dev/null +++ b/packages/chord/test/next/store/table-history.test.ts @@ -0,0 +1,73 @@ +// Tests of a table's history (slice 6a §9): what an activation does with the recorded version: create, use, migrate +// from there (resuming a `migrating` one), or refuse a version the code cannot reach. + +import { describe, expect, it } from "vitest"; +import { int, obj, str } from "../../../src/next/contracts/index.ts"; +import type { TableRecord } from "../../../src/next/store/catalog.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { defineTable, type TableDefinition } from "../../../src/next/store/table.ts"; +import { evolveTable } from "../../../src/next/store/table-evolve.ts"; +import { planTable } from "../../../src/next/store/table-history.ts"; + +const V1 = defineTable("edits", { version: 1, columns: { id: str(), at: int() }, key: ["id"] }); +const V2 = evolveTable(V1, (t) => t.add("dir", str({ default: "" })).rows(() => ({}))); +const V3 = evolveTable(V2, (t) => t.index("by_at", ["at"])); + +/** The record of `definition` in `state`. */ +function recorded(definition: TableDefinition, state: "ready" | "migrating" = "ready"): TableRecord { + const { version, record: columns, key, fingerprint } = definition; + return { table: "t", version, state, columns, key, fingerprint, indexes: {} }; +} + +describe("plans", () => { + it("create a table the store does not have; use one recorded at the newest version", () => { + expect(planTable(V3, undefined, "app.store")).toEqual({ kind: "create" }); + expect(planTable(V3, recorded(V3), "app.store")).toEqual({ kind: "ready" }); + }); + + it("migrate through every later version, oldest first; resume a version recorded migrating", () => { + expect(planTable(V3, recorded(V1), "app.store")).toEqual({ kind: "migrate", versions: [V2, V3], first: "begin" }); + expect(planTable(V3, recorded(V2, "migrating"), "app.store")).toEqual({ + kind: "migrate", + versions: [V2, V3], + first: "resume", + }); + }); +}); + +describe("refusals", () => { + it.each<[string, TableDefinition, TableRecord, string]>([ + ["a newer version (old code)", V2, recorded(V3), "version 3, newer than the code's 2"], + [ + "a version older than a new base", + defineTable("edits", { version: 7, columns: { id: str() }, key: ["id"] }), + recorded(V3), + "older than the history's base 7", + ], + [ + "a history edited in place", + evolveTable(V1, (t) => t.add("dir", str({ default: "x" }))), + recorded(V2), + "recorded with another definition", + ], + [ + "a json(...) column with another schema at the same version (D9)", + defineTable("notes", { + version: 1, + columns: { id: str(), meta: json(obj({ b: int() }), { version: 1 }) }, + key: ["id"], + }), + recorded( + defineTable("notes", { + version: 1, + columns: { id: str(), meta: json(obj({ a: int() }), { version: 1 }) }, + key: ["id"], + }), + ), + "recorded with another definition", + ], + ["a migrating version without a data step", V3, recorded(V3, "migrating"), "has no data step to resume"], + ])("%s", (_, definition, record, message) => { + expect(() => planTable(definition, record, "app.store")).toThrow(message); + }); +}); diff --git a/packages/chord/test/next/store/table-migration.test.ts b/packages/chord/test/next/store/table-migration.test.ts new file mode 100644 index 000000000..c55ae4b96 --- /dev/null +++ b/packages/chord/test/next/store/table-migration.test.ts @@ -0,0 +1,329 @@ +// Tests of tables brought to their version at activation (slice 6a §9): fresh tables created at the newest version, +// eager migrations one version at a time in their commits ((A), chunks of 100, (Z)), `rows` and `update` steps, +// index edits, resumption at `migrating`, and the refusals that leave the data as it was. + +import { afterEach, describe, expect, it, vi } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, nul, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { lt } from "../../../src/next/store/expression.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { query } from "../../../src/next/store/query.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable, type TableDefinition } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import { evolveTable } from "../../../src/next/store/table-evolve.ts"; +import { TableHandle } from "../../../src/next/store/table-handle.ts"; +import { MIGRATION_CHUNK, prepareTables } from "../../../src/next/store/table-migration.ts"; +import { memoryStore, rejection, tableOwner } from "../support.ts"; + +const TABLE = "client:app.store:edits"; +const OPEN = { admit: () => true, adopt: () => {} }; + +const EditsV1 = defineTable("edits", { + version: 1, + columns: { id: str(), file: str(), at: int() }, + key: ["id"], + indexes: { by_file: ["file"] }, +}); +const EditsV2 = evolveTable(EditsV1, (t) => + t + .rename("file", "path") + .add("dir", str({ default: "" })) + .rows((row) => ({ dir: row.path.slice(0, row.path.lastIndexOf("/")) })), +); +const EditsV3 = evolveTable(EditsV2, (t) => t.drop("dir").index("by_path", ["path", "at"])); + +let store: ChordStore; +afterEach(async () => { + await store?.close(context); +}); + +/** Prepares `definition` as the edits table of `store`; a handle to it. */ +async function prepared(definition: TableDefinition): Promise { + await prepareTables([{ definition, table: TABLE }], store, OPEN, "app.store", context); + return new TableHandle(definition, TABLE, store, tableOwner()); +} + +/** A store whose edits table is at version 1 with `count` rows e000.. (file `d/f`, at `i`). */ +async function filled(count: number): Promise { + store = await memoryStore({ outbox: true }); + const edits = await prepared(EditsV1); + const staged: StagedWrite[] = []; + const changes = new TableChanges(edits, staged); + for (let i = 0; i < count; i++) changes.put({ id: id(i), file: `d${i % 3}/f${i}`, at: i }); + await store.commit(staged, OPEN, context); +} + +function id(i: number): string { + return `e${String(i).padStart(3, "0")}`; +} + +/** The kinds of the writes of every commit after `seq`, each kind with its count. */ +async function commitsAfter(seq: number): Promise { + const commits = await store.outbox!.read(seq, 100, context); + return commits.map(({ writes }) => { + const counts = new Map(); + for (const { kind } of writes) counts.set(kind, (counts.get(kind) ?? 0) + 1); + return [...counts].map(([kind, count]) => (count === 1 ? kind : `${kind} x${count}`)); + }); +} + +/** Every row of `handle`, in key order. */ +async function all(handle: TableHandle, definition: TableDefinition): Promise { + const rows: unknown[] = []; + for await (const row of handle.rows(query(definition), context)) rows.push(row); + return rows; +} + +describe("fresh tables", () => { + it("are created at the newest version, ready, with their indexes, in one commit (no replay)", async () => { + store = await memoryStore({ outbox: true }); + await prepared(EditsV3); + expect(store.table(TABLE)).toMatchObject({ + version: 3, + state: "ready", + fingerprint: EditsV3.fingerprint, + indexes: { by_file: ["path"], by_path: ["path", "at"] }, + }); + expect(await commitsAfter(0)).toEqual([["table", "index x2"]]); + }); + + it("a table at the newest version, ready, is used as recorded: nothing is written", async () => { + await filled(1); + const seq = store.lastSeq; + await prepared(EditsV1); + expect(store.lastSeq).toBe(seq); + }); +}); + +describe("eager migrations", () => { + it("one version at a time: (A), chunks of 100 in key order, (Z) with the later edits and ready", async () => { + await filled(250); + const seq = store.lastSeq; + const edits = await prepared(EditsV3); + expect(await commitsAfter(seq)).toEqual([ + ["alter x2", "table"], + ["update x100"], + ["update x100"], + ["update x50", "table"], + ["alter", "index", "table"], + ]); + expect(store.table(TABLE)).toMatchObject({ + version: 3, + state: "ready", + columns: { id: { type: "str" }, path: { type: "str" }, at: { type: "int" } }, + indexes: { by_file: ["path"], by_path: ["path", "at"] }, + }); + const rows = await all(edits, EditsV3); + expect(rows).toHaveLength(250); + expect(rows[4]).toEqual({ id: "e004", path: "d1/f4", at: 4 }); + }); + + it("the data step reads the columns as they stand at its version and fills the one it adds", async () => { + await filled(3); + const edits = await prepared(EditsV2); + expect(await all(edits, EditsV2)).toEqual([ + { id: "e000", path: "d0/f0", at: 0, dir: "d0" }, + { id: "e001", path: "d1/f1", at: 1, dir: "d1" }, + { id: "e002", path: "d2/f2", at: 2, dir: "d2" }, + ]); + }); + + it("the migration's rows carry the internal form: the patch and the version stamped as $v", async () => { + await filled(1); + const seq = store.lastSeq; + await prepared(EditsV2); + const [, chunk] = await store.outbox!.read(seq, 10, context); + expect(chunk!.writes).toEqual([ + { kind: "update", table: TABLE, key: { id: "e000" }, set: { dir: "d0" }, columns: {}, count: 1, version: 2 }, + expect.objectContaining({ kind: "table", version: 2, state: "ready" }), + ]); + }); + + it("an empty table still runs (A) and (Z); a version without a data step is one commit", async () => { + await filled(0); + const seq = store.lastSeq; + await prepared(EditsV3); + expect(await commitsAfter(seq)).toEqual([["alter x2", "table"], ["table"], ["alter", "index", "table"]]); + }); + + it("a dropped column's indexes are dropped first, in the same commit", async () => { + await filled(2); + const Dropped = evolveTable(EditsV1, (t) => t.drop("file")); + const seq = store.lastSeq; + const edits = await prepared(Dropped); + const [commit] = await store.outbox!.read(seq, 10, context); + expect(commit!.writes.slice(0, 2)).toEqual([ + { kind: "dropIndex", table: TABLE, name: "by_file" }, + { kind: "alter", table: TABLE, version: 2, edit: { kind: "drop", name: "file" } }, + ]); + expect(store.table(TABLE)).toMatchObject({ columns: { id: { type: "str" }, at: { type: "int" } }, indexes: {} }); + expect(await all(edits, Dropped)).toEqual([ + { id: "e000", at: 0 }, + { id: "e001", at: 1 }, + ]); + }); + + it("an update step is one updateWhere over the rows below its version, stamping it, with ready", async () => { + await filled(5); + const Bumped = evolveTable(EditsV1, (t) => t.update({ where: (e) => lt(e.at, 3), set: { file: "old" } })); + const seq = store.lastSeq; + const edits = await prepared(Bumped); + const commits = await store.outbox!.read(seq, 10, context); + expect(commits.map(({ writes }) => writes.map(({ kind }) => kind))).toEqual([ + ["table"], + ["updateWhere", "table"], + ]); + expect(commits[1]!.writes[0]).toMatchObject({ kind: "updateWhere", count: 3, version: 2, set: { file: "old" } }); + expect((await all(edits, Bumped)).map((row) => (row as { file: string }).file)).toEqual([ + "old", + "old", + "old", + "d0/f3", + "d1/f4", + ]); + }); + + it("a json(...) version change records the column's version: no DDL, cells migrate on access", async () => { + const Notes1 = defineTable("edits", { + version: 1, + columns: { id: str(), meta: opt(json(obj({ a: int() }), { version: 1 })) }, + key: ["id"], + }); + const Notes2 = evolveTable(Notes1, (t) => + t.json( + "meta", + json(obj({ a: int(), b: int() }), { version: 2, migrate: (old) => ({ ...(old as { a: number }), b: 0 }) }), + ), + ); + store = await memoryStore({ outbox: true }); + const v1 = await prepared(Notes1); + const staged: StagedWrite[] = []; + new TableChanges(v1, staged).put({ id: "a", meta: { a: 1 } }); + await store.commit(staged, OPEN, context); + const seq = store.lastSeq; + const v2 = await prepared(Notes2); + const [commit] = await store.outbox!.read(seq, 10, context); + expect(commit!.writes).toEqual([ + { + kind: "alter", + table: TABLE, + version: 2, + edit: { kind: "json", name: "meta", column: expect.objectContaining({ type: "json", version: 2 }) }, + }, + expect.objectContaining({ kind: "table", version: 2, state: "ready" }), + ]); + expect(await v2.get({ id: "a" }, context)).toEqual({ id: "a", meta: { a: 1, b: 0 } }); + }); +}); + +describe("the chunk yield", () => { + it("yields between chunks without Node's setImmediate (Workers, browsers), through a message, not a timer", async () => { + await filled(3 * MIGRATION_CHUNK); + const saved = globalThis.setImmediate; + // Runtimes without it: Cloudflare Workers, browsers. + Reflect.deleteProperty(globalThis, "setImmediate"); + const timers = vi.spyOn(globalThis, "setTimeout"); + try { + await prepared(EditsV2); + expect(timers).not.toHaveBeenCalled(); + } finally { + timers.mockRestore(); + globalThis.setImmediate = saved; + } + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready" }); + }); + + it("yields between chunks without a timer (no 1 ms clamp), while timers due run in between (a latency guard)", async () => { + await filled(10 * MIGRATION_CHUNK); + let fired = false; + // Due during the migration: it fires between chunks (a microtask-only loop would starve it). + setTimeout(() => { + fired = true; + }, 0); + const timers = vi.spyOn(globalThis, "setTimeout"); + try { + await prepared(EditsV2); + expect(timers).not.toHaveBeenCalled(); + } finally { + timers.mockRestore(); + } + expect(fired).toBe(true); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready" }); + }); +}); + +describe("patches", () => { + it("a rows patch writes a JSON null into a json(...) cell and clears an opt column with undefined", async () => { + const Cells1 = defineTable("edits", { + version: 1, + columns: { id: str(), data: json(nul(), { version: 1 }), note: opt(str()) }, + key: ["id"], + }); + const Cells2 = evolveTable(Cells1, (t) => + t.add("more", opt(json(nul(), { version: 1 }))).rows(() => ({ more: null, note: undefined })), + ); + store = await memoryStore(); + const v1 = await prepared(Cells1); + const staged: StagedWrite[] = []; + new TableChanges(v1, staged).put({ id: "a", data: null, note: "n" }); + await store.commit(staged, OPEN, context); + const v2 = await prepared(Cells2); + expect(await v2.get({ id: "a" }, context)).toEqual({ id: "a", data: null, more: null }); + }); +}); + +describe("resumption", () => { + it("a data step that throws leaves the table migrating with its committed chunks; the next run continues", async () => { + await filled(250); + let fail = true; + const Failing = evolveTable(EditsV1, (t) => + t.rows((row) => { + if (fail && row.id === id(150)) throw new Error("crash"); + return { file: `${row.file}!` }; + }), + ); + expect(await rejection(prepared(Failing))).toMatchObject({ message: "crash" }); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "migrating" }); + const seq = store.lastSeq; + fail = false; + const edits = await prepared(Failing); + expect(await commitsAfter(seq)).toEqual([["update x100"], ["update x50", "table"]]); + const files = (await all(edits, Failing)).map((row) => (row as { file: string }).file); + // Every row patched once: the first chunk's rows were skipped on the second run. + expect(files.every((file) => file.endsWith("!") && !file.endsWith("!!"))).toBe(true); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready" }); + }); + + it("a refused commit (the activation abandoned) stops the migration between chunks", async () => { + await filled(MIGRATION_CHUNK * 2 + 1); + let admitted = 2; + const fence = { admit: () => admitted-- > 0, adopt: () => {} }; + const migration = prepareTables([{ definition: EditsV2, table: TABLE }], store, fence, "app.store", context); + expect(await rejection(migration)).toMatchObject({ code: "released" }); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "migrating" }); + }); +}); + +describe("refusals", () => { + it.each<[string, () => Promise, string]>([ + ["a newer version recorded", async () => prepared(EditsV1), "newer than the code's 1"], + [ + "a version below the history's base", + async () => prepared(defineTable("edits", { version: 7, columns: { id: str() }, key: ["id"] })), + "older than the history's base 7", + ], + [ + "a history edited in place", + async () => prepared(evolveTable(EditsV1, (t) => t.rename("file", "name"))), + "recorded with another definition", + ], + ])("%s (the store at version 2) throws before anything is written", async (_, prepare, message) => { + await filled(1); + await prepared(EditsV2); + const seq = store.lastSeq; + expect(await rejection(prepare())).toMatchObject({ message: expect.stringContaining(message) }); + expect(store.lastSeq).toBe(seq); + }); +}); diff --git a/packages/chord/test/next/store/table-writes.test.ts b/packages/chord/test/next/store/table-writes.test.ts new file mode 100644 index 000000000..d1bd06c0a --- /dev/null +++ b/packages/chord/test/next/store/table-writes.test.ts @@ -0,0 +1,233 @@ +// Table writes (slice 6a §6, §9): schema writes, row writes as the outbox records them, and alter writes. + +import { afterEach, describe, expect, it } from "vitest"; +import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts"; +import { int, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { add, eq, prefix, replacePrefix } from "../../../src/next/store/expression.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { query } from "../../../src/next/store/query.ts"; +import type { StagedWrite } from "../../../src/next/store/step.ts"; +import type { ChordStore } from "../../../src/next/store/store.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; +import { TableChanges } from "../../../src/next/store/table-changes.ts"; +import type { TableEdit } from "../../../src/next/store/table-history.ts"; +import { stageEdit, stageIndex } from "../../../src/next/store/table-writes.ts"; +import { memoryStore, metered, tableIn } from "../support.ts"; + +const Edits = defineTable("edits", { + version: 2, + columns: { + id: str(), + path: str(), + hits: int({ default: 0 }), + note: opt(str()), + meta: opt(json(obj({}), { version: 3 })), + }, + key: ["id"], + indexes: { by_path: ["path"] }, +}); +const TABLE = "session:app.store:edits"; +const FENCE = { admit: () => true, adopt: () => {} }; + +let store: ChordStore; +afterEach(async () => { + await store?.close(context); +}); + +async function commitWith(fill: (changes: TableChanges) => void): Promise { + const staged: StagedWrite[] = []; + fill(new TableChanges(handle, staged)); + await store.commit(staged, FENCE, context); +} + +let handle: Awaited>; + +async function opened(): Promise { + store = await memoryStore({ outbox: true }); + handle = await tableIn(store, Edits); +} + +describe("schema writes", () => { + it("record the table and its indexes; the table write carries version, state, columns, key, fingerprint", async () => { + await opened(); + const [commit] = await store.outbox!.read(0, 1, context); + expect(commit!.writes).toEqual([ + { + kind: "table", + table: TABLE, + version: 2, + state: "ready", + columns: Edits.record, + key: ["id"], + fingerprint: Edits.fingerprint, + }, + { kind: "index", table: TABLE, name: "by_path", columns: ["path"] }, + ]); + expect(store.table(TABLE)).toMatchObject({ version: 2, state: "ready", indexes: { by_path: ["path"] } }); + const indexes = await store.read( + (executor) => executor.all(`SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = ?`, TABLE), + context, + ); + expect(indexes).toContainEqual({ name: `${TABLE}:by_path` }); + }); + + it("an index write for a table the store does not have fails the step (storage_failed)", async () => { + await opened(); + const write = stageIndex({ kind: "index", table: "session:app.store:nope", name: "x", columns: ["id"] }); + await expect(store.commit([write], FENCE, context)).rejects.toMatchObject({ reason: "storage_failed" }); + }); +}); + +describe("row writes in the outbox", () => { + it("carry what a replica needs: put with v and json versions, update and delete with counts, the canonical set", async () => { + await opened(); + await commitWith((changes) => { + changes.put({ id: "a", path: "x/1", meta: {} }); + changes.put({ id: "b", path: "x/2" }); + changes.update({ id: "a" }, (e: { hits: never }) => ({ hits: add(e.hits, 1), note: undefined, meta: {} })); + changes.update({ id: "zz" }, { note: "n" }); + changes.delete({ id: "b" }); + changes.delete({ id: "b" }); + }); + const [, commit] = await store.outbox!.read(0, 10, context); + expect(commit!.writes).toEqual([ + { + kind: "put", + table: TABLE, + row: { id: "a", path: "x/1", hits: 0, meta: {} }, + version: 2, + columns: { meta: 3 }, + }, + { kind: "put", table: TABLE, row: { id: "b", path: "x/2", hits: 0 }, version: 2, columns: {} }, + { + kind: "update", + table: TABLE, + key: { id: "a" }, + set: { hits: { add: [{ col: "hits" }, 1] }, note: null, meta: { json: {} } }, + columns: { meta: 3 }, + count: 1, + }, + { kind: "update", table: TABLE, key: { id: "zz" }, set: { note: "n" }, columns: {}, count: 0 }, + { kind: "delete", table: TABLE, key: { id: "b" }, count: 1 }, + { kind: "delete", table: TABLE, key: { id: "b" }, count: 0 }, + ]); + }); + + it("a put is an upsert: the whole row replaced", async () => { + await opened(); + await commitWith((changes) => changes.put({ id: "a", path: "x", hits: 5, note: "n" })); + await commitWith((changes) => changes.put({ id: "a", path: "y" })); + expect(await handle.get({ id: "a" }, context)).toEqual({ id: "a", path: "y", hits: 0 }); + }); + + it("updateWhere and deleteWhere are one statement each, recorded with where, set and count", async () => { + await opened(); + await commitWith((changes) => { + for (let i = 0; i < 5; i++) changes.put({ id: `e${i}`, path: i < 3 ? `a/${i}` : `b/${i}` }); + }); + const { counts, context: counting } = await metered(); + const staged: StagedWrite[] = []; + const changes = new TableChanges(handle, staged); + changes.updateWhere( + (e: { path: never }) => prefix(e.path, "a/"), + (e: { path: never }) => ({ path: replacePrefix(e.path, "a/", "c/") }), + ); + changes.deleteWhere((e: { path: never }) => eq(e.path, "b/4")); + await store.commit(staged, FENCE, counting); + // The two bulk writes, the outbox commit, lastSeq. + expect(counts()["chord.store.statements"]).toBe(4); + const [, , commit] = await store.outbox!.read(0, 10, context); + expect(commit!.writes).toEqual([ + { + kind: "updateWhere", + table: TABLE, + where: { prefix: ["path", "a/"] }, + set: { path: { replacePrefix: [{ col: "path" }, "a/", "c/"] } }, + columns: {}, + count: 3, + }, + { kind: "deleteWhere", table: TABLE, where: { eq: ["path", "b/4"] }, count: 1 }, + ]); + expect((await handle.page(query(Edits), 10, context)).items.map((r) => r.path)).toEqual([ + "c/0", + "c/1", + "c/2", + "b/3", + ]); + }); +}); + +describe("alter writes", () => { + /** Applies `edits` to the edits table as the migration to version 3 does, one commit. */ + async function altered(...edits: TableEdit[]): Promise { + await store.commit( + edits.map((edit) => stageEdit(TABLE, 3, edit)), + FENCE, + context, + ); + } + + /** The physical columns of the edits table. */ + async function physical(): Promise { + const rows = await store.read( + (executor) => executor.all<{ readonly name: string }>(`SELECT name FROM pragma_table_info('${TABLE}')`), + context, + ); + return rows.map(({ name }) => name); + } + + it("rename a column: a json(...) one's version column, the key and the indexes follow", async () => { + await opened(); + await commitWith((changes) => changes.put({ id: "a", path: "p" })); + await altered({ kind: "rename", from: "meta", to: "info" }, { kind: "rename", from: "path", to: "file" }); + expect(await physical()).toEqual(["id", "file", "hits", "note", "info", "info$v", "$v"]); + expect(store.table(TABLE)).toMatchObject({ indexes: { by_path: ["file"] } }); + expect(Object.keys(store.table(TABLE)!.columns)).toEqual(["id", "file", "hits", "note", "info"]); + }); + + it("add a column: existing rows get its default; an opt(...) json(...) one adds its version column", async () => { + await opened(); + await commitWith((changes) => changes.put({ id: "a", path: "p" })); + await altered( + { kind: "add", name: "level", column: { type: "int", default: 3 } }, + { kind: "add", name: "extra", column: { type: "json", optional: true, version: 1, fingerprint: "f" } }, + ); + expect(await physical()).toEqual([ + "id", + "path", + "hits", + "note", + "meta", + "meta$v", + "$v", + "level", + "extra", + "extra$v", + ]); + const rows = await store.read((executor) => executor.all(`SELECT level FROM "${TABLE}"`), context); + expect(rows).toEqual([{ level: 3 }]); + expect(store.table(TABLE)!.columns.level).toEqual({ type: "int", default: 3 }); + }); + + it("drop a column (with its version column); a json edit changes the record only", async () => { + await opened(); + await commitWith((changes) => changes.put({ id: "a", path: "p" })); + const column = { type: "json", optional: true, version: 4, fingerprint: "g" } as const; + await altered({ kind: "drop", name: "note" }, { kind: "json", name: "meta", column }); + expect(await physical()).toEqual(["id", "path", "hits", "meta", "meta$v", "$v"]); + expect(store.table(TABLE)!.columns.meta).toEqual(column); + await altered({ kind: "drop", name: "meta" }); + expect(await physical()).toEqual(["id", "path", "hits", "$v"]); + }); + + it("are recorded as alter writes with the migration's version; index edits as index and dropIndex", async () => { + await opened(); + const seq = store.lastSeq; + await altered({ kind: "dropIndex", name: "by_path" }, { kind: "drop", name: "path" }); + const [commit] = await store.outbox!.read(seq, 1, context); + expect(commit!.writes).toEqual([ + { kind: "dropIndex", table: TABLE, name: "by_path" }, + { kind: "alter", table: TABLE, version: 3, edit: { kind: "drop", name: "path" } }, + ]); + }); +}); diff --git a/packages/chord/test/next/store/table.test.ts b/packages/chord/test/next/store/table.test.ts new file mode 100644 index 000000000..70acab99d --- /dev/null +++ b/packages/chord/test/next/store/table.test.ts @@ -0,0 +1,66 @@ +// `defineTable` (slice 6a §5): what a declaration materializes, the checks it makes, and its fingerprint. + +import { describe, expect, it } from "vitest"; +import { int, obj, opt, str } from "../../../src/next/contracts/index.ts"; +import { json } from "../../../src/next/store/json-column.ts"; +import { defineTable } from "../../../src/next/store/table.ts"; + +const columns = { id: str(), path: str(), at: int(), note: opt(str()), meta: json(obj({}), { version: 1 }) }; + +describe("defineTable", () => { + it("materializes columns, key, indexes and the record", () => { + const Edits = defineTable("edits", { version: 2, columns, key: ["id"], indexes: { by_path: ["path", "at"] } }); + expect(Edits.name).toBe("edits"); + expect(Edits.version).toBe(2); + expect(Edits.key).toEqual(["id"]); + expect([...Edits.indexes]).toEqual([["by_path", ["path", "at"]]]); + expect(Object.keys(Edits.record)).toEqual(["id", "path", "at", "note", "meta"]); + expect(Edits.rows.physical).toEqual(["id", "path", "at", "note", "meta", "meta$v", "$v"]); + }); + + it("checks the table name, the version, key, index names and index columns", () => { + expect(() => defineTable("Edits", { version: 1, columns, key: ["id"] })).toThrow("expected lowercase"); + expect(() => defineTable("a:b", { version: 1, columns, key: ["id"] })).toThrow("expected lowercase"); + expect(defineTable("@pi/file-history.edits", { version: 1, columns, key: ["id"] }).name).toBe( + "@pi/file-history.edits", + ); + expect(() => defineTable("edits", { version: 0, columns, key: ["id"] })).toThrow("positive integer"); + expect(() => defineTable("edits", { version: 1, columns, key: [] })).toThrow("no column"); + expect(() => defineTable("edits", { version: 1, columns, key: ["note" as "id"] })).toThrow( + "non-opt str() or int()", + ); + expect(() => defineTable("edits", { version: 1, columns, key: ["meta" as "id"] })).toThrow( + "non-opt str() or int()", + ); + expect(() => defineTable("edits", { version: 1, columns, key: ["id", "id"] })).toThrow("occurs twice"); + expect(() => defineTable("edits", { version: 1, columns, key: ["nope" as "id"] })).toThrow('no column "nope"'); + expect(() => defineTable("edits", { version: 1, columns, key: ["id"], indexes: { ByPath: ["path"] } })).toThrow( + "[a-z0-9_]+", + ); + expect(() => + defineTable("edits", { version: 1, columns, key: ["id"], indexes: { by: ["note" as "path"] } }), + ).toThrow("non-opt"); + }); + + it("fingerprints columns, key and indexes, not their declaration order", () => { + const a = defineTable("edits", { version: 1, columns, key: ["id"], indexes: { x: ["path"], y: ["at"] } }); + const reordered = { meta: columns.meta, note: columns.note, at: columns.at, path: columns.path, id: columns.id }; + const b = defineTable("edits", { + version: 1, + columns: reordered, + key: ["id"], + indexes: { y: ["at"], x: ["path"] }, + }); + expect(b.fingerprint).toBe(a.fingerprint); + expect(defineTable("edits", { version: 1, columns, key: ["id"] }).fingerprint).not.toBe(a.fingerprint); + expect( + defineTable("edits", { version: 1, columns, key: ["id", "at"], indexes: { x: ["path"], y: ["at"] } }) + .fingerprint, + ).not.toBe(a.fingerprint); + const otherMeta = { ...columns, meta: json(obj({ a: int() }), { version: 1 }) }; + expect( + defineTable("edits", { version: 1, columns: otherMeta, key: ["id"], indexes: { x: ["path"], y: ["at"] } }) + .fingerprint, + ).not.toBe(a.fingerprint); + }); +}); diff --git a/packages/chord/test/next/store/text.test.ts b/packages/chord/test/next/store/text.test.ts new file mode 100644 index 000000000..f6db55f3f --- /dev/null +++ b/packages/chord/test/next/store/text.test.ts @@ -0,0 +1,62 @@ +// Text rules (slice 6a §5): code point order, stored text (lone surrogates replaced, NUL refused), prefix bounds. + +import { describe, expect, it } from "vitest"; +import { compareText, prefixUpperBound, storedText } from "../../../src/next/store/text.ts"; + +describe("compareText", () => { + it("orders by code point: a supplementary character after every BMP character, U+E000 to U+FFFF included", () => { + const sorted = ["\u{10000}", "\uFFFF", "\uE000", "\uD7FF", "z", "a", "", "ab"].sort(compareText); + expect(sorted).toEqual(["", "a", "ab", "z", "\uD7FF", "\uE000", "\uFFFF", "\u{10000}"]); + expect(compareText("a", "a")).toBe(0); + expect(compareText("\u{1F600}", "\u{1F601}")).toBeLessThan(0); + }); + + it("agrees with UTF-8 byte order", () => { + const encode = (text: string) => [...new TextEncoder().encode(text)]; + const texts = ["\u{1F600}", "\uFF5E", "\u00E9", "e", "\u{10FFFF}", "\uE000", "aa", "a"]; + const byBytes = [...texts].sort((a, b) => { + const [x, y] = [encode(a), encode(b)]; + for (let i = 0; i < Math.min(x.length, y.length); i++) if (x[i] !== y[i]) return x[i]! - y[i]!; + return x.length - y.length; + }); + expect([...texts].sort(compareText)).toEqual(byBytes); + }); +}); + +describe("storedText", () => { + it("keeps well-formed text as it is and replaces a lone surrogate with U+FFFD", () => { + const text = "a\u{1F600}b"; + expect(storedText(text, "v")).toBe(text); + expect(storedText("a\uD800b", "v")).toBe("a\uFFFDb"); + }); + + it("refuses NUL with invalid/args", () => { + expect(() => storedText("a\0b", "edits.path")).toThrow( + expect.objectContaining({ code: "invalid", reason: "args" }), + ); + }); +}); + +describe("prefixUpperBound", () => { + it("is the next code point after the last one", () => { + expect(prefixUpperBound("a/")).toBe("a0"); + expect(prefixUpperBound("a\u{1F600}")).toBe("a\u{1F601}"); + expect(prefixUpperBound("\uFFFF")).toBe("\u{10000}"); + }); + + it("skips the surrogates and carries past U+10FFFF", () => { + expect(prefixUpperBound("\uD7FF")).toBe("\uE000"); + expect(prefixUpperBound("a\u{10FFFF}")).toBe("b"); + expect(prefixUpperBound("\u{10FFFF}\u{10FFFF}")).toBeUndefined(); + expect(prefixUpperBound("")).toBeUndefined(); + }); + + it("bounds exactly the strings starting with the prefix, in code point order", () => { + const texts = ["a", "a/", "a/x", "a/\u{1F600}", "a/\uFFFF", "a0", "a.", "b"]; + for (const prefix of ["a/", "a", "a/\uFFFF"]) { + const upper = prefixUpperBound(prefix)!; + const inRange = texts.filter((t) => compareText(t, prefix) >= 0 && compareText(t, upper) < 0); + expect(inRange).toEqual(texts.filter((t) => t.startsWith(prefix))); + } + }); +}); diff --git a/packages/chord/test/next/support.ts b/packages/chord/test/next/support.ts new file mode 100644 index 000000000..7ad2b9cd7 --- /dev/null +++ b/packages/chord/test/next/support.ts @@ -0,0 +1,973 @@ +// Test support: cost counters through the in-memory telemetry adapter, GC checks, and hosts, links, watches and +// rejections for the slice tests. + +import { setFlagsFromString } from "node:v8"; +import { runInNewContext } from "node:vm"; +import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry"; +import { afterEach, beforeAll, beforeEach, describe, expect, vi } from "vitest"; +import { BACKGROUND_CONTEXT, type Context, withTelemetry } from "../../src/context/index.ts"; +import type { Op } from "../../src/delta/index.ts"; +import type { JsonValue } from "../../src/json.ts"; +import { + type Address, + arr, + bool, + bytes, + type Contract, + hostKind, + int, + local, + method, + obj, + opt, + type ParamsOf, + remote, + str, +} from "../../src/next/contracts/index.ts"; +import { currentVariants, variantNames, variantOrder } from "../../src/next/contracts/internal.ts"; +import { ChordError, isChordError, PluginError, type Principal } from "../../src/next/data/index.ts"; +import { PROTOCOL_VERSION } from "../../src/next/data/internal.ts"; +import { defineState } from "../../src/next/documents/index.ts"; +import type { Attempt } from "../../src/next/host/attempt.ts"; +import { + type Ctx, + createHost, + defineFacet, + type FacetHost, + type Host, + type HostError, + type RefState, + type Self, + type StoreSelf, +} from "../../src/next/host/index.ts"; +import type { ChordObject } from "../../src/next/host/object.ts"; +import { type Compression, type Link, memoryLinkPair, type PartLimits } from "../../src/next/link/index.ts"; +import { + binaryCodec, + type Codec, + checkMessage, + jsonPart, + type Message, + openPart, + type Part, + parsePart, + textCodec, +} from "../../src/next/link/internal.ts"; +import type { ArchivePart } from "../../src/next/store/archive.ts"; +import { sqliteBackend } from "../../src/next/store/backend.ts"; +import { changeDocument, createDocument, retireDocument } from "../../src/next/store/engine.ts"; +import { json } from "../../src/next/store/json-column.ts"; +import type { StagedWrite } from "../../src/next/store/step.ts"; +import { type ChordStore, openStore, type Store } from "../../src/next/store/store.ts"; +import { defineTable, type TableDefinition } from "../../src/next/store/table.ts"; +import { TableChanges } from "../../src/next/store/table-changes.ts"; +import { evolveTable } from "../../src/next/store/table-evolve.ts"; +import { TableHandle, type TableOwner } from "../../src/next/store/table-handle.ts"; +import { prepareTables } from "../../src/next/store/table-migration.ts"; +import { nodeCompression } from "../../src/next-node/compression.ts"; +import { openNodeSqliteDatabase } from "../../src/sqlite/node.ts"; +import type { NativeWatchHandle, WatchCall, WatchHandle } from "../../src/watch.ts"; + +/** Counts of the `chord.data.*` counters. */ +export type Counts = { walks: number; validations: number; compilations: number }; + +/** Runs `work` with a counting context and returns the counters it added. */ +export async function counted(work: (context: Context) => void): Promise { + const telemetry = new InMemoryTelemetryContext(); + await telemetry.startSpan({ name: "test" }, (span) => work(withTelemetry(span, BACKGROUND_CONTEXT))); + const counts: Counts = { walks: 0, validations: 0, compilations: 0 }; + for (const { name, value } of telemetry.getMetrics()) { + const counter = name.replace("chord.data.", "") as keyof Counts; + counts[counter] += value; + } + return counts; +} + +setFlagsFromString("--expose-gc"); +const gc = runInNewContext("gc") as () => void; + +/** Expects every referent to be collected once nothing else holds it. */ +export async function expectCollected(...refs: WeakRef[]): Promise { + for (let round = 0; round < 10 && refs.some((ref) => ref.deref() !== undefined); round++) { + await new Promise((resolve) => setImmediate(resolve)); + gc(); + } + expect(refs.map((ref) => ref.deref() === undefined)).toEqual(refs.map(() => true)); +} + +/** The heap in use once pending work ran and garbage was collected, repeatedly. */ +export async function settledHeap(): Promise { + for (let round = 0; round < 5; round++) { + await new Promise((resolve) => setImmediate(resolve)); + gc(); + } + return process.memoryUsage().heapUsed; +} + +/** `true` when `A` and `B` are the same type; for type-level tests where `expectTypeOf` cannot compare (bytes). */ +export type Equal = (() => T extends A ? 1 : 2) extends () => T extends B ? 1 : 2 ? true : false; + +/** A telemetry adapter, a context counting through it, and the counters recorded so far, by name. */ +export async function metered(): Promise<{ + readonly telemetry: InMemoryTelemetryContext; + readonly context: Context; + counts(): Record; +}> { + const telemetry = new InMemoryTelemetryContext(); + // A settled span still records through its adapter's metrics. + const span = await telemetry.startSpan({ name: "test" }, (span) => span); + const counts = (): Record => { + const result: Record = {}; + for (const { name, value } of telemetry.getMetrics()) result[name] = (result[name] ?? 0) + value; + return result; + }; + return { telemetry, context: withTelemetry(span, BACKGROUND_CONTEXT), counts }; +} + +/** Waits until queued watch frames have been delivered. */ +export function delivered(): Promise { + return new Promise((resolve) => setTimeout(resolve, 0)); +} + +/** Starts `watch` and collects its frames as `[value, ops]`. */ +export function framesOf(watch: WatchHandle): [T, readonly Op[]][] { + const frames: [T, readonly Op[]][] = []; + watch.start(async (call) => { + if (call.kind === "value") frames.push([call.value, call.ops]); + }); + return frames; +} + +/** Starts `watch` and collects every call it gets: values (the `retired` flag included) and failures. */ +export function changesOf(watch: WatchHandle): WatchCall[] { + const calls: WatchCall[] = []; + watch.start(async (call) => { + calls.push(call); + }); + return calls; +} + +/** The values of `calls` (a failure as `{ failed: message }`), for short expectations. */ +export function valuesOf(calls: readonly WatchCall[]): (T | { readonly failed: string })[] { + return calls.map((call) => (call.kind === "value" ? call.value : { failed: (call.error as Error).message })); +} + +/** + * Fails the live attempt `self` as a runtime failure of its owner does (slice 6a §1.4); `objects` is the test file's + * `instancesOf(ChordObject, "attempt")`, whose objects other than `self`'s ignore it. + */ +export function failLive(objects: readonly WeakRef[], self: Self, error: ChordError): void { + // The factory's `self` is its attempt. + for (const object of objects) object.deref()?.failLive(self as unknown as Attempt, error); +} + +/** A local type whose objects' attempts a test ends by hand (`restartable`); it ends with its last reference. */ +export const Restartable = local("app.restartable", { + version: 1, + params: { id: int() }, + members: { touch: method() }, +}); + +/** A local type with empty params whose attempts a test ends by hand (`storeRestartable`): it may persist (P3). */ +export const StoreRestartable = local("app.store", { version: 1, params: {}, members: { touch: method() } }); + +/** + * Serves `Restartable` with `factory` per attempt (its number from 1) and gets object 1 live. `fail()` fails the live + * attempt as a runtime failure of its owner does (slice 6a §1.4); `retry()` asks for the next attempt and resolves + * once it is live; `restart()` does both. `objects` is the test file's `instancesOf(ChordObject, "attempt")`; `serve` serves other + * types on the same host. + */ +export function restartable( + objects: readonly WeakRef[], + factory: (attempt: number, self: Self) => void, + options: Parameters[1] & { readonly serve?: (host: FacetHost) => void } = {}, +) { + return restartableOf(Restartable, { id: 1 }, objects, factory, options); +} + +/** As `restartable`, for `StoreRestartable`: its factories may declare tables and persisted documents. */ +export function storeRestartable( + objects: readonly WeakRef[], + factory: (attempt: number, self: StoreSelf) => void, + options: Parameters[1] & { readonly serve?: (host: FacetHost) => void } = {}, +) { + return restartableOf(StoreRestartable, {}, objects, factory as (attempt: number, self: Self) => void, options); +} + +async function restartableOf( + Type: C, + params: ParamsOf, + objects: readonly WeakRef[], + factory: (attempt: number, self: Self) => void, + options: Parameters[1] & { readonly serve?: (host: FacetHost) => void }, +) { + const attempts: Self[] = []; + let started = 0; + const { host, errors } = await loaded((h) => { + options.serve?.(h); + // Both types have the one method `touch`; the factory's `self` is the attempt either way. + h.serve(Type as typeof Restartable, (_, self) => { + factory(++started, self); + attempts.push(self); + return { touch: async () => null }; + }); + }, options); + const get = () => host.get(Type as typeof Restartable, params as ParamsOf, BACKGROUND_CONTEXT); + const reference = await get(); + const fail = (): void => { + failLive(objects, attempts.at(-1)!, new ChordError({ code: "unavailable", message: "boom" })); + }; + const retry = async (): Promise => { + reference.retry(); + (await get()).release(); + }; + const restart = async (): Promise => { + fail(); + await retry(); + }; + return { host, errors, reference, attempts, fail, retry, restart }; +} + +/** A host with background errors collected, loaded with one facet (`@test/app`) whose setup is `setup`. */ +export async function loaded( + setup: (host: FacetHost) => void, + options: { + readonly telemetry?: InMemoryTelemetryContext; + readonly context?: Context; + readonly closeTimeout?: number; + readonly readyTimeout?: number; + readonly store?: Store; + } = {}, +): Promise<{ readonly host: Host; readonly errors: HostError[] }> { + const errors: HostError[] = []; + const { telemetry, closeTimeout, readyTimeout, store } = options; + const host = createHost({ telemetry, closeTimeout, readyTimeout, store, onError: (error) => errors.push(error) }); + await host.load([{ name: "@test/app", facet: defineFacet(setup) }], options.context ?? BACKGROUND_CONTEXT); + return { host, errors }; +} + +/** A store on a fresh SQLite `:memory:` database, as Chord's layers see it. */ +export async function memoryStore(options?: { readonly outbox?: boolean }): Promise { + const db = await openNodeSqliteDatabase(":memory:"); + return (await openStore(sqliteBackend(db), BACKGROUND_CONTEXT, options)) as ChordStore; +} + +/** + * Holds `store`'s next commit step inside its backend transaction, after its admission: `entered` resolves there, + * `release()` lets it finish; later commits run as usual. + */ +export function holdNextCommit(store: ChordStore): { readonly entered: Promise; release(): void } { + const gate = Promise.withResolvers(); + const entered = Promise.withResolvers(); + const commit = store.commit.bind(store); + store.commit = (writes, fence, ctx) => { + store.commit = commit; + const held: StagedWrite = { + order: "write", + apply: async () => { + entered.resolve(); + await gate.promise; + return { kind: "dropIndex", table: "x", name: "y" }; + }, + }; + return commit([...writes, held], fence, ctx); + }; + return { entered: entered.promise, release: () => gate.resolve() }; +} + +/** The owner of a test's table handles: its failures collected. */ +export function tableOwner(failures: ChordError[] = []): TableOwner & { readonly failures: ChordError[] } { + return { label: "app.store", scope: {}, assertLive: () => {}, fail: (error) => void failures.push(error), failures }; +} + +/** Creates `definition` (and its indexes) in `store` as `session:app.store:` and returns a handle to it. */ +export async function tableIn( + store: ChordStore, + definition: TableDefinition, + owner = tableOwner(), +): Promise { + const table = `session:app.store:${definition.name}`; + const fence = { admit: () => true, adopt: () => {} }; + await prepareTables([{ definition, table }], store, fence, "app.store", BACKGROUND_CONTEXT); + return new TableHandle(definition, table, store, owner); +} + +/** The error `promise` rejects with, as ChordError JSON when it is one. */ +export async function rejection(promise: Promise): Promise { + try { + await promise; + } catch (error) { + return isChordError(error) ? error.toJSON() : error; + } + throw new Error("expected a rejection"); +} + +/** Fakes `setTimeout` and `Date` (from 0) in the calling `describe`; `setImmediate` stays real for `flushed()`. */ +export function fakeTimers(): void { + beforeEach(() => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"], now: 0 }); + }); + afterEach(() => { + vi.useRealTimers(); + }); +} + +/** Waits until every pending promise callback has run. */ +export function flushed(): Promise { + return new Promise((resolve) => setImmediate(resolve)); +} + +/** The kinds and objects of the reported `activation_failed` errors, with their causes' kinds. */ +export function failures(errors: readonly HostError[]): string[] { + return errors.flatMap((error) => + error.kind === "activation_failed" ? [`${error.object}: ${error.failure.kind}`] : [], + ); +} + +/** The principals of the remote tests. */ +export type TestPrincipal = + | { kind: "owner" } + | { kind: "guest"; session: string } + | { kind: "server" } + | { kind: "worker"; session: string }; + +/** + * A server (principal `server`, loaded with one facet whose setup is `setup`) and a client connected to it, the + * client's link stamped `child`; background errors collected per host. + */ +export async function linked( + setup: (host: FacetHost) => void, + options: { + readonly child?: TestPrincipal; + readonly context?: Context; + readonly telemetry?: InMemoryTelemetryContext; + readonly readyTimeout?: number; + } = {}, +): Promise<{ + readonly server: Host; + readonly client: Host; + readonly errors: { readonly server: HostError[]; readonly client: HostError[] }; + readonly links: readonly [Link, Link]; +}> { + const context = options.context ?? BACKGROUND_CONTEXT; + const errors = { server: [] as HostError[], client: [] as HostError[] }; + const { telemetry, readyTimeout } = options; + const server = createHost({ + principal: { kind: "server" }, + telemetry, + readyTimeout, + onError: (error) => errors.server.push(error), + }); + await server.load([{ name: "@test/app", facet: defineFacet(setup) }], context); + const client = createHost({ telemetry, readyTimeout, onError: (error) => errors.client.push(error) }); + await client.load([], context); + const links = memoryLinkPair(); + server.accept(links[0], { principal: options.child ?? { kind: "owner" }, codec: testCodec() }, context); + client.connect(links[1], { principal: { kind: "server" }, codec: testCodec() }, context); + return { server, client, errors, links }; +} + +/** The incarnation a raw peer greets with. */ +export const RAW_PEER = "00000000-0000-7000-8000-000000000000"; + +/** + * A peer speaking the wire by hand, in the current test codec, at the other end of `link`: it sends `hello` (unless + * `hello: false`, announcing `calls`, default 256), records what arrives, acks every `bound` and `state` with its origin + * (unless `ack: false`: then `ack()` sends one), and sends what a test forges. + */ +export class RawPeer { + readonly received: Message[] = []; + closed = false; + readonly codec: Codec; + readonly #link: Link; + #waiters: (() => void)[] = []; + + constructor( + link: Link, + options: { + readonly hello?: boolean; + readonly calls?: number; + readonly ack?: boolean; + readonly codec?: Codec; + } = {}, + ) { + this.#link = link; + this.codec = options.codec ?? testCodec(); + link.onMessage((bytes) => { + const message = checkMessage(this.codec.decode(bytes), BACKGROUND_CONTEXT); + this.received.push(message); + if (options.ack !== false && (message.kind === "bound" || message.kind === "state")) this.ack(message.origin); + for (const wake of this.#waiters.splice(0)) wake(); + }); + link.onClose(() => { + this.closed = true; + for (const wake of this.#waiters.splice(0)) wake(); + }); + if (options.hello !== false) { + this.send({ kind: "hello", protocol: PROTOCOL_VERSION, incarnation: RAW_PEER, calls: options.calls ?? 256 }); + } + } + + send(message: Message): void { + this.#link.send(this.codec.encode(message)); + } + + /** Acknowledges one `bound` or `state` message of `origin` (none: the peer's own). */ + ack(origin?: number): void { + this.send(origin === undefined ? { kind: "ack" } : { kind: "ack", origin }); + } + + close(): void { + this.#link.close(); + } + + /** Sends a message as raw bytes. */ + sendBytes(message: Uint8Array): void { + this.#link.send(message); + } + + /** The first received message of `kind` not taken yet, waiting for it; throws if the link closes first. */ + async next(kind: K): Promise> { + for (;;) { + const index = this.received.findIndex((message) => message.kind === kind); + if (index >= 0) return this.received.splice(index, 1)[0] as Extract; + if (this.closed) throw new Error(`link closed before ${kind}`); + await new Promise((resolve) => this.#waiters.push(resolve)); + } + } + + /** The kinds received and not taken, in order. */ + kinds(): string[] { + return this.received.map(({ kind }) => kind); + } +} + +/** A part's value as its owner reads it: opened (inflated when deflated) without limits, and parsed. */ +export function partValue(part: Part): unknown { + const unlimited = { part: Number.POSITIVE_INFINITY, decoded: Number.POSITIVE_INFINITY }; + return parsePart(openPart(part, unlimited, nodeCompression, BACKGROUND_CONTEXT)); +} + +/** The codecs every protocol test runs with. */ +export const CODECS: readonly Codec[] = [binaryCodec, textCodec]; + +let currentCodec: Codec = binaryCodec; + +/** The codec of the protocol test running now (`forEachCodec`); links in tests use it at both ends. */ +export function testCodec(): Codec { + return currentCodec; +} + +/** Runs `suite` once per codec, as `binary codec` and `text codec`. */ +export function forEachCodec(suite: () => void): void { + for (const codec of CODECS) { + describe(`${codec.name} codec`, () => { + beforeAll(() => { + currentCodec = codec; + }); + suite(); + }); + } +} + +/** The wire's indexes of a remote contract's variant (2b): the variant's, and its state members' and methods' by name. */ +export function wireNames(Type: Contract, variant: string) { + const names = variantNames(currentVariants(Type).get(variant)!); + const indexes = (list: readonly string[]) => Object.fromEntries(list.map((name, index) => [name, index])); + return { + variant: variantOrder(Type).indexOf(variant), + state: indexes(names.state), + methods: indexes(names.methods), + /** A message's members by index as an object by name, their parts parsed. */ + named: (members: readonly { readonly member: number; readonly ops?: Part; readonly value?: Part }[]) => + Object.fromEntries(members.map(({ member, ops, value }) => [names.state[member], partValue((ops ?? value)!)])), + }; +} + +/** A server host as in `linked` and a raw peer on a link it accepted as `child`, with `limits`. */ +export async function rawLinked( + setup: (host: FacetHost) => void, + options: { + readonly child?: Principal; + readonly hello?: boolean; + readonly readyTimeout?: number; + readonly limits?: PartLimits; + readonly compression?: Compression; + /** What the raw peer is to the host: a child (`accept`, the default) or its parent (`connect`). */ + readonly role?: "child" | "parent"; + } = {}, +): Promise<{ readonly server: Host; readonly peer: RawPeer; readonly errors: HostError[] }> { + const errors: HostError[] = []; + const server = createHost({ + principal: { kind: "server" }, + readyTimeout: options.readyTimeout, + compression: options.compression, + onError: (error) => errors.push(error), + }); + await server.load([{ name: "@test/app", facet: defineFacet(setup) }], BACKGROUND_CONTEXT); + const [a, b] = memoryLinkPair(); + const child = (options.child ?? { kind: "owner" }) as TestPrincipal; + const link = { principal: child, codec: testCodec(), limits: options.limits }; + if (options.role === "parent") server.connect(a, link, BACKGROUND_CONTEXT); + else server.accept(a, link, BACKGROUND_CONTEXT); + return { server, peer: new RawPeer(b, { hello: options.hello }), errors }; +} + +/** Waits until messages and closes in flight on memory links (one task per hop) have arrived. */ +export async function transported(): Promise { + for (let hop = 0; hop < 6; hop++) await flushed(); +} + +/** The variants of the notes types: an editor variant for owners, a reader one for guests. */ +const NOTES_VARIANTS = { + editor: { + state: { notes: arr(obj({ text: str() })), count: int() }, + methods: { + add: method(obj({ text: str() }), int()), + fail: method(obj({ how: str() })), + wait: method(), + echo: method(obj({ data: bytes() }), bytes()), + }, + }, + reader: { state: { notes: arr(obj({ text: str() })), count: int() }, methods: { wait: method() } }, +}; + +/** The remote type of the link tests: notes per session, an editor variant for owners, a reader one for guests. */ +export const Notes = remote("app.notes", { version: 1, params: { session: str() }, variants: NOTES_VARIANTS }); + +/** The kind of the relay tests' workers: one per session. */ +export const Session = hostKind("session", { id: str() }); + +/** `Notes` placed on the session's worker: the relay tests' type. */ +export const PlacedNotes = remote("app.placed-notes", { + version: 1, + params: { session: str() }, + placement: ({ session }) => Session({ id: session }), + variants: NOTES_VARIANTS, +}); + +const NotesDoc = defineState("notes", { + schema: obj({ list: arr(obj({ text: str() })) }), + version: 1, + initial: () => ({ list: [] }), +}); + +/** The shared store behind `Notes` (§6): the notes of a session, once, whoever views them. */ +export const NoteStore = local("app.note-store", { + version: 1, + params: { session: str() }, + members: { list: arr(obj({ text: str() })), add: method(obj({ text: str() }), int()) }, +}); + +/** What `serveNotes` records: the handlers' contexts of `wait` calls, and every factory run of `Notes`. */ +export type NotesProbe = { readonly waits: Ctx[]; readonly factories: string[] }; + +/** + * Serves `NoteStore` and `Notes` over it: `notes` and `count` are derives over the store's list; guests get the reader + * variant. `wait` resolves when its context is aborted; `fail` throws a ChordError (`chord`), a PluginError (`plugin`) + * or a plain Error. + */ +export function serveNotes( + host: FacetHost, + Type: typeof Notes | typeof PlacedNotes = Notes, +): NotesProbe { + const probe: NotesProbe = { waits: [], factories: [] }; + host.serve(NoteStore, (_, self) => { + const doc = self.state(NotesDoc); + return { + list: self.derive([doc], ([d]) => d.list), + add: async ({ text }, ctx) => { + await doc.change((draft) => void draft.list.push({ text }), ctx); + return doc.value.list.length; + }, + }; + }); + // The two types differ in their name and placement only. + host.serve(Type as typeof Notes, ({ session }, principal, self) => { + probe.factories.push(`${session} ${principal.kind}`); + const store = self.get(NoteStore, { session }); + const notes = self.derive([store.list], ([list]) => list); + const count = self.derive([store.list], ([list]) => list.length); + const wait = async (ctx: Ctx) => { + probe.waits.push(ctx); + await new Promise((resolve) => ctx.abortSignal?.addEventListener("abort", resolve)); + return null; + }; + if (principal.kind === "guest") return { variant: "reader", members: { notes, count, wait } }; + return { + variant: "editor", + members: { + notes, + count, + wait, + add: async (args, ctx) => store.add(args, ctx), + fail: async ({ how }) => { + if (how === "chord") throw new ChordError({ code: "forbidden", message: "not yours" }); + if (how === "plugin") throw new PluginError("quota", "too many", { limit: 1 }); + throw new Error("plain"); + }, + echo: async ({ data }) => data, + }, + }; + }); + return probe; +} + +/** Adds a note to the session's store from the server itself; the reference it takes keeps the store. */ +export async function addNote(server: Host, session: string, text: string): Promise { + const store = await server.get(NoteStore, { session }, BACKGROUND_CONTEXT); + await store.add({ text }, BACKGROUND_CONTEXT); +} + +/** A client connected to a raw peer playing the provider (the link's limits `limits`), after the handshake. */ +export async function rawProvider( + options: { readonly readyTimeout?: number; readonly limits?: PartLimits; readonly compression?: Compression } = {}, +) { + const errors: HostError[] = []; + const client = createHost({ + readyTimeout: options.readyTimeout, + compression: options.compression, + onError: (error) => errors.push(error), + }); + await client.load([], BACKGROUND_CONTEXT); + const peer = await rawParent(client, options.limits); + return { client, peer, errors }; +} + +/** + * Connects `host` to a new raw peer playing its parent (a first link, or the next after a drop), the link's limits + * `limits`, after the handshake. + */ +export async function rawParent(host: Host, limits?: PartLimits): Promise { + const [a, b] = memoryLinkPair(); + host.connect(a, { principal: { kind: "server" }, codec: testCodec(), limits }, BACKGROUND_CONTEXT); + const peer = new RawPeer(b); + await peer.next("hello"); + return peer; +} + +/** Starts a `ref.state` watch and collects the states it delivers, by kind (failed ones with their error's message). */ +export function statesOf(watch: NativeWatchHandle): string[] { + const states: string[] = []; + watch.start(async (call) => { + if (call.kind !== "value") return; + const state = call.value; + states.push(state.kind === "failed" ? `failed: ${state.cause.error.message}` : state.kind); + }); + return states; +} + +/** The wire's indexes of `Notes`'s editor variant. */ +export const EDITOR = wireNames(Notes, "editor"); + +/** Answers the next hold with `bound`: the editor variant with `notes` and `count`. */ +export async function bindNext(peer: RawPeer, notes: { text: string }[] = []) { + const { hold } = await peer.next("hold"); + bindHold(peer, hold, notes); + return hold; +} + +/** Sends `bound` for `hold`: the editor variant with `notes` and `count`. */ +export function bindHold(peer: RawPeer, hold: number, notes: { text: string }[] = []): void { + const members = [ + { member: EDITOR.state.notes!, value: jsonPart(notes) }, + { member: EDITOR.state.count!, value: jsonPart(notes.length) }, + ]; + peer.send({ kind: "bound", holds: [{ hold, variant: EDITOR.variant, members, failed: [] }] }); +} + +/** + * Records weakly every instance of a class that calls `method` from now on, for GC tests of objects a test cannot + * reach (holds, remote objects). Patches the prototype for the rest of the test file. + */ +export function instancesOf( + Class: { readonly prototype: T }, + method: keyof T & string, +): WeakRef[] { + const instances: WeakRef[] = []; + const prototype = Class.prototype as Record unknown>; + const original = prototype[method]!; + prototype[method] = function (this: T, ...args: unknown[]) { + instances.push(new WeakRef(this)); + return original.apply(this, args); + }; + return instances; +} + +/** What a route's `resolve` returns in the relay tests. */ +export type Resolved = { + readonly link: Link; + readonly principal: TestPrincipal; + readonly trusted?: boolean; + readonly codec?: Codec; + readonly limits?: PartLimits; +}; + +/** What `relayed` builds: the hosts, the links the server holds, the resolves it ran and the errors per host. */ +export type Relayed = { + readonly tab: Host; + readonly server: Host; + /** The worker of each resolved session address, the latest one per key. */ + readonly workers: Map>; + /** The server's end of each resolved route link, in order. */ + readonly routeLinks: Link[]; + /** The server's end of the tab's link. */ + readonly tabLink: Link; + /** The address keys `resolve` ran for, in order. */ + readonly resolves: string[]; + readonly errors: { readonly tab: HostError[]; readonly server: HostError[]; readonly worker: HostError[] }; +}; + +/** + * Three hosts: a tab connected to a server, and a server routing `Session` to an in-process worker per address (made + * by each `resolve`, connected before it loads, W6) that serves `PlacedNotes` and `Notes`'s store. The tab acts as + * `tab` (owner by default); `authorize` admits everyone by default; route links are trusted when `trusted`. + * `server` and `worker` add setups of their own; `resolve` wraps the default one (`worker()` makes the worker). + */ +export async function relayed( + options: { + readonly tab?: TestPrincipal; + readonly authorize?: (principal: TestPrincipal, address: { readonly key: string }) => boolean; + readonly trusted?: boolean; + readonly codecs?: { readonly tab?: Codec; readonly route?: Codec }; + readonly server?: (host: FacetHost) => void; + readonly worker?: (host: FacetHost) => void; + readonly resolve?: ( + address: Address, + worker: () => Resolved, + context: Context, + ) => Promise; + readonly context?: Context; + /** The server's and the workers' `revokedBy`. */ + readonly revokedBy?: (principal: TestPrincipal) => readonly JsonValue[]; + /** The server's `revokedBy` when it differs from the workers'. */ + readonly serverRevokedBy?: (principal: TestPrincipal) => readonly JsonValue[]; + /** The compressor of each host: tab, server, workers. */ + readonly compression?: { + readonly tab?: Compression; + readonly server?: Compression; + readonly worker?: Compression; + }; + /** The limits of the tab's link at the server. */ + readonly limits?: PartLimits; + } = {}, +): Promise { + const context = options.context ?? BACKGROUND_CONTEXT; + const errors = { tab: [] as HostError[], server: [] as HostError[], worker: [] as HostError[] }; + const workers = new Map>(); + const routeLinks: Link[] = []; + const resolves: string[] = []; + const { revokedBy, compression = {} } = options; + const server = createHost({ + principal: { kind: "server" }, + revokedBy: options.serverRevokedBy ?? revokedBy, + compression: compression.server, + onError: (error) => errors.server.push(error), + }); + const codec = options.codecs?.route ?? testCodec(); + const worker = (address: Address): Resolved => { + resolves.push(address.key); + const [a, b] = memoryLinkPair(); + const principal = { kind: "worker", session: address.params.id } as const; + const host = createHost({ + address, + principal, + revokedBy, + compression: compression.worker, + onError: (error) => errors.worker.push(error), + }); + host.connect(b, { principal: { kind: "server" }, trusted: options.trusted, codec }, context); + const facet = defineFacet((h) => { + serveNotes(h, PlacedNotes); + options.worker?.(h); + }); + void host.load([{ name: "@test/worker", facet }], context); + workers.set(address.key, host); + routeLinks.push(a); + return { link: a, principal, trusted: options.trusted, codec }; + }; + const resolve = options.resolve ?? (async (_address, made) => made()); + server.route( + Session, + { + authorize: options.authorize ?? (() => true), + resolve: (address, routing) => resolve(address, () => worker(address), routing), + }, + context, + ); + await server.load( + [{ name: "@test/server", facet: defineFacet((h) => options.server?.(h)) }], + context, + ); + const tab = createHost({ compression: compression.tab, onError: (error) => errors.tab.push(error) }); + await tab.load([], context); + const [a, b] = memoryLinkPair(); + const tabCodec = options.codecs?.tab ?? testCodec(); + const accepted = { principal: options.tab ?? { kind: "owner" }, codec: tabCodec, limits: options.limits } as const; + server.accept(a, accepted, context); + tab.connect(b, { principal: { kind: "server" }, codec: tabCodec }, context); + return { tab, server, workers, routeLinks, tabLink: a, resolves, errors }; +} + +/** + * A `resolve` for `relayed` whose workers are raw peers playing the provider, collected in `peers` in order; their + * links trusted when `trusted`. + */ +export function rawWorkers( + peers: RawPeer[], + options: { + readonly calls?: number; + readonly ack?: boolean; + readonly trusted?: boolean; + readonly limits?: PartLimits; + } = {}, +): (address: Address) => Promise { + return async (address) => { + const [a, b] = memoryLinkPair(); + peers.push(new RawPeer(b, options)); + const principal = { kind: "worker", session: address.params.id } as const; + return { link: a, principal, codec: testCodec(), trusted: options.trusted, limits: options.limits }; + }; +} + +/** A host's incarnation, which its handshakes send. */ +export function incarnationOf(host: Host): string { + // Every host is a `ChordHost`. + return (host as unknown as { readonly incarnation: string }).incarnation; +} + +/** Counts the bytes every `DecompressionStream` puts out while installed; `restore` puts the real one back. */ +export function countInflated(): { readonly bytes: number; restore(): void } { + const Real = globalThis.DecompressionStream; + const counter = { bytes: 0, restore: () => vi.unstubAllGlobals() }; + class Counted { + readonly writable: WritableStream; + readonly readable: ReadableStream; + constructor(format: CompressionFormat) { + const real = new Real(format); + this.writable = real.writable; + this.readable = real.readable.pipeThrough( + new TransformStream({ + transform(chunk, controller) { + counter.bytes += chunk.byteLength; + controller.enqueue(chunk); + }, + }), + ); + } + } + vi.stubGlobal("DecompressionStream", Counted); + return counter; +} + +/** The tables of `archiveStore`: edits (v1 with an index; v2 adds `done`) and notes, whose v2 migration crashed. */ +export const ArchiveTables = (() => { + const EditsV1 = defineTable("edits", { + version: 1, + columns: { + id: str(), + path: str(), + at: int(), + note: opt(str()), + meta: opt(json(obj({ a: int() }), { version: 1 })), + }, + key: ["id"], + indexes: { by_path: ["path", "at"] }, + }); + const EditsV2 = evolveTable(EditsV1, (t) => t.add("done", bool({ default: false }))); + const NotesV1 = defineTable("notes", { version: 1, columns: { id: int(), text: str() }, key: ["id"] }); + const NotesV2 = evolveTable(NotesV1, (t) => + t.rows((row) => { + if (row.id === 120) throw new Error("crash"); + return { text: `${row.text}!` }; + }), + ); + return { EditsV1, EditsV2, NotesV1, NotesV2 }; +})(); + +/** + * A store holding a bit of everything an archive carries: `client:app.store:edits` at version 2 with 150 rows (two + * export pages; some written at version 1, `NULL` and `json(...)` cells, an index), `client:app.store:notes` left + * `migrating` at version 2 after one committed chunk, and documents: a singleton, keyed ones with deltas after their + * base, and a retired one. + */ +export async function archiveStore(options?: { readonly outbox?: boolean }): Promise { + const { EditsV1, EditsV2, NotesV1, NotesV2 } = ArchiveTables; + const store = await memoryStore(options); + const fence = { admit: () => true, adopt: () => {} }; + const edits = await tableAt(store, "client:app.store:edits", EditsV1); + const notes = await tableAt(store, "client:app.store:notes", NotesV1); + const staged: StagedWrite[] = []; + const editChanges = new TableChanges(edits, staged); + for (let i = 0; i < 150; i++) { + editChanges.put({ + id: `e${String(i).padStart(3, "0")}`, + path: `p/${i % 4}`, + at: i, + ...(i % 2 === 0 ? { note: `n${i}` } : {}), + ...(i % 3 === 0 ? { meta: { a: i } } : {}), + }); + } + const noteChanges = new TableChanges(notes, staged); + for (let i = 0; i < 150; i++) noteChanges.put({ id: i, text: `t${i}` }); + await store.commit(staged, fence, BACKGROUND_CONTEXT); + await prepareTables( + [{ definition: EditsV2, table: "client:app.store:edits" }], + store, + fence, + "app", + BACKGROUND_CONTEXT, + ); + await prepareTables( + [{ definition: NotesV2, table: "client:app.store:notes" }], + store, + fence, + "app", + BACKGROUND_CONTEXT, + ).catch(() => {}); + const address = { namespace: "chord", owner: "client:app.store", kind: "memo" }; + const base = (value: JsonValue) => ({ version: 1, fingerprint: "f1", value }); + let ids: readonly number[] = []; + const created = { + admit: () => true, + adopt: (_: number, created: readonly number[]) => { + ids = created; + }, + }; + await store.commit( + ["", "a", "b", "gone"].map((key) => + createDocument({ ...address, key }, "latest", base({ title: key }), JSON.stringify({ title: key })), + ), + created, + BACKGROUND_CONTEXT, + ); + const [, a, b, gone] = ids; + const delta = (title: string) => ({ kind: "delta", version: 1, ops: [["s", ["title"], title]] }) as const; + await store.commit( + [changeDocument(a!, 1, delta("a2"), JSON.stringify(delta("a2").ops)), retireDocument(gone!, 1)], + fence, + BACKGROUND_CONTEXT, + ); + await store.commit([changeDocument(a!, 2, delta("a3"), JSON.stringify(delta("a3").ops))], fence, BACKGROUND_CONTEXT); + await store.commit([changeDocument(b!, 1, delta("b2"), JSON.stringify(delta("b2").ops))], fence, BACKGROUND_CONTEXT); + return store; +} + +/** `definition` prepared as `table` in `store`, and a handle to it. */ +async function tableAt(store: ChordStore, table: string, definition: TableDefinition): Promise { + await prepareTables( + [{ definition, table }], + store, + { admit: () => true, adopt: () => {} }, + "app", + BACKGROUND_CONTEXT, + ); + return new TableHandle(definition, table, store, tableOwner()); +} + +/** Every part of `store`'s export. */ +export async function exported(store: Store): Promise { + const parts: ArchivePart[] = []; + for await (const part of store.export(BACKGROUND_CONTEXT)) parts.push(part); + return parts; +}